I'm no expert and don't have this problem but it sounds like plum
curculio.

Oriental Fruit Moth and plum
curculio.

Plum curculio East Rocky mountains Just fruit damage.
Spray during the petal-fall stage They lay eggs in fruit when small as match head after size marble they want that reason you see damage down to seed moth layed egg when was small egg needs to be right next to seed. You have to kill moth you can't kill egg, so timming of spray is before lay egg, when blooms start fall begin spraying. Then stop spraying at marble size same all stone fruits. I look every morning around my bug light to see if they are around if Peach get to right stage I spray them to, O I spray around bug light to in morning while humidity up and wing almost don't work until 10:00 a.m., They fly and lite in short distance that time day. I use Permethrin 33% you find 10% at many places. Pets insecticide dog and cats live stock supplies are full of Permethrin.
Oriental Fruit Moth everywhere mostly twig damage from where egg layed twig die from there out to tip.

Thanks for the info! The plum curculio sound like tough little critters. I will spray next Spring.

Don't know where you live. Plum Curculio and Oriental Fruit month are the two biggest pest problems of plum and peaches in the east coast. (they attack other pomme and stone fruits, too)

PC has about 2 generations in one growing season while Oriental Frult moth could have up to 5 generations. Timing of spraying in the key.

If you search old threads on PC and OFM on this forum, you'll see.