Aphids suck on the leaves of fruit trees, not the fruit. But it is another type of insect burrowing into the fruit. You should dispose of the fruit with holes in it to cut the life cycle of the insect inside. Could be peach twig borer or oreintal fruit moth https://utahpests.usu.edu/ipm/htm/fruits/fruit-insect-disease&pub-peach-pests

Would that affect the blooming or leafing this year?

Look at the base of the tree for a jelly like like goo. If it's there, you have borers and you need to dig them out. They can kill the tree.

there's no ooze at the base.

This is my tree today

Very few leaves, no blooms

The tree could use a good pruning, but that's not the problem. You may not have received enough chill hours, and that's why the tree is still "asleep."

The buds do look health though, its probably only a matter of time before bloom

i would prune that tree extensively though (perfect time to do so), did you know about half the wood on a peach tree should be pruned away every year?

I did not know that. I will research the pruning techniques to start pruning my tree properly. Thank you

Should I prune it more than this?

That's what I'm talking about!!

i would say just to get rid of the branch going straight up into the middle to allow more light, right where the saw is actually