Yes, pinch off all blossoms and blossom buds if you can tell which they are with your fingers. You should really pinch them off again in the spring for the next few years so your tree builds its root and branching system to handle lots of peaches. I usually leave a few on my tree in the second year as I am excited to taste them.

I have have had the same happen on a couple of my trees. I usually wait till the flower dies so the bees can eat. Then I take them off.