Are you picking just the petals or pulling off the entire flower?

If you look at how fruits form, they form on the flower, so if you pick the entire flower you are really picking the fruit before it has formed.

It probably wouldn't hurt to just pick a few blossoms if your tree is normally loaded with fruit, or if it's a small tree and it is likely to produce too many fruits for its size.

Removing the blossoms will remove the future fruit, period.

You don't mention age and size of the tree. Keep in mind that every blossom on a citrus tree will produce fruit. The tree will abort what it cannot carry anyways. If you want to pick some of the flowers, then choose those that are in clusters as not all of them will turn into fruit.