Best guess is the tomato fruit worm. The only 2 "worm" pests that bother with my tomatoes are these and the imfamous Hornworms.

{{gwi:1299541}}
Tomato Fruit worm

Are they mostly green? If so, they are likely the tomato looper. They are more likely to be found all over the leaves (in large numbers) than the fruitworm.