For the most part insects are not harmful to your worms, they are just trying to get in on all the composting material as a food source. They can however be a sign of issues that might effect your worms' habitat.

For example, when your compost pile outdoors is too dry it can get overrun by ants. The ants do not kill worms, but once conditions are less favorable for the worms and they begin to die off the ants might feed on the dead.

I personally would not be worried. I would recommend freezing your scraps in the future as that will kill off whatever eggs are on them as well as break down cells walls of the food to speed up decomposition. Banana peels are notorious for setting off fruit fly and fungus gnat infestations.