HI!

Worms don't have any natural predators except maybe some rodents. So, if they aren't rodents, they won't hurt the worms and you don't have to remove them.

One exception might be the Black Soldier fly where its larvae eats so much and so fast that the worms may not be able to get enough food. The larvae are black and 1/4" or so long and don't look like maggots.

Cheers!