Composting worms don't eat plants or plant roots. If there was something wrong with the plant it was probably water, lack of or to much. Having a worm or group of worms would be healthy for your plant as they would be producing a rich compost that the plants can use readily.

If you were overwatering the plant the roots would have rotted and the worm could have eaten those rotting roots, getting huge and fat.

If you want to get rid of worms in your houseplants, just very gently submerge them in a bucket of water. The earthworms will come out of the soil. I had a very large earthworm in one of my pots for about a year, and disposed of him in this manner. I don't think they do much damage, but they make the pot look unsightly.

I purposefully put 2 worms in a houseplant after reading these two articles:

The Smallest Horticulturist
Earthworms: Champions of the Soil

According to these and other sources, having a few worms (alive or dead) in your pots is good for the plant.

My plant is growing now, but since the worms have only been in the pot about a month, I'm not sure if the improvement is caused by the worms or just a seasonal change.

BTW, I used Allolobophora chorotica (green worms) in the pot. Since these worms are such slackers in the soil (they seem to just hang out/curl up in the soil and not do much), I figure it's no big loss if they die in the pot.