Contrary to what most everyone believes ants are not major pests that must be eliminated at all costs. Ants are part of Ma Natures very large recycling machine here to clean up this planet after us. It is not very likely that the ants are actually the problem you perceive since they may be after what is the real problem.
According to the link below even Fire Ants may have a place in our world.

Here is a link that might be useful: the good about fire ants

Ants on plants usually means they are eating sap from holes made by other insects, like aphids. A hard stream of water on plants will knock off aphids. Check along stems and under leaves. Ants live in gardens. Keep plants healthy, and keep other bugs in check and you should have no problem with the ants.

By the way, you don't have to add worms. They will just eat the nutrients that are there. Add compost and the worms will come and break that down. Good luck.

Just so I understand, you're getting ants in your pants 'cuz there's ants on your plants?

Try Susan's water treatment first.

tj