I've just noticed that I have a good number of tiny (flea sized) ants roaming around on most or all of my pepper plants. Is this something I need to be concerned about? Are they damaging the plants or just living on them?

I know that ants sometimes indicate aphids, but my plants look pretty good; the leaves are big and nice looking. Some of my plants have few peppers, but I think that that was an over-fertilizing issue.

So, should I get rid of the ants? And if so, how do I do that?