I'd get rid of them.
If you don't have aphids you will.
They will bring them in and protect them.
Aphids spread disease.
They might be accidently pollinating your buds just by walking around the buds.

Ants don't get any mercy from me when I see them.

Your sudden spurt of buds setting could be due to wind polination,weather change,calcium changes in the soil etc.

Terro ant killer,belly up to the bar boys.Drinks are on me.
Here ya go,take some home for the queen...

I have observed ants collecting pollen, but they seemed to be taking it home with them to eat. By the way, be sure your black ants aren't fire ants. I stuck my hand in a pot about a week ago and came out covered with fire ants, the imported kind. Needless to say I got stung several times and they are just now healing.
john

I agree with Smokemaster. I overwinter my peppers and trust me Ants = Aphids = Dead plants.

Get rid of them in any way you can right now.

Ravi

I have seen with my own eyes ants carrying aphids on their back onto the plant, ants are there only to enjoy that juice aphids produce. So get rid of them if you can, not easy though bloody ants

Oh man! And I was hoping for a mutated brood of high pollinating super ants.

Points taken. Today they meet their maker. Belly up indeed.

Thanks Everybody!

My peppers are in containers outside and they are full of ants...the little brown ones...they bite, but I don't think they're fire ants. They're everywhere in the ground here and come up through the pots. I haven't seen any aphids all season. Maybe they're just after pollen? Do all ants farm aphids? They haven't done any damage to the plants, but I water thoroughly and I think that washes many of them out regularly and keeps them under some control. I brought aphids inside on a plant one year, but there were no ants...just aphids...took a while to get rid of them.