Others have their opinions but I don't think the garden is a good place for ants unless you're a messy gardener and leave rotten fruit on the ground which isn't a good idea because that draws more pests. Some ants farm pests like aphids and scale so if the ants aren't eating rotten fruit they might be there for the honeydew that these pests create. I get rid of ants in my garden and keep any fallen fruit picked up.

Your garden will be fine if you leave the ants to roam. To prevent any aphid farming by ants (and having ants everywhere), just spray any aphids you see off the plants. My cowpeas seem to be the only vegetables that ants are interested in -- they love the nectar produced at the leaf-axils. But they don't harm the plants themselves so I don't worry about them.

It doesn't appear that they are damaging the plants,

Then no you don't need to do anything. There's no such thing as a bug-free garden and the vast majority of them are either beneficials or neutrals. Ants are nature's gardeners and they not only help to break down organic matter and aerate the soil, but some species aid in the control of more damaging pests.

Dave

PS: red fire ants being an exception to the rule ;)