Ants are usually a net neutral within a garden system. They don't eat the plants and they don't eat the things that eat the plants. Potentially there are two problems. 1)disruption of the roots through tunneling, which is rare and personally I have never actually seen a plant damaged by this. 2)ants sometimes care for and "farm" aphids so they can eat aphid secretions. This activity does harm plants because the aphids harm the plants. So, as long as you don't see any aphids, I wouldn't worry.