There seems to be some misconceptions about ants and aphids. You can have gobs of aphids and no ants. You can have ants all over the plants, but no aphids. Some ants might be going for the tomatoes (fire ants, for example), or moisture, or even nectar. The ants may have taken up housekeeping in the potting soil and are simply using the tomatoes as foraging territory.

So what should I do with the ants? Just leave them or ?

I see ants on my tomato plants too. I have been growing mine in 5 gallon buckets. I just gently hose down the leaves when the sun is setting. I am not sure if this helps or not, as this is my first time planting tomatoes.
I do know to wait until the plants are not in direct sunlight because then it could get sunburned. I hope this helps.