I had an ant problem recently too, farming pupae in the foliage of one of my plant. I soaked the topsoil with a pyrethrin-based solution, sprayed the leaves and stems, and put the pot on feet in little bowls of water. You can forgo the chemicals by just replacing the soil in the pot, but you have to cut off their access if you don't want them coming back.

If they're not in the soil, a strip of tanglefoot or tape around the pot might work, but they might decide to burrow up through the drainage holes.

If they're not hurting anything and they're not on your food crops, why not just let it be? I throw borage seeds around the outside of my compost pile every spring and it always grows like gangbusters and attracts all sorts of native bees and insects. I also have little black ants in the yard, and aside from keeping them out of the house, I tend to just ignore them. They don't generally harm healthy plants. If you notice they're farming aphids, that's the time to selectively apply Neem and look at tanglefoot, but other than that...

Ants are not a problem. Their presence can discourage other more damaging pests.

Do you have aphids? When I have a ton of ants it's always a concern I've found a great pesticides that is safe for humans and pets from home Depot by Dr earth, says it's safe to apply even on day of harvest. Smells like peppermint and sage

As for all these comments that say ants aren't a problem with borage, they destroyed the roots of my borage plant - literally devoured them :(