Could it be that you had put down some kind of poison for the ants you forgot about? Boric acid seems to kill ants and mixed with water and something sweet like sugar, makes for a great ant bait. Here, I see ants all over my garden, which don't seem to affect any plant growth. I am now growing tomatoes, peppers, dill, basil, string beans, peas, limas, thyme, oregano, rosemary, asparagus, onions, garlic, chives, fruit trees including apple, peach, raspberries, blueberries. None of these seem to be bothered by any ants.

Ants will often come and go according to the weather. They'll often come into the house if it rains, or in a drought, for instance.

If they're after termites, they'll invade, too. Once the population of termites has depleted enough not to be able to support the population of ants, they'll go hunting elsewhere.

Sage, pennyroyal, peppermint, rue, mint, bay, lavender, marigolds, pyrethrum, wormwood, yarrow or tansy will all deter ants.

If they got to your coffee grounds, they wouldn't like that, either. And they don't like vinegar, or chilli, or cucumber.

Boiling water isn't so good for them, either!

Thank you guys! I have mint and marigolds back there and I do put coffee grounds in my pots as well. I thought it may have been my echinasia.