well it could be yellow jackets they nest in the ground. Many stung my elderly mom last year when she was weeding around a tree. Had to go to ER.
I had a little bird house, decorative, on the ground, and wasps made a home in it, with that grey honeycomb nest in there. My little poodle must have nosed around it and they stung her in a swarm, got 6 stings before I could get her in. They got very infected and she had to go to vet and have them cared for (too gross to go any further) and on antibiotics. So be very careful. You might even need an exterminator to check it out and destroy. Or if you even begin to think about finding, destroying them do so at night when they are most inactive. Maybe google for some more info on what they look like and nesting habits.

They were probably cicada killers. They are ground-dwelling wasps, but they are beneficial, non-stinging wasps. The males are normally the ones that you see out flying, and they do fly very aggressively (but cannot sting). They look huge and very mean, but they do not sting. In my area, they are around for probably 6 weeks and peak for about 3 weeks. If this is what you have, you can just leave them alone to hunt cicadas (in fact, they are probably gone by now).

{{gwi:1263221}}

I'll bet a billion dollars that if they were Cicada Killers, the words huge, gi-normous, humongous or similar would have been part of the original post.

Yellow jackets are small, around 1/2 inch in length. Cicada Killers are extremely large, from 1 to 2 inches! No mistaking them from one another.

Carolinian, who knows what you may be seeing. Lots of different kinds of wasps live or make nests in the soil, and some just collect loose soil to make their nests. I've watched that kind of a flurry of activity from mud daubers and potter wasps many a time.

Wow, you guys answered! I should have checked back sooner

I think they were probably blackjacket wasps
See: http://www.cirrusimage.com/Hymenoptera/blackjacket_800.jpg for large picture

Those are basically a rarer form of yellow jacket. We had trouble with them last season, but we haven't seen an actual nest out back this season.

We did spot the enormous cicada killer wasps here too and they're scary...
but they don't swarm, since they're solitary

These are solitary Digger wasps which dig a burrow for each female. Probably not yellowjackets which are colonial in a single nest with a single entrance hole and are aggressive. These solitary wasps are pretty harmless and are busy collecting prey to drag down their holes where they lay an egg. The prey is "babyfood". There are many types and are interesting to watch-even close-up. You will not find the huge cicada killers (also a digger wasp) in groups. Paul.