No pics showing..................

Swarming as "in flight" or as crawling on the deck surface? Wasps unlikely to do this if deck has no food material spilled on it.

I've had wasps taking the wood off my unmaintained deck before. I assume they built the actual nest under someone else's deck. :)

They may be carpenter bees. Or possible a hornet wasp. Both have been a problem for me in my yard and deck.

The carpenter queen bores a fascinating hole and burrows in to lay egg in a tunnel. The ones swarming are the males. They don't physically attack but sure do look menacing. They remember their "home" area every year. From my research the carpenters are not pollinators so I did not feel bad plugging up the hole they made.

For the carpenters look on the underside of your deck joists (if you can) and the underside of the hand railing. Sometimes you can see the shavings they left behind as well. The hole is about the size of a dime. I think they choose the underside as it is less visible but more importantly is not painted or stained. As Fori importantly mentions, poorly maintained wood is more susceptible.

I sprayed their hole then glued in a wooden dowel. Unfortunately my efforts to stop the carpenters failed. We saw them buzzing again this year and as we pulled off the deck railings to re-build I found several holes on the unpainted underside. We had to pull off the rim joists as well and are putting in new ones. I plan on staining the underside this time. However, all the other joists are remaining so I hope those buggers don't start under there. They actually can do a lot of damage over time and ruin the structure.

The wasps made a paper nest high in a tree. Is your deck several stories above grade? My guess is that if your deck is close to grade, they are not making a nest there. If you were several stories up, it is possible. The color of the wasp nest tends to blend into the landscape and of old grayed wood. When I called the exterminator, they told me to leave the nest alone as it was high enough and they would not attack unless provoked. Unlike the carpenters, they will attack if they feel threatened. Do not destroy the nest if you can reach it.

HU, we are waiting on pictures from you. Without them, it's just a fruitless guessing game.

I think the wasps have carried off the OP. Never to be heard or seen again.