Are there wasps living in those nests or are they old nests from last year? Wasps, including the Yellow Jackets, do not use the same nests from year to year so if there are no live wasps in those nests (it is too early for the queens to be out and about here) there is no problem.
I know many gardeners that are Anaphylactic, severely alergic to insect bites as well as many othger things, that refuse to allow that condition to limit their lives and work in the garden, regularly and often. With the exception of the Yellow Jackets they all tell me that the other wasps and bees are not a problem and do not bother them at all unless they threaten the life of that wasp or bee.