I don't know about the soap but could you possibly hang a yellow jacket trap near by?

I had yellow jackets in my swallow boxes, I assume yours aren't occupied by birds yet or the bees wouldn't be there. I took the boxes down, (just after dusk, flashlight) and dropped them into a 5 gal bucket of soapy water. Cleaned them out the next morning with a brush and removed all traces of papery hive, returned them to their posts. End of yellow jackets. (my guess would be the soap would not appeal to birds either)

I've been on the bluebird forrum and they said to put a light coating of vaseline on the ceiling of the bird house - guess the wasps don't like that sticky feeling!

I make home made yellow jacket traps every year and catch thousands.
Take a 2 liter pop bottle and cut 4, 3/4 inch holes, evenly spaced around the bottle. (About 5 inches up from the bottom)
Then boil up some sugar water. About 1/2 water to 1/2 sugar. Let it cool, then pour a couple of inches in bottle.
Replace the cap and hang it from a tree.
The yellow jackets will crawl in the holes for the food but can't crawl out. All they want to do is fly upwards. Eventually, they die and fall into the water.
Be carefull throwing away the old bottle when you feel like it. Then replace it with a new one.
I have many hummingbird feeders and found that this is the only way I can keep them under control.
Shiza

For an easy and fun yellow jacket trap....open a wine cooler, drink half. Place the half filled narrow necked bottle on your deck railing and leave it there. The drunken/drowned yellow jackets no longer annoy the guests you are trying to serve dinner to on your deck at least....

(but make certain no one mistakes the wine cooler bait for their own beverage)

Moogies, thank you for the vaseline tip, that might just work as a preventative, I doubt if it has enough scent to deter birds.