You can remove that nest when you do not see any wasps going in and coming out. I have had numerous of these nests started that are never inhabited and if nothing is seen for a couple of days that is an indication it is not being used.

Thank you. I've never seen one wasp ever, but it still makes me nervous to stick my hand in there.
I think I'll see if my neighbor is interested in doing it :)

Most of the wasps die in the fall, and the rest leave the nest and find a sheltered crevice to overwinter.

Tie a piece of metal window screen over the light's opening to discourage them next year.