It's considered an invasive species in Maryland too. Might I suggest purchasing a sterile version, that won't spread aggressively by seed, rather than planting those outdoors, that clearly will?

Yes, please, only a sterile version.

do those seedlings even look like Buddleia???

I have never seen a wild one here.

http://www.buddlejagarden.co.uk/seeds.html

Mine do look similar.

Oh, there's no doubt they're Buddleja, Ken. I've weeded out a lot in my time.