Richters and Horizon Herbs can help you out!

FataMorgana

Nichols Garden Nursery also has a lot of them. That's where I got most of mine.

Here is a link that might be useful: Nichols Garden Nursery

I second horizon herbs for seed, and non-seed plants like bay from colonial creek nursery website.

Also check out Crimson Sage and Companion Plants.

Also, some of those you list are native wildflowers. Try Natives in Harmony in Harmony, Ohio, Prairie Nursery in Westfield, Wisconsin or just Google "Wildflower Nurseries" for a lot of others. You did not mention what state you are in, but I saw listings for nurseries from a variety of states.

Prairie Moon Nursery is awesome for native seeds and plants.

FataMorgana

Seeds of white borage available from J L Hudson Seedsman. Very beautiful and just as easy to grow-- with enough moisture-- as the blue borage. Place it in the vegetable or flower garden too.

Other plants in stardrifting's original list besides mugwort that, in my experience, are spreaders and take up a lot of room or are abundant self-seeders are costmary (this one really sprawls!), milk thistle (seeds will germinate all around), anchusa (I have seen this take over a roadside area), agrimony, and sweet cicely. These last two do not spread terribly fast, but they do self-sow in my yard. Climbing milkweed, while a native and attractive to pollinators, can become a problem since of course the seeds drift off in the wind. Depending on how much room you have, including a greenhouse, and the various ecosystems in your yard (sun, shade, moist, dry) many in the list above are not practical to try to grow in zone 5. Be especially cautious with non-natives in that a species may be introduced here that can be detrimental to the local habitat. (Witness kudzu, Elaeagnus spp., vining & shrubby non-native honeysuckles, multiflora rose, and countless others!)

I think oregano should be added to the list of invasives. I fight it constantly to keep it in bounds. And some people say they have a problem with chives although it's never been very aggressive in my garden.

Ah, yes - chives and garlic chives both can spread themselves rather quickly given the chance, mostly by means of those thousands of seeds! The bulbs reproduce and spread outward, but not as quickly as the seeds make new plants. I (mostly!) deal with it by watching the flowers closely, and as soon as they begin to fade, clip them before the seed ripens. Easy to grub out new seedlings if you get 'em really young. :-) I root-prune vigorously-spreading oregano at least once a year.

I've been growing oregano for years and never had an issue with it being invasive.

tripleione, you're lucky! I have oregano popping up yards away from the herb garden. Just have to keep digging it out.

Oregano may be aggressive but it isn't strictly speaking invasive unless it has started appearing in wild habitats and ousting the native flora.