You have so much to choose from! A lot will depend on the weather - our springs can be early or late - but referring to my garden journal for what's in bloom in my coastal Monmouth County garden May 20-June 4:

Herbaceous peonies, spiderwort, fernleaf bleeding heart (dicentra eximia, not spectabilis), rhododendron, roses, iris, mullein, salvia, lavender, hardy geranium, thrift, columbine, weigelia 'Wine & Roses', mock orange, coral bells, deutzia, hydrangea, mountain laurel, tiarella, nepeta.

Great list. Thanks! Gives me a lot to work with. Can these be planted in October?

If you're starting with plants in containers, just about anything can be planted in September/October and will be better off for it. In spring, the air is warm and the sun strengthening, while the soil is still cold. In fall, the air is cool and the sun weakening, placing less demand on the plant, yet the soil is still nice and warm, promoting good root growth in the new site.

Fall is ideal for planting bare-root peonies. The herbaceous peonies are beautiful and should be in bloom in early June. The gorgeous, but less commonly seen, tree and intersectional (Itoh) peonies bloom a bit earlier, so check bloom times on those.

I meant to check our list at work and add to the list I posted on Friday ... will try to remember to do that on Monday.