Basic rules of thumb:
Most plants (though certainly not all--Baptisia for example)that bloom early, look bad for much of the rest of the season. Use them sparingly.

Go for good foliage of various colors and consider the flowers the icing on the cake.

Consider the texture of the foliage (large leaves versus small) and the shape of the plant (domed, spiky, etc) as you plan. Interesting textures and plant shapes, as well as differing foliage colors, keep an area very interesting even when nothing is in bloom.

groundcovers I've found to be "civilized" (won't strangle other plants): sweet woodruff (May bloom, part shade), Irish moss (it's neither Irish nor moss - full sun), Platt's Black (sun or part shade), iceplant (full sun, dry & hot)

Like laceyvail, I plant with height & foliage type in mind.

I've had good luck with these keeping good foliage through the season, although some look untidy if they aren't deadheaded: sea thrift, dwarf monarda, germander, thymes, meadowsweet, lavender, iceplants, heather, chrysanthemums, and artemisias. All available in the bloom colours you want.

HTH!

I like using sedum with different colored foliage for a long lasting effect - some of my favorites are Xenox, September Charm, Class Act, and Purple Emperor. Purple Emperor needs to be cut back by at least half in late May/early June to keep in compact and avoid sprawl.