Here are a few to consider:

-Lithodora doesn't need a ton of water once establised, but the flowers are blue or blue/white.
-Potentilla has yellow flowers and requires little water.
-Baby's breath might work in white.
-Green and Gold (Chrysogonum virginianum) has yellow flowers.
-Thread leaf coreopsis has yellow flowers and
makes a nice groundcover. Coreopsis nana has orange/yellow flowers.
- Ice plant is marginally hardy in the mid-Atlantic, but has yellow flowering varieties.
-Lysiachia nummularia has yellow flowers.

I can't think of any red flowering groundcovers offhand.

I hope this gives you some ideas!

Helianthemums - yellow, white, orange, pink and red available.

Several low growing hardy geranium species.

Aubretia, Iberis sempervirens, Aurinium saxatile, Saponaria ocymoides, Anthemis punctata subsp. cupaniana, Anthemis tinctoria, Dianthus, Festuca glauca (a grass), dwarf bearded iris - all classic tough full sun low growers.

Tough hardy herbs like the various thymes, Oregano and different coloured Salvia officinalis. Lavender.

BTW - Lysimachia nummularis needs moisture. It's a bog plant in its natural habitat.

To start you off this season you could scatter Nasturtium, Limnanthes, Eschscholzia and other sun loving annual seed until your perennials fill out.

Here are a few to consider:

-Lithodora doesn't need a ton of water once establised, but the flowers are blue or blue/white.
-Potentilla has yellow flowers and requires little water.
-Baby's breath might work in white.
-Green and Gold (Chrysogonum virginianum) has yellow flowers.
-Thread leaf coreopsis has yellow flowers and
makes a nice groundcover. Coreopsis nana has orange/yellow flowers.
- Ice plant is marginally hardy in the mid-Atlantic, but has yellow flowering varieties.
-Lysiachia nummularia has yellow flowers.

I can't think of any red flowering groundcovers offhand.

I hope this gives you some ideas!

Just a thought - what is on the other side of your fence? By their nature, ground covers spread, and a fence won't stop that. You might want to put in an edging just to one side or the other of the fence if you want your groundcover to stay where you put it.

Sedums like it sunny, hot and well-drained. I can't think of any red-flowered ones, though there are hot pink, like dragons' blood sedum, which has reddish foliage. Sedum acre has yellow flowers and will readily root from broken stems. Sedum 'Angelina' has yellow foliage (and in cool weather adds orange tints.) I imagine you can find other red or yellow foliaged or yellow flowered sedums with a search.

On the Lysimachia -- I have seen it growing well through our tough humid dry summers here (mid-Atlantic) without much issue or supplemental water provided it gets afternoon shade (in fact it loves to spread rapidly). The problem for you may be that the flowers are not the main attraction and they are only temporary (not ever blooming).

I second the geraniums -- I love Rozeanne (brilliant purple).

Thanks for all the great suggestions!!! I didn't think I'd have this many choices, so I'll put in a mix of plants with clusters of taller varieties surrounded by lower ones.

I hadn't thought about them spreading under the fence, but that could actually be a happy accident since I'm going for a lush cottage garden look (though it's going to take a few years to actually get there). There are deep flower beds on the inside of the fence, but since it's 4-5 feet tall, its shadier than the sidewalk side.

How about creeping thyme, which is suppose to be drough tolerant and have pinkish to red flowers.