Before you start painting, powerwash the siding and the brick. Then paint the trim brick-red. The red that's there is too red; add some brown to it. Some siding can be painted if you're looking for a color that's more harmonious with the brick. Paint the door the same reddish brown as the trim.

Other ideas: For a more inviting entry, when budget allows, put in a brick walkway and brick patio to flank the front door. Or create front garden beds with shrubs, perennial plants, and reddish brown mulch.

I agree with above - a brownish red/brown.

Here're a yellow home with red trim, though redder than what you should go for. Yours needs a bit more brown in the red trim.

PIck out the darkest color in the bricks to use for trim. With reddish brick and dark roof, you already have a classic Southern combination. The siding needs to be powerwashed or soap-and-water washed before priming for new paint.

I tried some trim colors today. A grey and a reddish brown. Nothing helps that yellow. We are going to paint it a light grey and go white with the trim. Thanks!!!

Oh no, the yellow would have to go...

I very much agree with PaintColorHelp's thoughts, above. The siding does not complement the brick and I think that is the main issue.

I think my personal preference, with your house, would be red siding and grey trim, so I would be looking to paint the siding a red color. Another option could be grey siding with white trim.

Some inspiration pics:

We decided to paint the siding. It was so much easier than we thought. Sherwin-Williams makes a great product in vinyl safe colors. Here’s our in progress photos. We like the darker trim which was a surprise. Thanks for the suggestions to paint the siding so glad we did.

New photos?

I don't think your pics uploaded. Glad you guys are happy. :)

Still need to finish the top section this week.

Make sure to get your roof powerwashed. It'll brighten up the whole house. Then you can devote your energies to something rewarding that will really take the house up a notch, landscaping with shrubs, ground cover, perennials.