You could paint the siding a shade of grey to match/complement the roof.

Personally, I love brick houses, and particularly red brick houses. So the idea is, I love brick and I am not afraid of red houses, etc. so for me, I would have no problem painting the siding a shade of red. Maybe a brick red... between that and a deep rust color... along those lines. Because honestly, since the house is primarily brick, the two toned thing would not be particularly appealing to me. And the white trim on the house is stark to me, also - so I would probably do a grey or taupe / some other color on the trim. And maybe a black door.

I would look to match or complement existing brick with the siding color.

Or maybe wood.

I like the combo of colors shown below and think it would work well with your homes brick and existing roof color on your home.

hardie board combined with stone exterior - Google Search · More Info

You might want to remove the link since it shows your address.

I think I don't like the brown trim on the clapboard. I would paint the clapboards brown like the trim. Or a lighter shade of brown. Paint the door surround brown, too.

Get a new door. The Victorian style doesn't fit your house. Paint it a bright interesting color.

Put shutters on the middle window. It will draw the eye to the center of the house where the front door is.

Plant foundation plantings on the left side of the house.