Hi liser8277! We love the Georgian/Federal style of your home. The only recommendation we would make is to replace your current shutters with shutters that are properly fitted for your windows. We think with decorative hardware that, that will truly make a difference! You could combine louver and panel styles on your home or even stick to one style of shutters.

Make the front door a color that contrasts with whatever you use for shutter/trim colors
agree that the current shutters are basically faux since they don't really work to cover the windows...
Black/white are usually the traditional choices but with gray being so popular that might work since brick is basically monotone...
Add color for door that pops--like darker red or maybe orange or even turquiose...
green would not be my pick nor yellow...since brick is so pink but dark purple/aubergine might look better than you think...