No need to take out the weigela or forsythia - these will make a good start to a mixed shrub border. Others you can add are ninebark (available in a range of foliage colors), shrub dogwoods, PeeGee hydrangea, elderberry, viburnums, lilac and clethra (in the shadier area). These are all very tough, durable shrubs adaptable to range of conditions. A soil test shouldn't be necessary, provided what's there is growing well, but it won't hurt and it's good information to have on hand. Deer will eat anything if they are hungry or curious enough but these plants will all tolerate occasional browsing well. I would offer some protection to very young or small plants though, at least until they get established and put on some growth.

Consider also flowering quince, hydrangea arborescens and hamamelis. Maybe lindera benzoin, too. These should have a good chance of being left alone by the deer. I would also plant a couple of blueberries just to see if they make it.

Consider also planting some nice flowering trees and ring fencing them for a few years.

The deer here in SC LOVED my blueberries, until I fenced the bushes in and the deer out.

Cranberries might do, as might one of the deciduous hollies, for berries - you would need a male to get berries. Inkberry is an evergreen holly that might work, although the dark berries aren't especially interesting.

Either red or black chokecherry is a possibility. Another might be either grey or silky dogwood, or red osier/red twig dogwood, Cornus amomum, recemosa or sericea, respectively.

Hinoki or Sawara false cypress might have several cultivars of a size and shape that suits, taking both sun or part-shade. Another evergreen, although more for the sunny spots, would be a Juniper cultivar such as 'Grey Owl' - not all junipers are low and spreading.

Mountain Laurel is a broad-leaf evergreen that might suit for the sunnier parts - it will take shade, but will flower less for you than here, where it needs shade - there are other colors of flowers in the cultivars than just the pink/whites, shading into reds. And some of the native azaleas, both the evergreen and deciduous, might well add flowering interest - some also are heavily scented.

Deerproof is tough. Forsythia and weigela should be OK and the big, white-flowering spireas - or really any spireas. Maybe lindera benzoin, flowering quince or hamamelis. Hydrangea arborescens varieties did OK in my garden against the deer, but Z5 was really too cold for oakleafs.

Deer ate my lilacs, mockoranges, roses, viburnums, rhododendrons, junipers, myricas - the list goes on and on!

Consider adding a few flowering trees for height/variety and fence them for a few years until they are no longer so vulnerable.