Consider Cranberry Cotoneaster and/or Royal Burgundy Barberry. Both are Deer resistant and produce little berries. Nice color as well. Check your local nursery for availability.

Thank you! My only concern is that both of those plants require full sun and this area is mostly shaded.

I have the same situation in my yard. Erosion of the soil was also a problem. Also the deer seem to eat everything in sight. The side of the yard where the slope is, is about 30+ feet. We inherited some design from the previous owners who mentioned that they had no gardening knowledge at all. They scattered large rocks sporadically about and planted away. Most of it just washed away or the deer devoured it. There is a very large evergreen tree in the middle top which keeps it very shaded. This is what works for us. Bleeding heart in deep pinks, Lenten rose in paler pinks and greens. They also start blooming in late winter. Ferns are great grouped in masses. Try some Coral bells too. They can take shade. I have probably 30 or so astible scattered in the yard and also on the slope in groups. They give beautiful color in spring and maybe into June.
My favorite though is the Japenese Andromeda, also called Pieris japonica. Look into the burgandy. It gives amazing color all winter and then in summer it pales a bit and then deepens. It is truly a year round jewel.
Use pachysandra on the slopes. I want color and form all year so the choices I made do work. You don't need a lot of variety, just a lot of a few things.
And mulch, mulch, mulch to keep the erosion under control!