seems ( to me ) like the ground it hard and there are alot of roots present creating a hard layer of soil which is nutrient deficient in the toplayer. i think you need to improve your soil.

Remember variegated forms often need more light

From the top of my mind :

Fatsia japonica
Bergenia cordifolia ( purpea is also a good one )
lysimachia nummularia '' aurea''
Skimmia japonica
Pachysandra terminalis ( variegated form is also nice)

Most ferns can be grown in shade, asplenium scolopendrium is good looking and reliable. shield ferns ( polystichum setiferum ) are evergreen and shade tolerant.
Athyrium niponicum pictum
Cyrtomium fortunei clivicola
Dryopteris erythrosora
Matteuccia struthiopteris

Stephanandra incisa
Prunus laurocerasus
Waldsteinia ternata
Ajuga reptans ( burgundy cultivar is good looking )
hedera helix purpurea
Hedera helix buttercup

Many carex species can be grown in shade. carex morrowii , '' ice dance is a good looking cultivar.

epimidium species ( difficult in sandy soil but possible)
salomons seal ( prefers humus rich soil)
Tiarella species ( tiarella cordifolia )
asarum species (asarum europaeum)
Hops can grow in shade , silver lace vine as well
Azalea
rhododendron
Liriope muscare
luzula sylvatica
Mahonia species.. especially Mahonia ÃÂ media
certain Eonymus and berberis species can grow in shade
Vinca minor , major etc
Certain bamboo species ( clumping species are less invasive )
Hydrangea
gynostemma pentaphyllum ( need humus rich soil and full shade )

certain viburnum species ( especially evergreen ones ) can grow in shade.. viburnum tinus flowers in winter and is evergreen.

certain Cotoneaster species
Fallopia aubertii
Aristolochia macrophylla
Jasminum nudiflorum ( can grow in shade but needs half shade )
Lonicera nitida
Parthenocissus species
Phyla nodiflora
Celastrus orbiculatus ( invasive )
groundcover Geranium species
Helleborus species
Rhus

some stuff is invasive like vinca , so check it out if its invasive or not

Camellia

Some good news, two and a half of those trees will be going away over the course of the next several months, and the rest will be getting thinned out a bit. Hopefully that will improve the light situation a bit. Then we can worry about improving the soil where it needs it...

I'm trying to make these cages out of hardware-cloth to keep the leaves from burying stuff. It's kind of a pain but I'm hoping moss and stuff will grow there if I don't have to rake that one spot.

Re: the bamboo. There are both clumping and running forms of bamboo. Clumping bamboo spreads no more than any other sort of ornamental grass and since it blooms so infrequently, of no issue with setting unwanted seed. And even running bamboo can be easily contained using the correct planting methods.

Horror stories or expressions of extreme aversion to planting bamboo usually originate from situations where the bamboo variety was not thoughtfully selected, not planted properly or just through lack of knowledge.

That space does not appear to be as heavily shaded as to prevent the growth of many shade loving plants. I'd suspect the lack of growth or establishment of other plants is due to excessive root competition from the large trees. These hog soil nutrients and any available soil moisture, starving smaller plants. Removal of the trees should open things up considerably.

I love bamboo but if you order it from out of the country there can be problems I've heard. It might end up in quarantine or something like that.

Not sure about all that cooperdr_gw, some folks on the bamboo forum can help you get a small clump of a not so rare variety of bamboo if you ask nicely. However if you are after a rare type of bamboo, you may need to search a bit. I only plant the clumping type, myself. Bamboo, it's not for heavy shade, but can take a good bit of shade.

The area of the original picture seems quite small. If that is all we're talking about, some hostas and a non aggressive groundcover (Geranium maccrorhizum, for example, a lovely groundcover for shade--even dry shade), would be sufficient. Perhaps the ferns were struggling because it's too dry for them--very likely. Most Epimediums would also do well, and there are dozens and dozens of handsome cultivars. Native spring ephemerals would also work--trilliums, dodacatheons, etc.

I just reread this whole thing, the OP may have solved their problem, but, I wonder what made the originally thriving ferns and Gardenia suddenly not do so well? I'm wondering of TOO much fall cleanup by a new homeowner could've done it. This may sound strange, but, the fall leaves and woodsy soil making duff is what ferns and the like love to grow in. I guess 1 time of ridding of the stuff should'nt make a great deal of difference, well, they do need a mulch to stay moist, unless the soil stays moist from a rainy climate. I'm thinking Va being coastal zone 8 should get some decent rain. Well, anyways, I am glad to see the suggestions above by Lode above, I am currently doing all I can to make a small place as inviting to woodsy soil loving plants as I can. I tried just planting there in the shady pockets, but, my soil is too neutral or something, not moisture holding enough despite mulching. So I am using peat moss, rotting wood from tree stumps, free shredded trees using soil acidifier, old potting soil, fall leaves and needles, bunny poo all that stuff and just letting it all rot, and I'm hoping it will be good enough soon, likely a couple years. I'll likely add a couple plants sooner, even this spring. I will have to use artificial shade though, because the oak tree is too small. I had it right in there, but, I moved it because it was too close to another tree. It soon will shade the area, and another area. I'll use poles and shade cloth in the mean time. Sorry for going on about my stuff, but, woodsy plants like woodsy soil, so that may help if some plants aren't happy, some of the bagged "garden soils" have woody based soil conditioners in them that may help with growing some finicky shade plants, but, of course, many aren't finicky at all, and those bagged soils are expensive. I think it's Earthgrow, or some company, that even sells soil conditioner in a bag, but, all it is is ground up wood fines. You may be able to get the stuff free if your area has a compost site where they take the ground up wood from when utility trucks trim trees under the lines. Okay, Later.

After all I wrote above.... try mulching really well first, the soil may be good enough......

"I love bamboo but if you order it from out of the country there can be problems"

Why would you ever need to order from out of the country? A huge variety of bamboo - both running and clumping forms - are readily available here in local nurseries or by mail order from specialty bamboo nurseries.