Where do you live Boomer? I notice with mine they do best in wet spots that get full hot afternoon sun. Less water or less sun for me equaled less growth. I hardly ever fertilize, and if I do it's less than 1 handful every year but maybe that would make the difference for plants in slightly less than optimal conditions? Do you have any pictures from the first or second year vs the third or fourth year?

My backyard "soil" (LOL) is dreadful, but by fertilizing every now and then
and by making more light available to my banana plants (cutting back other nearby trees)
I am finally eating my own bananas, after nearly 2 years of struggling with them!

Bananas like lots of light, lots of water (preferably RAIN)
and lots of either nice rich soil -or human-supplied 'food' i.e. fertilizer.

Slip up on any one of this Holy Trinity for Bananas
and they'll not do well.

Read this: http://www.banana-plants.com/

I've had bananas a for long time in three houses. In south Florida I can say we do not have "soil" in a conventional sense. It's either sand, or shell rock unless it's been amended before planting.

That said, here are my observations on bananas in the ground down here.

Leaving them with only rainfall and no fertilizer stunts or even stops their growth and they have never fruited for me or any fruit was small and was aborted before maturing to where you can eat it under these conditions.

Watering regularly (2-3x a week) and mulching around them AND fertilizing them with a handful of food bi-monthly and
occasionally dumping ash from my chiminea on their soil a few times a year makes them grow like crazy and provide huge amounts of fruit.

I once read that bananas are "voracious" feeders. So I think that if you have poor soil, you should feed more often than someone with rainforest like soil :-)

My best fruiting ones get sunlight ~70% of the day. More sunlight = more water and mulch to keep moisture in.