Is it possible you have one of the varigated varieties?

Last year, I planted a fatsia that was yellow and refused to grow--it looked like it was at death's door, frankly, for an entire year and I was ready to give it away. But then I gave it extra shade with one of those plastic footstools designed to match plastic Adirondack chairs, and it recovered beautifully. It's still not growing much, but after six weeks of shade this spring, it turned back to a normal green and I was able to remove the footstool. The weird thing is that if I understand your post, the fatsia getting the most sun in your garden was happiest. Huh. In my case, the sun definitely caused the yellowing. Huh.