If it were mine, I'd move it back into the smaller pot with a grittier mix -- more coarse builders sand (not playground or beach sand), less soil -- give it lots of sun and occasional fertilizer and just let it develop a good root system and grow. I wouldn't do any pruning right now. The leaves make food to help the plant grow, so it doesn't make sense to remove them when you're looking for growth. After a few seasons of strong growth you can start trimming/shaping the tree. Just my opinion...

Tl;dr smaller pot, grittier mix, fertilizer, sun, time, and patience.

Agree with Liz. That soil does not look great, especially in that large pot. You mention that there is perlite in there, but it needs to be at least 50% of the total mix. The large pot with that soil will hold water longer than you want. Which will evaporate and be empty first - a large bowl of water or a small one? Same thing.

Mostly, it takes patience with proper conditions.

These plants are in 4 inch pots -and lazy me, I haven't pruned either yet.

3.5 years old -

2 years old -

Your plant is not looking unhealthy, so don't get the wrong idea from us. Liz told you the optimum conditions for good growth. Good luck!