First never use regular soil (dirt) in a container. With each watering the fines in the soil compact, drining out the the oxygen. Without proper root zone oxygen the tree will begin to have too many problems to mention all of them here. Container citrus require a fast draining, high aeration, lose texture mixture for the roots to grow in. I read above you used coffee grounds and bloom mean as your fertilizer. It is also very important to include all of the trace minerals as part of an ongoing nutrition regiment. Do what you wish, but I would recommend the use of a regular commercial fertilizer high in nitrogen, low in phosphorous and high in potassium. One that contains all the trace minerals. A good growing mixture contains course ingredients such as bark or coconut chips, plus peat moss and perhaps perlite. Just make sure what ever you use, it is capable to holding enough water, while yet supplying good aeration to the root system.

Thank you for the reply. I posted some photos as well if that is at all helpful. Here is the link below.

Here is a link that might be useful: sad lemon