Where do you live? I hate to say it, but to me it looks like it could be HLB (Citrus Greening) see the USDA citrus greening quarantine map if you don't know if you are in a quarantine area (assuming you live in the US). https://www.aphis.usda.gov/aphis/ourfocus/planthealth/plant-pest-and-disease-programs/pests-and-diseases/citrus-health-response-program/ct_citrus_greening/

There are other possibilities, perhaps magnesium deficiency combined with biuret toxicity if you are using a urea based fertilizer.

Im in Tampa Florida... I have looked up citrus greening but I am not sure if thats it since the fruit really arent effected. I got 6 perfectly healthy lemons last year.

I am thinking a fungus problem. Spray with fungicide. If I am wrong it won't hurt your tree.

Steve

Do you have a particular fungicide that you recommend? I tried a copper based one that didnt seem to help.

That should work on fungus. The leaves won't repair them selves though

Try googling an iron and manganese deficiency, and see how it compares to what you have. Also, if it is a nutrient deficiency, you can narrow it down by determining whether old growth or new growth is affected first.

Ok will do and its definitely old growth that is more affected.

Hard to tell from the photo, but looks and sounds like scale insects, and the ants are feeding on the honeydew they produce.

If it's only on a small area you can wipe, or scrape and wash off the scale.

Otherwise you need a spraying oil to control this pest. And try and find and destroy any nearby ants nests.

Here's some useful info on scale:

Scale Insects on Citrus

If the tree is not blooming, I would use malathion on scale since it works on contact vs. 24-48 hours after an application of neem oil. Yes I am old school and sometimes old school is better than taking the organic route.

Ok ya definitely looks like scale thanks Ill look into malathion ... I have a whole host of pest problems on the tree and neem doest seem to help no matter how often I apply.