Overwatering of citrus is not uncommon but it does take a lot to damage an older tree such as the one you have. Overwatering symptoms are generally lower and inner leaves yellowing overall and falling soon thereafter.

Your citrus symptoms look more like iron chlorosis. Simply a deficiency of iron (most evident in summer, with "easing" in winter). It could also include a deficiency of other trace elements such as zinc, magnesium and manganese. Simply applying a complete "citrus food" will start greening up your little tree. A fast way to get results is to do a folliar spraying with liquid zinc-iron.

Citrus do like and need water -- during periods of active growth. Make sure you soak deeply when you do water.

Some good news: it's not "too big" for the site. Citrus will accept an incredible amount of pruning. You can turn yours into a beautiful topiary or espalier. And DEEP watering will help alleviate any roots pushing on the wall.

Joe

I mistakenly said I feed it nitrogen it was iron twice before. I didn't spray it on, but I will try that.
I have 10% iron cheleate by Growmore and says not to apply with fruit on tree so will have to wait. I am out of citrus food since it didn't seem to help or give any fruit to speak of I didn't buy any more.

Mulch. I know you have a neat and pretty spot there but put some attractive mulch down for this tree. It will help bring the soil in line with what citrus need. Also, there is a whole form here at Gardenweb devoted just to citrus and they have lots of experience with diagnosing all sorts of "pour yellow leaves" : ) Lots of good threads in the archives too.
Mulch will do wonders but not over night, you will see improvement over the course of a year.

I agree 100% with Joe. You will notice the new growth is more green while the old foliage turns yellow. This is typical for nitrogen deficiency. Buy a bag of good quality citrus fertilizer being sure it lists the minors Joe mentioned, scratch it into the surface following the recommendations on the bag,water it in well and mulch the surface preferable with compost about 2 inches deep. Al

I respectfully disagree with it being nitrogen deficiency as the new growth is bright green. Think it is a physiological problem having to do with changes in growing conditions, i.e. watering. Either going from a very wet condition to a dryer condition or vice versa.

I should add it doesn't get a lot of blossoms maybe 20 and only 3 fruit this time. They all just fall off. It was yellow when moving here so the same problem exists no matter what I do.

probably 80% of the problems I see are water related. So, I agree, it's probably getting to much or to little and most likely to much.
Still I say mulch. Mulch covers a multitude of problems. It helps cool the soil, increases microbial life, slowly feeds the full spectrum of nutrients, helps balance the soil PH, helps with tilth for good root activity and more. I would go with 4 inches or more. Start about 2 inches away from the base of the trunk and go out for 4 feet in each direction.
Without seeing the surrounding area, I'm guessing from the small retention wall that the soil was very poor subsoil to start with and from the picture, you really need to try to build some soil quality since it doesn't really have a lot of dirt to grow in. The soil also probably has a very high salt content, especially to much calcium, which is locking up the iron, zinc, magnesium, and other elements in the soil which is why it's deficient. Mulch will help bring the PH down so the calcium isn't doing as much harm and other nutrients become more readily available.
If you use store bought mulch use one that has been fortified with nutrients for the first time. (Next spring when you will need to top dress and add more, you can use just about anything.) If you use something like leaves or lawn trimmings now, make sure you add some nitrogen at first. Mulch will take up nitrogen a little at first while it starts decomposing, then will release it later as it finishes it's cycle.
I usually don't recommend chemical fertilizers, but this tree could use a little boost to get it going until the mulch has time to start working it's wonders. A simple watering with Miracle-Gro I think would help. Sprinkle it down through the mulch to "feed" the mulch too. The mulch will slow-release those good nutrients.
But mulch, really, it will work. It doesn't work overnight but in the spring I believe you will see some real improvement, more blossom and more fruit.
All this to say the long way what I said at first... mulch. It's simple and it simply works.

To say it another way, the underlying problem here is soil. Without a soil test I'm guessing that the PH is over 8.0. Mulch is the simplest way and the most effective way to get soil on the road to being healthy. Healthy soil, healthy tree. Soil is the foundation, build it, grow on it.
good luck, let us know what you try and how it's working.

I think I do have a sinple soil tester. It's gonna be difficult to give it less water being the neighbors water twice a day. They don't have much and haven't put in anything so just watering dirt and wasting water. I even change the watering on my roses in front due to over spray. I think they had their gardener put in new heads.
I put in new plants to the left of it and are drought tolerant. The lemon is getting close to the patio cover and I don't want critters on the roof. Much quieter nights since the apple is gone. Darn raccoons!

Okay, I've rechecked the picture. There are several problems occuring. Yellow leaves: The yellow leaves appear to all be older leaves which do not adjust to change very well (the older I get I don't either). The new growth is big, lush & green. So fertilizer is not the problem. There is a tendency to apply fertilizers as a cure to all plant problems, especially if yellow leaves are present. This is a water problem. If, as you say, the neighbors are watering frequently then some of that water is affecting this tree. The roots are now down below the wall foundation and spread out to the edges of the canopy of branches. When the builder put in the tall wall and the short retaining wall, he probably back-filled with whatever soil was availble. Lots of times it has abundant clay mixed with it, which can affect the drainage. The surface drys out quickly and the tendency to apply water is overwhelming. A deep watering twice a month is sufficient during hot weather. Otherwise monthly. The yellow leaves may fall off. Not a problem, new ones will come next spring. The new growth looking so lush indicates to me that the tree is growing well and is currently adjusting the older foliage by eventually dropping it. Mulching will help to hold soil moisture and keep it cooler. Also reduces the tendency to water a plant by hiding the surface. If you have a compost pile (all good gardeners should have a compost pile) then that compost can be used as a mulch. As long as your neighbors are watering as you say, you may never have to water this tree. Also check to see if the uprights for the hose spigots are not leaking below the soil surface. Enjoy the fruit next year.

I have tested the soil and is definately alkaline. The test was darker than the new green growth!
So maybe by closing the nearest sprinkler head earlier this year the tree is finally showing the affects and dropping the old leaves. I thought they would turn green.
The soil is clay as I have tested that before. Sometimes it's like digging into granite.

I was having sprinkler valve problems earlier this year, so nothing got watered for a week except manually so while fixing all those I made sure no leaks elsewhere.

I'll get more citrus food and definately check the numbers and some mulch.

Thanks for all your help!