Should you worry? Probably not, but we don't have quite enough info to be able to give you a good opinion of what's happening. In general, citrus have about a 15 to 24 month leaf drop cycle. The leaves that the tree drops will turn yellow, then fall off. This usually happens with more mature trees. Also, there is always some smaller degree of leave drop in the spring and fall, with flushes, so you'll see older (and usually lower) leaves turn solid yellow, then drop as the new flush (which starts up higher, and then can progress to the lower branches at the nodes) starts to push out. This also usually coincides with the beginning of blossom formation. The Univ. of Arizona Coop Extension has a really nice explanation of some of the other causes of leaf drop that I like to quote. I think this is the best, most concise explanation that I've found:

Citrus Leaf Drop: Leaves turn yellow and fall off

CAUSE: Over fertilization

MANAGEMENT: Prevent by following fertilizer application instructions on the product. Treat by watering heavily to flush the fertilizer below the root zone.

CAUSE: Inappropriate watering strategy, too much, or not enough

MANAGEMENT: As a general rule citrus need a deep watering once a month in the winter and once a week in the summer. Apply water to the entire root zone, well out beyond the drip line. Apply enough water to wet the soil at least a foot below the surface. Check the depth of the irrigation with a long handled screw driver, you will only be able to push the screw driver as deep as the water has gone. (My comment: This can happen by having drenching rainfalls one after the other, and not always due to external watering practices. AND, the recommendation to check moisture levels is excellent. I use one of those cheap metal plant stakes that are covered with green plastic you can get at Lowes/Home Depot and stick it into the ground. I should be able to push it down about 18" or so. Perfect. The deep watering concept is now considered passé. You're wasting water, letting water sink below the feeder roots.)

CAUSE: Citrus Mesophyll Collapse: Grapefruits are particularly susceptible but any citrus may be affected. It is not uncommon for grapefruit trees to be affected and other nearby citrus to be fine. Leaves and fruit of citrus trees dry up and drop. Stems and branches remain green and living. Rarely is the whole tree affected. Mesophyll collapse occurs when there are unseasonable weather changes. For example if it gets abnormally cold in October for several days and then heats back up above average temperature in November. This fluctuation stresses the trees as they kick into and out of and then back into their winter slow down. It is especially stressful after a dry summer.

MANAGEMENT: Remove fallen fruit to prevent attracting fungal and bacterial diseases. Continue to water the tree on its normal cycle and it will leaf-out again in the spring. Do not prune out the defoliated branches until spring, and then only after the tree has completely leafed out.

I can tell you've I've caused all 3 of these situations myself :-) And, I had a grapefruit (my Rio Red) has the catastrophic Citrus Mesophyll Collapse that nearly had my hysterical. Exactly as described. The grapefruit was nearly denuded, while all other citrus right next to it were just fine. It is recovering nicely :-)

If you post photos of your trees it helps a lot. Just upload your photos to Photobucket. Then, copy and paste the HTML coding string right into the body of your message and your photos are embedded directly into your message. Very handy for us on the forum :-)

Patty S.