Is the side that's dying the same side that has the patches of bark missing? If so, that's what's causing the dying leaves. It looks like the bark is missing all the way around one of those main branches (called 'girdling') which will kill the branch as no water or nutrients can get up to the leaves at that point. Like a tourniquet on an arm, the blood can't flow. (kind of)

If that's the case the question becomes, what caused the girdling? To me it looks possible like something gnawed away the bark. Are there critters in your area that could do that?

Scale won't cause that kind of damage.

Thanks for the reply @BarbJP CA 15-16/9B.

Yes its mainly the same side. Do you think it could it be ants? Have seen them on the tree. We have other citrus trees (lemon, lemonade etc.) around the property that don't have this problem :(

It sort of looks like it's buried in a pit.....is it planted too deeply?

Unlikely ants. They could have brought the scale but neither ants nor scale would have damaged the bark like that. I was thinking more like some kind of mammal like mice, squirrels or maybe deer.

I couldn't see where the trunk goes into the soil from any of those pics so I can't comment on what rhizo_1 may have noticed.