Perhaps citrus rust mite?

Just a thought, if the fruit is good and the tree is otherwise fairly healthy, maybe you can live with a little leaf crinkle? It doesn't sound like the problem is really all that bad enough to use chemicals and such.
Are you using mulch around your tree?

"mulch around the tree" - yes. I rake it up late in the fall, compost and put new fresh in spring.

It grows next to an oleander, which is poisonous to eat, but have never heard that it affects other plants nearby. It doesn't bother the mandarin nor cut off sun.

I'll check out rust mite to see if that could be a factor. The leaf looks a lot like aphis, but I can't see evidence of it.

Thanks for replies.

Bejay

My money is on powdery mildew, which can look a lot like white fly and both can be present at the same time. This would explain how rinsing the tree actually helped -- crinkly (distorted or curly?) leaves would be another give away. I get PM on my citrus on occasion, but it never runs amok. Water spray and removing a few leaves seems to help. UC Davis has some pretty good info on PM (see link below). Not sure if the hard outer peel is related or not -- can you still get some decent zest from your peels?

Here is a link that might be useful: Quick Tips Series: Powdery Mildew