Did you spray your trees and with what? The reason I ask is I had a simular problem and it was the product I was using to spray that was harming the trees (it must of went bad). After I stopped spraying the new leaves did not show the problem.

Much of the brown patches and spots look like a fungal disease to me. Have you had periods of rain lately? Spray for fungal disease. The holes look like insect damage.