Ladybugs or a hard water spray with a hose can help although a miticide may be more useful when one has a large infestation. No, I would not remove the leaves, especially since the plants take a while to generate new leaves. I had this issue with azaleas and began to release beneficial insects (ladybugs, pirate buds, etc) early (the next year) and in other cases, spray early next year.

And I read between the lines of your message... the darn leaves looked ugly that year and irritated me whenever I saw the shrub. At worst, consider just taking off the worst ones.