If it's really lichen, it is harmless, but may indicate too little sun. Best way to remove is to just rub it off.

Actually, lichen often suggests too much sun :-) It tends to form most readily on wood that is exposed to a lot of sunlight. Typically it will tend to colonize plants that are already under stress for some reason or another or otherwise failing and in poor health. As a result, theses tend to develop a rather sparse canopy that allows more sunlight penetration, which the lichen love. Older fruit trees in my area tend to be very prone, also deciduous azaleas.

And lichen is harmless. Attempts to remove it can be damaging to bark so best to just leave it alone. The best control is to keep the supporting/host plant in as good a health as possible. Remember that the lichen is NOT the problem - only a symptom of a deeper, underlying problem.