can you scrape them off.. or is it part of the leaf ...

Usually with scale insects you have to spray when they are at what is called the crawler stage, with the fully developed adult stage like you have noticed being beyond the point where they are vulnerable to insecticides. But try searching the web for combinations like "olive tree scale" and seeing what comes up.

@ken_adrian Adrian MI cold Z5 I can scrape them off but then they sometimes leave a white mark.

@nel5397 and @Embothrium oh noooooo! and thankyou, I will research further.

OR, should I just cut off all the infected branches?

Or easier, use Bayer Tree and Shrub systemic insecticide - it is not approved for food plants, but I don't think you are going to be harvesting olives anytime soon.

either treat it.. or get rid of it.. but dont butcher/torture it ... just hoping for the best ...

ken

This is scale insect. The worst insect to treat because no insecticide spray works on it except Dinetofuran. Spray it twice in 2 weeks interval.

It will kill the insect but the spots will still remain. But at least new leaves will not be affected.

You may also later use soil drench of Dinetofuran to prevent new infestation.