Hi, remove the moss - it's both competing for water with the plant and preventing it from drying when necessary. It's a sales gimmick - moss doesn't belong on olive trees, which grow in hot, dry climates. The soil should be sandy, gritty, fast draining, but 'sandy' doesn't mean anything from the beach (and NO salt), but you can use that tiny, rough edged gravel sold in aquarium stores. Potting soil (full of peat usually) is not a good idea either. Lots of sunlight all day long. Water when most of the soil is dry (or you see leaves just beginning to droop, etc., but don't keep the bulk of the soil soggy and wet all the time at all. Your yellow leaves are probably due to either watering way too often, and/or a plant that was kept in 'ideal' house plant conditions and is not yet used to full midday sun, but that seems less likely with an olive.