It could be lack of nutrients. Container Citrus are heavy feeders. Yellowing can mean many things. Lack of nutrients and over watering/underwatering are the most common. You plant has set fruit recently so some leaf drop is normal. Use organic fertilizer because synthetic fertilizer will leave a heavy salt reside on top of the soil. I would also try using some cheleated nutrients like in the picture below. Dilute it in a spray bottle and mist the leaves every few weeks see if that helps. Also buy a clay pot they allow the roots to get air citrus love that. Hope it helps.