Anything with a worthwhile R value will add too much thickness and interfere with blinds etc. If it feels that cold remove the plaster and install rock wool, then Re plaster.

Is that area accessible at all through the drop ceiling tiles?

If not, as Hallet said, you won't be able to accomplish much on the interior side. You'd need to add 1"+ insulation board and then sheet rock over that. If it won't impede any soffit venting, the easiest thing would probably be to cut a few holes and dense pack that area with cellulose.