Mine does the same thing, so I hope you find the solution.

Andrew, I found this on the Victorian DPI site:

Summer fruit drop

Citrus normally shed large numbers of fruitlets shortly after blossoming in the spring and at early fruit set (pea size). However, it is also common for a sudden drop of small fruit (20 mm diameter) to take place in the summer, when warmer weather places stress on the tree. The problem .... may be related to lack of water at, or soon after, fruit set. Root diseases and lack of nitrogen or trace elements may also be responsible.

Are you feeding your plant anything other than thrive?