Mine did that and I think it was caused by being too dry. I see it's in a pot. Don't drown it but you might try more water. Then again I could be all wrong. My tree in CA was a good bearing tree and it got lots of water.

Thanks fruitnut.It felt fairly moist down a few inches.They're going to get fertilized soon,so I'll water then.
I guess I can't rule out a bird or squirrel,but they're not taking the fruit. Brady

Mine is in a pot and is doing that too, even to fruits turning purple. Did you ever find a fix?

It took a couple years,but now the fruit is ripening.Yes,they fall off easily. Brady