Probably lack of cross-pollination.

How is the pollination factor?

Unpollinated fruit will begin to develop, but abort. It's the source of many small immature fruit you find at the foot of trees laying on the ground. I have a native persimmon tree, capable of self-pollination. This year, it started out with hundreds of potential fruits, but only a couple per branch are still developing. Normal.

Santa Rosa is a Japanese plum and they are self fertile. I don't know if planting another variety will help or not. I have a Methley plum, another Japanese variety. It took a few years for this gal to get with the program. Last year we had maybe five fruits survive to maturity. This year I have hundreds. We've been eating them as they ripen, and two days ago I just made ten jars of plum jam. There is still enough ripening to make another. You may just need to give it some more time.

Thanks for all the input..
I had another santarosa plum that did very well as far as quantity of fruit and not dropping off, it was great until my lanscaper mauled the entire girdle of the trunk and I could not save her.. I actually cried when she did not come back the following year.. so I don't think cross pollination is a factor as it was the only plum tree I had.
Maybe I just need to give it more time or maybe I can buy another one to plant next to it, maybe that might maker her happier...is there any sort of food or fertilzer that might help?
Thanks again everyone.

I think for a five year old Japanese plum, your tree is behaving pretty normally. Mine didn't start to produce heavily until this year, and there are no other plums in the vicinity, so I know they are self fertile. Mine is just a little older than yours. No, if it looks otherwise healthy, nothing you can dump on it will make it become more fertile until it's mature.