I don't have an answer. This article is about domestic plum trees not fruiting and may provide some information. Also read here that there are ten species of wild plums in Texas. Mostly am bumping this thread up so it will be noticed and hopefully answered by someone with experience in growing them. I wish you the best in getting some fruit.

Could the birds/squirrels be getting the fruit?

No, the fruit forms but falls off while still very tiny.

That sounds like drought stress.