Sounds like some kind of subtropical fruit. Litchi are red or orange and the fruit is dimpled. Surinam cherry has dark red to black fruit with 8 ribs. Acerola cherry is bright red to yellow. The Capulin cherry is another possibility. Sweet or sour cherries common to the temperate zones don't have woody fruit.

The Fruitnut

Sounds like Cornus mas to me. Sometimes called Cornelian Cherry. They do well here in northern California. Al