Here is some information for you about the Sweet Olive, Fragrant Tea Olive, Sweet Osmanthus. Also, I just wondered if you have considered planting the Texas Wild Olive? They are very pretty and native to Texas.

http://easttexasgardening.tamu.edu/tips/trees/sweetolive.html

http://bexar-tx.tamu.edu/HomeHort/F1Column/2007%20Articles/Plant%20of%20the%20Week/MAR3SweetOlive.htm

http://www.almostedenplants.com/shopping/shopexd.asp?id=667

I keep mine trimmed to around 5 feet. Have to trim a couple of times a year. They want to grow tall. I planted the first ones 4 feet apart. Second ones 3 feet apart cause I wanted them to fill in faster. Either will work. Can plant close to trees but they grow in sparsely in the shade. Great privacy fence in sun. I love them and they bloom often, filling the yard with fragrance!

If you have any more questions, ask away! I've got around 60 of them in various locations.

Yes. Full hot Texas sun. I read they liked part shade and so planted the first ones there. Then I wanted to make some garden "rooms" and asked my nurseryman about full sun. He said it would be fine, and they have been! One shrub that would do fine in part shade is the banana shrub, my favorite fragrant plant. They like to grow tall too, but can be kept trimmed. I've got lots and lots of those. Eleagnus grows full in any location but would require constant trimming. I've got them out in the woods in full shade around the property line. They grow well there- and hide my neighbor
I love fragrance in the garden and grow just about everything that is fragrant, trying things even if they aren't supposed to do well in Texas. Right now my garden is full of fragrance from wintersweet shrubs and winter honeysuckle, both which are perfect for Texas, growing happily through heat and cold and without ANY water other than rainfall. The Japanese apricots and witch hazels are blooming too, but their fragrance is more subtle.

Eleangnus is a “Dirty Dozen”
plant— https://www.lewisginter.org/elaeagnus/