For a shrub, how about Rosemary?

I already have one of those on the NORTH side of the house and let me tell you it is HUGE. The cable guy came out and asked my husband "is that Rosemary" and my husband said sure is and the guy couldn't believe it. Said he had never seen one so big. I have actually seen larger in Texas. This thing is truly huge. I am looking for something that fruits or has nice smelly flowers. I was also considering planting the Loquat on this side of the house just for the fragrant flowers and it is also evergreen.

Juniper for seasoning, or perhaps yaupon. Yaupon is a broadleaf native in the FL panhandle and the only N.American plant that offers a caffeinated tea. I think its an evergreen as well, being in the holly family. Ignore the name of Ilex Vomitoria!

Here is a link that might be useful: yaupon

what about variegated or standard yucca (flower stalk like asparagus)
or a bamboo (probably want to contain its spread.) There's a nice, low variegated one.
There might be too much sun for Mahonia aquifolium. I think OSU has some in full sun, though.

I have 2 pineapple guavas next to my shed. I have had them about 3 years. Get blooms but no fruit yet. They are beautiful plants.

I'd call in person and ask the staff at Raintree Nursery (also online). Almost all of their stuff is edible landscape. You can probably use their search tool to narrow down evergreen edible for your zone. (I'd rather just ask, lol). I've bought three seasons worth of things from them (I keep adding) and they're dependable and very nice, good horticulture people.

Depending on what size you are looking for there is the strawberry tree, an evergreen dogwood, and Bay trees which can be containerized or pruned smallish.
For something bushy but smaller there are the Elaeangus; ebbingei and pungens are evergreen, tea plants, a few huckleberries, and barberry.
For something smaller there are many ground covers with varying degress of edibility, like the official Heather (tea and medicinal), wintergreen, kinnikinnik, cranberries, Salal, Oregon grape, and several herbs like sage, thyme, and lavender.
Hope this helps, researching edibles is nearly all I have been doing in the last few weeks, or has it been months now...yeesh!
For a reference online I have been using the Plants for a Future Database (you can google it) to help determine edibility and cultivation etc.

Check out Burnt Ridge Nursery, and One Green World for these plants. I really like Burnt Ridge Nursery's prices, plants, service, and variety of edibles.
I have not ordered from One Green World yet but they are the only place I have found some of these plants listed and sold as edibles :)

If you can grow loquats and pinapple guavas, you might want to try some named culinary varieties. Pineapple guavas produce better fruit in part shade in hot climates, and some varieties are reportedly better in heat than others. Some need cross-pollination to bear fruit, so if your current bushes are both the same variety, this may be why you haven't had fruit. You can eat the petals from the blossoms, though.

Loquats should also be planted with a pollinator. I believe that culinary varieties are often grafted to quince rootstocks to dwarf the trees somewhat.

Don't know about the East, but in the West some varieties suggested for warmer areas include Big Jim, Champagne, Vista White and Tanaka. There are many more. See the link below. I have found on the web additional information about Loquats from Purdue University in the past.

Here is a link that might be useful: CFRG Fruit Facts