Vines Botanical Gardens near the suburb of Grayson was nice the last time I visted there about 3 years ago.
Atlanta Botanical Gardens is a definite must-see.
I have not been there yet but Dunaway Gardens near Noonan is an UNFINISHED restoration of a 1920's Arts and Crafts Gardens that was built and run back then by a flamboyant actress, who also held a women's drama school there. [Sounds very intriguing, but I have learned the hard way not to get my hopes up on historic gardens restorations.]
Do not miss Callaway Gardens, though it is a good drive from Atlanta. It is best in the Spring during Azalea season, but the acreage was first set aside by Mrs. Callaway as one of the few natural habitats of the latest-blooming of all Azaleas, the Rhododendron prunifolium. I'm afraid it is still just a few weeks too late to see the R. prunifoliums in bloom, but still definitely do see Callaway. Especially their butterfly house is a year-round delight.
I wish I didn't have to say this, but the Georgia State Botanical Gardens in Athens really needs help. I was there just last year, and the Conservatory was nice, but the rest looked run-down. I love roses, and the rose garden is a huge disappointment - looked like it hadn't been touched, in literally years.
The best selection of unusual plant materials is Landarts near Monroe Georgia. I love that nursery. Jane Bath the owner is a famous Georgia plantswoman. And go by Social Circle, GA on your way to Monroe to get a buffet-style down-home southern meal in a beautiful neoclassical porticoed home (The Blue Willow Inn).

Drive out to Randy's Nursery and Watergardens in Lawrenceville.

I85N to exit 108 (GA Rt 120)
Go east 3 or 4 miles then turn south on US29.
Go about 1 mile. It's on the right, across from Central Gwinnett High School.

They carry bananas, tropicals, cold-hardy palms and desert plants.

If you want to see tropical exotics and bananas and stuff, you've coming to the wrong place. You'd probably want to go to Florida instead. Atlanta is a very much a temporate climate not subtropical. There are a limited number of bananas and palms that grow here and they are not all that uncommon to find. That's not to say there isn't anything to see, there's lots of things to see and do. Just not as tropical as you may be hoping for.

If you like roses, go to the Fernbank Museum. They have quite a rose garden there, with many test roses. And best of all, it's free. Just park in the museum parking lot and unless you want to go to the museum, you don't have to pay.

If you're in to trees, there's a sleepy arboretum in Milledgeville, GA:

Lockery Arboretum
www.lockerlyarboretum.org

Hi again. Well, thanks for the ideas. I visited Randy's place ... and spent a great time wandering around looking at the plants, the green tree frogs, lizards and chatting to the guys who work there. A great place to visit.

I also visited Land Art, and enjoyed home-grown orange watermelon with Jane. Her plants were less 'exotic' and more native, but we still had a good chat.

Also visited Pike Nurseries, and, of course, the Atlanta Botanical Garden. Excellent place to go.

Everyone I talked to was very friendly, and loved chatting.

Just got to figure out what to do now on Sunday before my flight leaves.

Again, thanks for the ideas guys!