I just mix in organic matter whether its compost, leaf mold, or soil conditioner. You don't have to worry about working it in too deep since azaleas are shallow rooted, just the 1st foot or so. I always plant mine slightly higher than the surrounding soil.

A gal from Post properties landscaping told me one day at Home Depot, that they mix one third compost/manure (cheap stuff, not the fancy Moo Nure) , one third Natures Helper, and one third original soil to plant anything. That was about 20 years ago, and I've been doing it successfully ever since. Killerv, above, is right to suggest planting a little high, since lack of drainage kills more plants than anyone suspects.
Bill

To me, two of the secrets of gardening in north Georgia is soil conditioner (Nature's Helper) and compose - usually mushroom compost.