In general, evergreen azaleas are not considered reliably hardy below zone 6. There are always exceptions and individual garden microclimates that can extend that range. But I would not like to bank on it :) Typically, deciduous azaleas are considered more reliable performers in colder zones. And there is a lot to choose from, many of which have a strong fragrance as well.

Personally, I would not like to risk the investment in plants which may or more likely may not survive a harsh winter and bloom well in favor of those which are known to take that in stride.

Anything else you would suggest that flowers but is evergreen aside from Azaleas?

That's a tough ask for your zone. Similar to the azaleas, zone 6 is the 'line in the sand' for most floriferous evergreen shrubs. If your soil is suitably acidic, kalmias (mountain laurel) and any of the PJM type rhododendrons should work.

There are evergreen shrubs that do not produce showy flowers that could provide year round interest - boxwoods, hollies, euonymus - and then you augment them with showier deciduous shrubs, like the azaleas. Or weigela, nine barks, roses, viburnums, hydrangeas, etc., etc.