It would be helpful to have a bit more info--where you live, what type of exposure, etc. However, to me I think the outlook is not good. Looks like root rot--common issue with rosemary. Often, by the time the symptoms show, it's too late to save the plant. You could pull it and check the roots, putting into a better draining soil, but it may already be too far gone.

Odd, the leaves look more like they have powdery mildew then brown/rust. Too much moisture it sounds like, either way.