Yes, I would try cutting it back a bit.

Thank you.
You mean to cut away the ones that don't look very healthy? Almost all of the branches have dead flowers, so I wouldn't want to cut them all away, especially since they do have new growth at the tips. But it's just a few that don't look very healthy. My plant isn't very big yet, so there wouldn't be much left if I cut them ALL back.

I would just leave it alone. Once the flowers have completely dried out, you can very lightly rub your fingers along the stem there and they will fall off. I have a lot of Tuscan Blue rosemary which does very well around here. The flowers are very lovely, but they do dry into little grey-brown crud. The stems harden off to a grey bark. As long as you have new growth above the flowers, I think the plant is doing okay.

Here's something to look forward to: In a couple of years, you will have some lanky, woody stems on your rosemary. Cut off a few of them, strip off the needles from the middle, soak them in water for an hour or so, then use them as skewers to grill on the barbecue. If you are a meat-eater, this make the most incredible grilled lamb in the world (brush a little olive oil and balsamic vinegar on the lamb).

Thank you for your input. I think I will just wait a while and see how it does. I may have to move it in fall, anyway, because I'm afraid the spot isn't big enough for it.