The plant does not care so if it bothers you then deadhead the spent blooms away. Of course, the ones are the top are always a pain. :o)

I let Mother Nature do the work for me, most of the time. The spent blooms will eventually fall down by themselves. But if there is a spent bloom left hanging that reeeeally bugs me then I cut the string that connects the spent flower to the stem.

The same thing happens when the leaves do not dry out and remain attached and alive during mild winters. I leave those on too in Spring. Then the new leaves open, get bigger and, eventually, cover the old leaves just as the old leaves finally dry out and begin to fall down on their own.