I wait until all the foliage turns brown and dies back and pulls away easily. In shadier areas that often exceed the 6 weeks trigger by a wide margin.

For me, area's where it's just clumps here and there I let it dieback. But this area we speak of, flanking Left/Right beds to the entrance.....I've always removed them at 6 weeks, and it's just this one particular cultivar that has just decided to not open up the last 2 years. I'm leaning on letting it go the full brown out, but by the looks of current foliage, it's not looking to give up the ghost anytime soon