If your place gets sharp frosts, it could be prudent to leave the dead foliage as protection for the buds below.

Are you growing anise hyssop? It usually self-seeds very liberally and it's easy to replace with a seedling.

You could take back the old flower stalks now but leave the base leaf clump. Later, in spring, those old stalks should come away quite readily. Meantime, take care when moving around because the stubs can cause scrapes on your hands if you garden without gloves.