I clean up whenever I have time to and pretty much ignore all the rules. I have a large garden and if I waited til Spring to start cutting things back it would be mid Summer before I got everybody trimmed and pruned. I have never lost something because it was trimmed back and exposed to cold wet weather. I tend to lose things in the Spring when the weather swings from frosty to hot, back and forth. I've already started pruning my fruit trees which is a big no-no, but again, it's just me and one set of pruners so I have to start now in order to get done by mid April.