There's no problem with leaving them on the plant. The plant will know that the flower was not pollinated then will drop the pod on its own.

Sometimes my outdoor ones do get pollinated and start to form a seed pod, at which point I snip them off.

Exactly, and you will notice which ones those are. I tend to my plants daily and know which blooms opened and when they fell off. If you notice that the pod stays on the plant more than two or three days before turning yellow then deadhead.

Or if you are up to it nip it in the bud and just pinch them off just after the bloom dies. It only takes a second to do.

Thanks everybody - I'll check them every day - they're so gorgeous it's not hard to do!