You'll have to give us an idea as to what herbs you are talking about specifically. The term "herbs" covers literally thousands of plants with different plants requiring different care and conditions.

You'll probably also want to at least give your zone as well. "Pretty cold" means different things to different readers.

FataMorgana

I'm in zone 5 and I was thinking about Rosemary and parsley specifically.

Rosemary can be a nice container plant. You'll have to bring it indoors for the winter since I live in the edge of what you can make the hardiest rosemary varieties survive. I have never successfully overwintered a rosemary indoors. But I'm sure others have and they can share their experiences with you.

Parsley is a biennial - 2 year life cycle. I usually consider it an annual as far as growing and harvesting purposes. I would start it outdoors (in your containers) in early spring.

FataMorgana

You'll definitely need to bring the rosemary in for the Winter, but you could probably just leave the parsley outside in its pot.

As fatamorgana said, parsley is a biennial, so it will go to seed next year. You could save these seeds and re-plant them, or let them fall back into the pot to re-seed themselves.

Overwintering rosemary indoors is hard, and I won't try to tell you otherwise. However it can be done. Let me share with you what I know about it (but I'm no expert, having only succeeded for the first time last year).

Rosemary is a tender perennial. It can handle cold weather, and even some freezing weather, but not super-cold weather. Different varieties are hardy to different zones, but zone 7 is pretty common for most of them. If you think you're up for the task, you could leave your rosemary outside whenever you expected the temps to stay above, say, 30F (or maybe even a bit lower), but bring it indoors if the temps are going to dip really low. Covering it for slightly colder weather might also be an option, especially if the pot was up against the house. I haven't tried this, in part because I don't trust myself to keep that much on top of the weather.

If you have an unheated breezeway, attached garage, or something like that, that would be a perfect place to keep your rosemary through the Winter. This would allow it to experience a "Winter" without getting too cold.

If you're like me, perhaps the only environment you have available is your home, kept at normal indoor temperatures. I think this makes it harder to overwinter, but not impossible. Keep it in the sunniest location you have available. Don't over-water it (rosemary prefers to dry out between waterings), but of course, don't forget to water it.

The other trick, for me, is remembering that rosemary can handle fairly cold temperatures, so leave it outside in the Fall as long as you can, and bring it back outside in the Spring as soon as possible. I think that this year, I only need to keep it going indoors for about 3 months, which is a whole lot easier than 6! When you do bring it back outside in the Spring, ease it out, like you would with "hardening off" seedlings in the Spring. First in shade for a couple of hours on a warm-ish (comparatively) day, then all day, then in the Sun, then spending longer amounts of time outdoors, until eventually it's outside all the time.

As mensioned, there are different varieties of rosemay.
In my zone (7b-8) it will over winter or so it has done in the past so far. This past week we have been under freezing mark almost every night and it has dippe to 15-18F few times. We sall see what will happen. Maybe rosemaries will be wiped out this year , maybe will survive
(keep my fingers crossed).

Parsley, in the garden (don't know about pots) will overwinter fine. I just put a light blanket of fall leaves over them. But starting parsley easy; just direct sow them outside very early spring. Parsley, onions and lettuce are the first ones to germinate for me. Then comes cilantro, radishes...
But I am too lazy to start rosemary, thyme, lavender, sage from seeds. I would just buy established plants.