w/o seeing them it is hard to say but from your description, it doesn't sound good. Certainly the plants will be stunted. Perhaps covering them would have helped?

Start more seeds ;-)

Don't berate yourself--we all make the same mistake (I had my seedlings outside, hardening off, and had to go out in the rain and cold to bring them back in). Gardeners get antsy this time of year and often try to rush the season.
If it were me, I'd rip out the poorest looking plants, get replacements at the local nursery and put the new ones out in about 10 days, after you have had a chance to acclimate them to the outdoors. Some of the damaged ones will survive and set fruit. We just don't know how much or when!
Good luck with your garden.

Mersmom,
Last year I set out some tomato plants way to early because I started too many and ran out of room when I potted up to the final pots. Instead of throwing the "extras" away, I put them in the garden. We had several hard frosts after I set them out and the plants looked terrible. Actually, they looked dead. Even though they looked dead, they came back to life eventually. It did take some time but they produced very similarly to the plants that I put in later, though they were probably 2 weeks behind. Bottom line, if you have room, leave them and see what happens.
Rick

I'd leave them. Last year I did the same thing. Surprisingly, the tomatoes came back with a flourish. Water deeply and see what happens. I'd probably also buy a couple more from the nursery to pop into the garden elsewhere...just in case.