I didn't look up what zone you are in but people in MN and Canada grow hot peppers so I'm sure Vermont can as well. I start mine inside, put them under shop lights once they sprout and grow away. Don't plant them outside until the nights stay at 50 degrees for a week. They will then be fine whether you plant them in the ground or in a pot. I start super hots in Jan, most of the rest in Feb and Mar. yes that means I spend a LOT of time with my living room looking like a greenhouse but it gets me peppers!

I live in western Montana with only a 90 day growing season and average night time temperature of 47 degrees F during the summers. I have successfully grown peppers for 5 years. I usually start with transplants from the nursery, but this year I started from seed March 1st under lights in the house and looks like it will again be a successful year.

I have used multiple methods to grow my peppers raised gardens, containers, and a Harbor Freight greenhouse. All methods have been successful, but the peppers in my raised beds always do the best.

The key to growing peppers in cold climates is to keep your plants warm. I spend a lot of time covering my plants every night with tarps (the heavier types are better insulators  I use the ones with brown on one side and silver on the other, purchased at WalMart), or plastic sheeting from Lowe's or HD. These probably provide more protection than the thin white garbage bags you are using.

Early in the season I sometimes use (unfilled) aqua domes over the plants, but most often not because I don't have enough of them, so I just keep the plants inside or wheel them in and out of the garage in carts until it is about 60 degrees out. Once planted I always drape a tarp over them at night. I place round tomato cages around in my garden soil to support the tarps and allow for some air under them. I usually cover them in this way at night until July when night temps warm up to the 50s and 60s at night. Once it is this warm you really don't need to cover them anymore. Leaving the bags on unless it is 70 degrees is not advisable for the reasons you mentioned.

I am covering my plants again by mid-August when night temps drop again! By September our days stay cold until early afternoon, so my husband built wooden frames on top of the raised beds and we put clear plastic sheeting on 3 sides and the top, and hang a tarp on the 4th side that can be rolled up and bungeed during the warmer part of the day, as well as for access to harvesting and watering. During this time I continue to cover the plants themselves with an additional tarp in the evenings. It is a LOT of work, but with this method I grow about 200 pepper plants a year, even successfully grow habaneros. My frames are built with 2 x 4 posts in each corner, and 1 x 2s across the top and top sides.

I am still experimenting with peppers in my greenhouse. While it does help keep them warm without as much work as tarping, I find that controlling aphids in the greenhouse is difficult. The ladybugs help, but it's still a problem, and I don't like spraying the plants.

In addition to keeping your plants warm, it is important not to over-water peppers, particularly in areas with short growing seasons. Once spring turns to summer, cut back on watering and your plants will switch from growing vegetation to producing flowers and setting fruit. Your plants will begin to take up different nutrients from the soil that allow them to produce once they sense that conditions are drier. Over-watering in areas with short growing seasons means less time for your plants to complete their growth and production cycle.