Often this happens with poor and incomplete pollination. If so, you might see few seeds inside the fruit. Can also happen if the temperatures were low (as in, below 60F) when the fruit was setting. My late season peppers often look kind of iffy. If the leaves look healthy, we're not talking about disease.

From my experience with bell peppers you had an excellent crop. Some years I got zero. Don't think I ever got 3 bell peppers in the same year. This year I grew Lipstick and Hungarian Wax and did better than I ever did with bell peppers.

I live in CA, used to live in an area where bell peppers were grown commercially, full sun, all day, 80-100 day time temps.

I recommend researching a plants needs before planting. Based on your results, bell peppers do not do well on your porch, in containers.

In CA, often the Sunset Climate guide is used to determine what will grow where. Sunset was a local magazine. Are you able to google your growing region, and then find plants suitable for your region?

My local newspapers have a garden section. Do you have a local newspaper?

Due to covid and CA shelter in place rules, I spent the summer at the new to me cabin I grew tomatoes, the tomatoes have no flavor compared to home grown tomatoes at home. Grow what grows on your porch, not something that will not thrive.

I've only grown mine in the dirt which is amended clay.

I believe bell peppers should do fine in containers. I've seen it done many times. The container does need to be large enough, and have good drainage, but drainage isn't your problem here.

Grow in full sun if possible. Use large enough pots with a container potting mix (not soil based). Fertilize throughout the season. And grow a variety recommended for northern gardens. Sweet non-bell types do better for me. Carmen is a favorite. Sweet Banana is reliable. King of the North and Better Belle Hybrid are bells that have done well for me in Michigan. They do best in warm/hot years. They do not like long cool springs. In upstate NY you will need to provide the best sun and warmest location possible for them.

Your plants do not look like they have dense foliage, and they may not have been vigorous enough to support more fruit. Therefore, any additional flowers would not set and would drop before developing fruit. More sun, heat, and fertilizer could help with growing larger plants next year which could produce a larger crop.

I've "grown" bell peppers, not intentionally but as volunteers from kitchen scraps buried in the garden bed. In sunny CA, they are vigorous and bore 4 fruits apiece but do need to be staked to prevent toppling. They do like some afternoon shade at the height of the summer, unlike the hot chillis which seem to take the heat better. Don't fertilize too much as it makes them leafy.

These are what the whole plants look like - they seem pretty healthy.