The culprit is white cabbage moths...they lay eggs on brassicas (cabbage, brussels sprouts, kale, etc) and the subsequent caterpillars are responsible for those holes. You either need to grow these crops under cover or spray with BT, make sure it's the strain meant for caterpillars.

Those cabbage moths or "white butterflies" are quite abundant this year, but I've noticed the wasps are abundant as well and are doing an incredible job patrolling the brassicas for caterpillars. Get the BT (bacillus thuringiensis).

Oh, and check again for any caterpillars/worms on that plant. With that much damage, I'll bet you'll find one (or more) somewhere. Look on the back of the leaves - each one - and look very carefully because they'll blend in and will be easy to miss. Also, check the growing tip area as they like to shimmy down into there.

The problem of the cabbage loopers has been covered well already and you can Google them for more details and pics. But you might also want to Google a pic of what a BS plant is supposed to look like vs. your plants and what it needs to actually grow and produce sprouts. The plants in your pic will not be productive for several reasons besides the pest issue. Remove the one plant closest to the edge and you may get a few from the remaining one.

https://www.google.com/#q=Brussel+sprout+plants

Dave

Thank you everyone! This was a good experiment for me. I will do things differently next spring.