What did the builder say?

It wasn't his cabinet maker, we hired him, but he and his finish carpenter thought about adding mdf on the back and sides of the space to cover the gaps and then painting it the color of our walls. It's an idea but we know that it's going to end up being around $1000. And with the cost of building a house in these times, ugh, we just don't want to pay that.

We thought about shiplap but it's a Prairie Modern house and it just doesn't fit.

The guy that installed should give you a new shelf that fits. I wouldn't let him add the mdf...

If you knew that he "usually trims out" but you did not inform him that you did not like the look, the redo is probably on your shoulders

When the alcove isnt square and plumb, you will get gaps. That should be pretty obvious as the issue. No one is a miracle worker to be able to get a rectangle to fit into a parallellogram. Mud the wall out to the shelves, and call it good. Or pay for the shelves to be rebuilt oversized, and then cut into the drywall, and slip them into the cut. Then do a bit of drywall tweak at the cut.

Thanks for the comments. The cabinet maker is going to make shelves that fit properly. He measured before building the shelves so he knew what the walls were like. 

I asked his lead installer what he would do if it was his house. Problem solved.