I like it -- are you open to adding a rug -- that could be used to tie the colours together. If anything, I might paint the shelves of the built in a colour that more closely matches the tile, so they fade into the background. Then your items would be the focus vs. the built in.

Maybe consider removing the top piece of trim on each bookcase side, put in a false build up and have the trim at the ceiling (so you don't have the hole above each bookcase). Bonus, you could add a light in the underside of the fake area. The goal is to make the whole wall act as one unit, rather than fireplace + bookcase on either side.

Then paint all the cabinets etc in a taupe/sandstone-ish color pulled from the fireplace tile.

All White:

It’s such a personal decision and hope this helps. Dark grey may overwhelm room and darken as well

On the opposite end of the scale, white is used about 99% of the time. Could use a pure or creamy white

Suggest getting a rug and cushions to pull room together

Another idea is to use a very light grey or just the grey on the back of shelves and either choice could be a happy medium

a rug that ties in all of the room's colors, and colored accessories, will work. I also agree with Patricia - move the tv to one side, and bring the tops of the shelving to the ceiling.

I prefer your idea of painting the backs of the built ins the same grey color as the doors.