No. Grout should not be coming loose and breaking down after it has cured. The rock itself is not causing the issue with the grout. Maybe your grout was overworked as it was installed/cleaned. Are you able to easily scrape at it and get sand from it?

Thank you for responding! All I'm getting is sand! The drain keeps getting clogged. Maybe the tiler didn't seal the tile? Should he have?

Sealing is a good idea but that is not your problem. It sounds like something went very wrong with the grouting itself. It is going to be tedious but I think you need to get the grout carefully scraped out as much as possible..then regrout and seal once it's cured properly.