I have read articles that describe fixing problem areas with 1/4" washed gravel covered with top soil. The nursery where I have bought plants and trees has gravel and the last tree I bought had the roots growing right through the gravel. So I don't think I'd worry about the gravel preventing good root growth on grass much. If it were me, I'd kill the existing grass, rake off and level some of the gravel (you could use it in mulch beds), fill with 3-4" of topsoil and plant new grass. Some gravel will work its way to the top over time, but a good turf grass cover should minimize that.

Grass will grow a lot better in dirt than in rocks. Dig out as much as you can or have someone else do it and use top quality lawn soil.
The easier you make it for the grass to have deep roots the better it will perform. Especially when it is hot or dry.