Cannot leave that loose material in the bottom. All of it must come out.

It looks like you are adding a wet deck in the above area, the chippings are being used as fill. This is not an uncommon way of doing this so long as rebar is doweled in using epoxy and an appropriate amount of gunite/concrete is applied over the fill material and rebar.

Those chippings are not approved fill material. It provides no structural support and does not have the same properties as crushed stone/gravel. Ask any structural engineer.

It's old plaster, which in itself is not a structural component, but merely a finish material.

Sounds to me like the contractor does not want to be responsible and pay for carting the material off site. So he'll just dump it in your pool.

The cost of gravel is minimal.

Those chippings are a layer of plasterer and portions of gunite that they are bonded to and are being used as a base from the look of the picture. Additional fill material would then likely be used before the rebar is tied and the gunite is shot. The original shell would carry the load transferred to it by doweling with epoxy the new rebar.