Did you ask your builder?

If the foundation walls weren't braced (also here) from the inside before backfilling and/or you don't have the first floor platform on, do one or the other before excavation and backfilling.

(I can already hear the builder: "We never do that and none of our houses have collapsed. Just some crap you read on the Internet.")

If the builder failed to request a required inspection, that's their mistake. Who pays for the mistake depends on the terms and conditions of your contract and your builder's sense of fairness.

As for the inspection, I would think removing fill in a couple of areas for visual inspection would be an acceptable alternative to completely removing the fill. Whether the code official will go for that depends on your builder's track record.

I feel it should be on the builders dime. Was it a mistake or maybe not a mistake. My hubby worked for a water authority. If a builder backfilled before inspection they would make them remove all the backfill. Yes, they caught illegally laid lines.

Here is what constitutes a foundation inspection in our current jurisdiction:

(Makes me wonder who paid whom when I see the two 8,000 sq.footers underway across the street from our project backfilled without lateral support.)

You don’t “forget” to have an inspection done.

"Should the builder pay for the labor to remove the fill?"

Yes. He's a dumba$$. And you have to pay when you're stupid.

Just to be clear. Generally, everyone pays for their own negligence in a contract, regardless of the terms of the contract. Even in a time and materials contract you are only required to pay for the necessary time and materials to complete the job and don’t have to pay for time to correct improperly performed work or buy replacement materials. Contracts regularly disclaim a duty, but not negligence. For example, your contract might disclaim responsibility for processes that fall outside norms.

So your contractor should be responsible for this problem.

Just as an example: My ceiling over my stairway wasn't to code (too short) and the builder had to eat the cost of fixing the ceiling to meet code.

You don’t “forget” to have an inspection done.

Is it his first day on the job?

Just as an example: My ceiling over my stairway wasn't to code (too short) and the builder had to eat the cost of fixing the ceiling to meet code.

And that's fair. You, the customer, are paying him to know code and meet it.

Did builder pull a building permit at all? You know he has and you have seen the permit. Because "at the designated points during the work, builder would call the inspection office to arrange for a visit to the worksite and review the quality of the work."

OP: "Now WE have to remove all the fill to have the inspection done."

Who is "we"?