Not sure what you mean by "clay," as I am only familiar with brick (which is kiln-fired clay) or concrete pavers. If they are not kiln-fired, they will be very brittle. The gravel is to provide a strong, compacted base that will allow water to drain. The sand, which also allows water to drain, is a bedding course.

Yes, kiln-fired clay pavers

What kind of dirt will the pavers cover? Gravel is cheap so why skip what could be a project ruining step.

My patio & walkways & driveway ( large ) are covered in pavers. The contractor just used sand, compacted by a machine of theirs, no issues at at all ... even though my cars obviously drive over them daily. I‘ve never seen gravel placed under them, but I suppose anything is possible.

" I‘ve never seen gravel placed under them, but I suppose anything is possible."

Interesting, as that is probably the standard practice :-) While one can lay pavers or flagstone on anything, including just digging out the lawn to fit the flags, for stability, permeability and long time endurance, the recommended method is to excavate 6-8 inches to a firm, level surface, lay 4-6 inches of a paver base (usually a crushed rock or fine gravel), compact it well, then lay at least 1' of sand, set your flags or pavers on to the sand base, level, firm and then fill the surrounding gaps with your material of choice.

You can short cut the process as you like but especially in areas that experience freeze-thaw cycles in winter, you risk instability and movement of the pavers or flags over time if you do not follow this standard method. Even in areas that are not subject to freeze-thaw cycles soil subsidence can create instability and movement of the pavers if you have not prepared a good solid base first.

My opinion? Don't take short cuts or skip any steps if you want a permanent, fixed-in-place, attractive walkway or patio.

The soil here in SW AZ is very sandy,but I don't know what it is called. It drains well. All of the online research that I have done speaks of using compacted gravel as a base with a layer of sand on top of that to ensure a stable paver walkway. We don't have a problem with frost here.