If your scraper has cutter knives (scarifiers), use them to loosen the material before you attempt to actually scrape and drag material.
Extend the scarifiers down as far as they will go, but don't try to dig that deep with them. Try to dig only an inch or so deep until you break up the soil/gravel and locate any buried rocks.
Once you have worked up some loose material, you can then retract the scarifiers and begin "scraping & dragging" the loose stuff to fill in your pot holes and wallows.
Don't think of the box scraper like it is a bulldozer blade, think of it more like a "big broom" to "sweep" loose material from one spot to another.
And if you do have some chronic pot holes or mud wallows to deal with, take this advice.
You need to try and DIG DEEP down into the bottom of such places to (hopefully) provide drainage so water will not stand in these holes.
Standing water in the holes results in your gravel getting flushed out of the hole when a vehicle wheel drives across it.
Try to break up the layer of mud and silt that seals the pot hole bottom to help stop the vicious cycle of pot hole return after grading. Try to put as much clean gravel (no dirt) into the bottoms of your pot holes and wallows as you can.
Pot holes and wallows can often be worked up by extending only 1 or 2 of the scarifiers knives at a time and moving the box over just an inch or so with each successive pass.
You might then use a spade or shovel to clean the loose junk out. Try not to reuse any of the muddy or soil laden gravel from the pot hole to refill a pot hole. Pitch it off to a spot where you can snag it with the box scraper and drag it/spread it out where there is not an actual pot hole.
If your tractor has a hydrostatic transaxle, make sure you clean off any grass or leaf debris from the top of the trans case to improve cooling, this type of work cause very much heat to be generated inside the trans.
Go slow with your ground speed but keep your engine RPM at governed maximum for best use of power and for heat shedding ability (of both engine and trans) of the machine.