Depends what you're putting on top of the fill. Most building departments will require accepted data (either because it's known stuff like gravel or concrete or it's been tested if soil) for fill you're going to build over or perhaps even put a driveway/roadway surface on.

As for making a yard, you can usually use whatever you can get.

There may however, be regulations governing filling your lot as well.

This will be fill for going under a foundation of a new home (concrete slab).
Some of it may go towards the driveway but mostly concerned about the house foundation right now.

Be leery of accepting "free fill" offers. Make sure you know the source of the fill. If someone gives you "free" fill that is contaminated, you will end up paying a lot to clean it up or to have it hauled away and incinerated (or whatever the appropriate clean up disposal is).

I understand the risks, that's why I wanted to know exactly what types of material would be suitable so I don't go picking up junk.
Example: I just had someone mention that there was a guy looking to get rid of limestone. This is one type of material that I don't know much about.

The requirement for fill under a foundation is different from under a slab and different from backfill so you need to better describe your foundation. For the slab, granular fill should be graded so it can be compacted. An alternative that does not have to be compacted or leveled is "flowable fill" from the local concrete supplier.

It is always best to take a foundation footing down to undisturbed soil.

Ok, I'm going to describe the situation as best I can.

I have a flat piece of land that will get the top layer scraped off to have poured concrete walls that will support a concrete slab that the house will be built on.
To be over the flood zone height the poured walls will basically be starting on top of the ground. I need to bring in fill for under the slab. The slab is going to be ~5ft higher than the base ground.
The backfill around the house I'm not worried about.

For a floor 5 ft off the ground I would suggest a crawl space rather than a slab on grade if the flood regulations do not forbid it. Is there a storm surge requirement?

Mighty is right. A crawl space or even grade beams is probably going to be cheaper than any sort of fill if you've got more than a foot to worry about.

When filling under my hangar floor, gravel was cheaper (but still expensive) comparing moving fill from other areas of the property and having it tested.

this is not uncommon in my area. all houses are built on slab here, and we are on a flood plain. new construction requires that it be built up above the floodplain. not the lowest floor above, EVERYTHING above. all i can tell you is that you will be seeing lots and lots of dump trucks with dirt/gravel. guy a few hosues up built his house 4-5 years ago and he had to build the DIRT up 5'. the lot next to him has to build up 6.5'! they then dig the footers in the built up area, usually 18" or so, and pour the slab.

due to our flood zone crawlspaces are nit used here either anymore. it costs more to build up a house 5-6 feet above grade than it does to fill it in and put a slab.

the best advice i can give you is to call the local permit office and speak directly with the inspector to see what he wants for fill.