Sand soil particles are the largest of the soil particles and have large spaces that will allow both moisture and nutrients to flow through that soil without feeding the plants attempting to grow there. There is nothing coarser then sand, although sand particles can and do vary in size some.
Silt is the next largest of the soil particles, although quite a bit smaller in size then sand. Silt is most common around river beds, or where rivers once flowed.
The single best thing anyone can do for the soil they have is to add organic matter and organic matter includes compost and other forms of vegetative waste. Organic matter, if enough is added to sandy soils, will fill in the pore spaces that allow moisture and nutrients to flow through and will hold them in the rhizosphere, plant root zone, so plants can utilize them.
Gravel is simply a mix of sand and small stones, and is not something you would want to add to the soil you have.
To get an idea of what you currently have these simple soil tests could be of some help,
1) Soil test for organic matter. From that soil sample put enough of the rest to make a 4 inch level in a clear 1 quart jar, with a tight fitting lid. Fill that jar with water and replace the lid, tightly. Shake the jar vigorously and then let it stand for 24 hours. Your soil will settle out according to soil particle size and weight. For example, a good loam will have about 1-3/4 inch (about 45%) of sand on the bottom. about 1 inch (about 25%) of silt next, about 1 inch (25%) of clay above that, and about 1/4 inch (about 5%) of organic matter on the top.

2) Drainage. Dig a hole 1 foot square and 1 foot deep and fill that with water. After that water drains away refill the hole with more water and time how long it takes that to drain away. Anything less than 2 hours and your soil drainsâ too quickly and needs more organic matter to slow that drainage down. Anything over 6 hours and the soil drains too slowly and needs lots of organic matter to speed it up.

3) Tilth. Take a handful of your slightly damp soil and squeeze it tightly. When the pressure is released the soil should hold together in that clump, but when poked with a finger that clump should fall apart.

4) Smell. What does your soil smell like? A pleasant, rich earthy odor? Putrid, offensive, repugnant odor? The more organic matter in your soil the more active the soil bacteria will be and the nicer your soil will smell, to9 a point. Too much organic matter can be bad as well.

5) Life. How many earthworms per shovel full were there? 5 or more indicates a pretty healthy soil. Fewer than 5, according to the Natural Resources Conservation Service, indicates a soil that is not healthy.

It's very easy. Plant things that are NATIVE to your area. They can cope with the local dirt.

I have about 1/4 acre of plants thriving in unamended AZ desert dirt, because they are all AZ desert plants, or from very similar areas. After I hacked out the planting holes with a pickaxe and planted the shrubs, I mulched heavily with wood chips and mesquite "duff" ... since then we've left most of the fallen leaves to decompose and to give the native birds something to peck through looking for seeds and bugs.

I tend it in cool weather, pruning off dead stuff and getting things under control, and taking care of storm damage as needed, but it's a very low-maintenance area.

Sometimes really deserve my screen name ...

Vegetable or flower garden? You still need to inject OM in the soil if you want to improve it. I would even consider hugelkultur.

Thanks LazyGardens. That is my plan now, mostly natives and plants that are notorious for liking crappy soil. The OM is super low in this soil, pretty much the only OM is the silt. No worms.

The site I have in mind (flower garden) wouldn't work for hugelkultur and I don't want to add too much OM because I want to plant things that like the crappy soil. BUT, the issue is the tilth. It gets such a huge crust on it after it dries and the particles (even though they are sand, they are VERY fine) are so small that I worry that the drainage isn't actually all that good and (more importantly) the soil won't be good for root growth. That is why I was thinking adding in some horticultural grit or something to break up the tiny particles and ensure good air space for root growth. Thanks!

Silt is not organic matter, silt is part of the mineral component of soil.
To have good tilth a soil needs a mixture of the mineral particles (sand, silt, clay) and organic matter. Organic matter helps bind soil (sand and silt) particles together and helps hold moisture and nutrients in that soil. Organic matter in soils that are mostly clay helps separate those soil particles so moisture, nutrients, and plant roots can move around more freely and find each other.

Here is a link that might be useful: soil particle size

To minimize the crust, apply a layer of mulch - wood chips or something.

I'm wondering if the OP has a lot of calcium in the soil. As an experiment, I would try applying Epsom Salts to a small area to see what that does to the crust.

According to the University of Wisconsin Epsom Salts will not help Wisconsin soils much, if at all.

I was going to suggest consulting a local master gardeners group, Ag Extension, nursery, neighbors, etc. for advice on how to deal with the local soil.

I like the mulch suggestion. High levels of organic matter (adding compost over time) will also improve tilth. Your idea of changing the particle size ratio by adding coarser material was something I thought of as well. If you can find it. I was just at a location on the banks of the Mississippi River here in MO yesterday where a huge deposit of gravelly sand was found, up to kitty litter size. I can mail you a truckload. Just kidding.