If there's vivible sawdust then it needs to compost more. If it's pressure treated sawdust from years ago- then it may be contaminated with copper, chromium, and arsenic.

Holding moisture is a good thing, unless it's say in a hole in clay where it will act as a bathtub.

Start by contacting your local office of the University of Nebraska USDA Cooperative Extension Service and inquire about having a soil test done then dig in with these simple soil tests;
1) Structure. From that soil sample put enough of the rest to make a 4 inch level in a clear 1 quart jar, with a tight fitting lid. Fill that jar with water and replace the lid, tightly. Shake the jar vigorously and then let it stand for 24 hours. Your soil will settle out according to soil particle size and weight. A good loam will have about 1-3/4 inch (about 45%) of sand on the bottom. about 1 inch (about 25%) of silt next, about 1 inch (25%) of clay above that, and about 1/4 inch (about 5%) of organic matter on the top.

2) Drainage. Dig a hole 1 foot square and 1 foot deep and fill that with water. After that water drains away refill the hole with more water and time how long it takes that to drain away. Anything less than 2 hours and your soil drains too quickly and needs more organic matter to slow that drainage down. Anything over 6 hours and the soil drains too slowly and needs lots of organic matter to speed it up.

3) Tilth. Take a handful of your slightly damp soil and squeeze it tightly. When the pressure is released the soil should hold together in that clump, but when poked with a finger that clump should fall apart.

4) Smell. What does your soil smell like? A pleasant, rich earthy odor? Putrid, offensive, repugnant odor? The more organic matter in your soil the more active the soil bacteria will be and the nicer you soil will smell.

5) Life. How many earthworms per shovel full were there? 5 or more indicates a pretty healthy soil. Fewer than 5, according to the Natural Resources Conservation Service, indicates a soil that is not healthy.

to learn more about that soil. Once you have the soil test results yo wil lhave a better idea of what needs to be done, if anything, to make this into a good, healthy soil.

Here is a link that might be useful: UN CES

Could well be that it's staying too wet and roots rot. Also toxic materials from woodworking wood are a possibility.

Any drastic change in soil texture will cause waterlogging, even when the surrounding soil drains well. Does not have to be clay to cause the flower pot syndrome. Water will not drain out of the wood chip area into the sandy soil until the wood chip area is totally saturated and that's bad for roots. That's why it's staying so damp there.

Also wood chips and sawdust are very low in nitrogen and are undecomposed, so not a good substrate for roots by themselves.

Try mixing some with some topsoil in a small area and see if you can get something to live in it, to test for toxics. If that works you can mix in topsoil in the area so you have a mix of mineral soil plus the woody material, and no drastic changes in texture from one to the other. Gradual changes won't back up drainage.

I've been reading here and there that earthworms are not native to north america and so do not necessarily appear in healthy soils that have been uncultivated. I know worms are not native here, in our recently glaciated soils. Have other people been seeing the same thing? If it's true it would mean that absence of worms does not mean anything about the health of the soil.

There are some earthworms native to north america, but there are large areas of the north that do not have native earthworms. I would guess that you're right. An absence of earthworms doesn't necessarily mean unhealthy soil.

In certain ecosystems ceratin earthworms are not good. Do not read the news articles the way they are written, that all earthworms are bad in any ecosystem. As usual most of the news articles do not truely represent what the researchers said.

Kimmsr's advice is good and detailed. Once you find out what you are working with you will have to decide what to do with it. If it's really toxic you might want to remove the stuff. You might also be able to find something to grow in the area to help remediate it. This is done on a large scale in remediating old mines, some of which contain large concentrations of very nasty stuff.
Also, while grubs in themselves are not something you are happy to see, you might be glad to at least know that whatever is there is not strong enough to quickly kill the grubs.