When did you first notice it? Did you fertilize in the fall and if so, with what? What is your soil like (sandy, clay, ect) ? When you dig in your gardens, do you see earthworms? Did you mulch in your fall leaves or sprinkle used coffee grounds?

What's the problem?

After reading PKponder's comment I'm wondering if you are seeing earthworm castings, Click on the green and scroll around and see if any of the pictures look like your situation. Earthworm castings are a good thing. If they are in little mounds just rake them flat.