First off, if your soil is clay and hard to work with I would build a raised bed and look into square foot gardening. If you don't mind the work and want to amend your soil, I would still use Mel's mix for the top eight inches of soil. The mix is one third vermiculite, one third peat or coir, and one third compost. Over time you will need to renew the compost, and VC is a great addition,but would be hard to use exclusively.

As far as adding granite, I would lean more towards rock dust that has a wide array of mineral content. A very popular one is azomite. There is also Gaia glacial rockdust and a couple of others available. Basically put, the soil has been depleted of it's mineral content, and people now try to add that back into the soil with powdered rocks. Not all rocks have the same components, and some only add one or two minerals like greensand. It takes very little to do a large raised bed.

To answer your question about worms just appearing, if you amended your soil or had a raised bed with healthy soil, yes there would eventually be worms and all sorts of living things to help your garden grow.

By just adding rock dust, I doubt that would be enough to completely revive the soil, but you could always make two plots of land. One amend and make a healthy soil, and the other just add crushed rock. Then plant a garden in both and see what happens. Mother nature is pretty resilient and might make it happen in both plots.