It is do-able ... but will take several years of INTENSE amending with organic materials, cover cropping, etc. DOWN SIDE: time and work. UPSIDE: you will know exactly what your soil contains, unlike when you buy 'top soil'.

You will want to learn what kind of soil you are dealing with. What color/consistancy is the pile of soil? Is it decomposed granit? Clay? Sand? Full of stones?

Also,it would help to figure out the consequences of putting a planter in a spot that collects water in heavy rain. Where will the water flow go, when that place is filled in?

Borderbarb has lots of good suggestions. In addition, I would take some soil samples to your local extension office to get more information on the pH and mineral contents.

Take a good look at the soil you have to determine what it might need to become good garden soil. In many instances what is sold as "garden soil", sometimes they even label this as "topsoil", is just what you have with some organic matter added to it. That pile of soil you have from your excavation most likely will be a fairly decent garden soil if organic matter is added to it. Use these simple soil tests on that soil to see what you need to do to create a good garden soil,

1. Structure. From that soil sample put enough of the rest to make a 4 inch level in a clear 1 quart jar, with a tight fitting lid. Fill that jar with water and replace the lid, tightly. Shake the jar vigorously and then let it stand for 24 hours. Your soil will settle out according to soil particle size and weight. A good loam will have about 1-3/4 inch (about 45%) of sand on the bottom. about 1 inch (about 25%) of silt next, about 1 inch (25%) of clay above that, and about 1/4 inch (about 5%) of organic matter on the top.
2. Drainage. Dig a hole 1 foot square and 1 foot deep and fill that with water. After that water drains away refill the hole with more water and time how long it takes that to drain away. Anything less than 2 hours and your soil drains too quickly and needs more organic matter to slow that drainage down. Anything over 6 hours and the soil drains too slowly and needs lots of organic matter to speed it up.

3) Tilth. Take a handful of your slightly damp soil and squeeze it tightly. When the pressure is released the soil should hold together in that clump, but when poked with a finger that clump should fall apart.

4) Smell. What does your soil smell like? A pleasant, rich earthy odor? Putrid, offensive, repugnant odor? The more organic matter in your soil the more active the soil bacteria will be and the nicer your soil will smell.

5) Life. How many earthworms per shovel full were there? 5 or more indicates a pretty healthy soil. Fewer than 5, according to the Natural Resources Conservation Service, indicates a soil that is not healthy.