Yes, you are off.

Far better to work with existing soil rather than replace it which will create new and different problems/challenges.

I also have clay soil. After adding 2 to 3 inches compost every year, I now have good stuff.

Lots of posts/threads about the topic on this forum.

Fertilizer - if it's granular stuff rather than in the form of compost - is highly soluble and should be added when the plants are growing.

I second the suggestion to amend soil rather than replace, but you know your soil and your situation better than anyone so it's up to you. I added silty soil and a lot of compost, so now I have raised beds of silty clay loam that is very fertile.

One thing you can do is, in the fall, layer on compostable materials and even include some layers of new soil if you want, as you build them up. In spring you can plant right in these. By fall or the following spring you can dig or till it all up together.

Read some of the clay threads here, there are a lot of ideas.

Oh, are you actually sure you need lime? Only add it if you have tested the soil and it's too acidic. If it's mildly acidic, say pH 6, it's close enough to the happy range for most plants. Besides, compost will tend to bring it toward neutral.

If you don't really need to raise the pH, gypsum can help break up clay soil without raising the pH.

Every where in Ontario I have been does not have a lot of clay in the soil it is clay soil.
Fall is the best time to prepare the garden for next season, but throwing some lime and fertilizer and tilling that in is not the best method of preparation.
A good reliable soil test for soil pH and major nutrients is the first step, and it is unfortunate that AgCanada does not do soil testing although they may be able to help you find a good lab. Then a good in depth look at your soil can be helpful and these simple soil tests may be of some use.
1) Soil test for organic matter. From that soil sample put enough of the rest to make a 4 inch level in a clear 1 quart jar, with a tight fitting lid. Fill that jar with water and replace the lid, tightly. Shake the jar vigorously and then let it stand for 24 hours. Your soil will settle out according to soil particle size and weight. For example, a good loam will have about 1-3/4 inch (about 45%) of sand on the bottom. about 1 inch (about 25%) of silt next, about 1 inch (25%) of clay above that, and about 1/4 inch (about 5%) of organic matter on the top.

2) Drainage. Dig a hole 1 foot square and 1 foot deep and fill that with water. After that water drains away refill the hole with more water and time how long it takes that to drain away. Anything less than 2 hours and your soil drainsâ too quickly and needs more organic matter to slow that drainage down. Anything over 6 hours and the soil drains too slowly and needs lots of organic matter to speed it up.

3) Tilth. Take a handful of your slightly damp soil and squeeze it tightly. When the pressure is released the soil should hold together in that clump, but when poked with a finger that clump should fall apart.

4) Smell. What does your soil smell like? A pleasant, rich earthy odor? Putrid, offensive, repugnant odor? The more organic matter in your soil the more active the soil bacteria will be and the nicer your soil will smell.

5) Life. How many earthworms per shovel full were there? 5 or more indicates a pretty healthy soil. Fewer than 5, according to the Natural Resources Conservation Service, indicates a soil that is not healthy.

What you see after these can help guide you in preparing the soil for next season.