Early spring, and yes it matters (a lot).

Get a couple of fruit-tree growing books - from the library through interlibrary loan or buy them - and read up on it. Go to the Fruit and Orchard Forum and look through old postings there, or ask. It IS important, and, while it probably won't ruin the tree for ever if badly done, it can impact the flowering/fruiting for the next year. It's too complicated to be dealt with in a written message, without diagrams, but it isn't rocket science, either. If your local nurseries, garden clubs, Master Gardener programs, etc. offer lectures come spring, go to them - examples are worth a thousand words. Maybe your local county extension agent can come show you, or can give a lecture on it (using your trees as the "test subjects"?).

The above information is good, however the ideal time is going to be a bit location dependent. Generally the recommendation for most fruit trees (there are exceptions) is to prune when the tree is fully dormant and before buds appear and begin to swell - late winter/very early spring. In my area, most fruit tree pruning is done in late January or February.

Fruit tree pruning is a bit of an art, as pruning techniques to encourage fruiting are quite different than those used for other, non-fruiting trees and shrubs. I agree that researching and educating yourself on the process before you begin is important. And hands-on pruning classes are great - you will learn far more from seeing and doing than just picking up tips in a book.

Late dormant season pruning is to promote tree health. One thins out dead branches, crossed limbs, canker or black knot. Summer pruning is to control tree height. Spring pruning is not recommended as the tree is in its weakest state. In the spring the tree is using stored energy for new growth and combat insects or disease.

Here is a link that might be useful: Pruning fruit trees