Inspect your mortar. Is it crumbling or otherwise deteriorating, or are there gaps that need to be fixed? If so, address those before you even think about shuffling through paint chips. You can probably seal minor holes yourself with spackle or caulk, but major damage means a call to your local mason. Scrub, scrub and scrub some more. Paint never adheres perfectly to a dirty surface (and plus, you risk sealing in the odd cobweb, dead bug or piece of lint). Use a stiff-bristled brush and a little dish soap diluted in water to clean the brick thoroughly, then wipe it down with a dry cloth. Let the bricks dry completely before you start painting.