Inspect your mortar. Is it crumbling or otherwise deteriorating, or are there gaps that need to be fixed? If so, address those before you even think about shuffling through paint chips. You can probably seal minor holes yourself with spackle or caulk, but major damage means a call to your local mason.
Scrub, scrub and scrub some more. Paint never adheres perfectly to a dirty surface (and plus, you risk sealing in the odd cobweb, dead bug or piece of lint). Use a stiff-bristled brush and a little dish soap diluted in water to clean the brick thoroughly, then wipe it down with a dry cloth. Let the bricks dry completely before you start painting.
IMO it is a point that is far too often neglected by Interior Designers and Colour Experts alike when offering advice. ie: countertops be polished or honed, hardwood flooring be satin or semi-gloss finish etc.