This is partly a visual / aesthetic question and partly a building code question. Overall, presuming that you comply with the code requirements (height varies depending on the depth and the material) you should think about whether you prefer to look at the mantel and fireplace surround (in which case it should be on the higher side) or objects on or above the mantel (in which case it should be on the lower side.)