How far up is it? If you can hit it with wasp spray, that should deter them. You can also put a homemade cap of sorts over the existing during nest-building season (assuming you won't be using your fireplace). The easiest thing is to just ignore them.

(Psst--don't use that husband afraid of heights excuse. You can get up there too! :P )

I fix a lot of things and I used to get on our one story roof all the time. This is a two story house and the pitch is steep there. Also, there must be a spot at the damper where they can enter. The problem was we had them flying in the house quite a bit last fall. We had two in the house today. He burnt another log tonight and only one came out. Perhaps there were only a few left in the nest? Idk, but I hope so.