First determine if the previous coating was coal tar or modified asphalt because they are not compatible with each other and with many other waterproofing coatings. This may not be a good DIY project.

My box gutters were leaking at a couple corners when I bought the house in '87, so a few years later when the roof was redone, the roofer used a rubber membrane to line them. I haven't had leaks from them since then. It is a black rubber, which is all I can say about it--I don't think there was ice dam material back in '92, so I'm pretty sure it's nothing exotic--just make sure its well adhered.