These are very slow growers so it will take some time to get very tall until established. They also form very thick trunks which can eventually support the mass without too much ancillary support. They have rootlets similar to ivy hold-fasts that will attach to the brick but I don't think it would be as damaging as an ivy since the hydrangea can support itself better. How hard to pull down will probably depend on how long it is up there. I have seen some that had some pretty thick stems and trunks and needed to be sawed down and cut back. I definitely wouldn't call it an invasive.

Will Climbing hydrangea stick to vinyl siding? I'll keep my fingers crossed.

Thanks everybody, you've helped me reach my decision. Since it is a slow grower I think I'm going to go ahead and plant it. I was just afraid it was going to be as bad as ivy and that I would come to regret planting it.

Hi, KM. I've had climbing hydrangea on a large (10x30) front-facing brick chimney for years. It doesn't seem to be destructive of the chimney, and it is a fairly slow grower. however, extremely beautiful as it ages. Care for it and manage its growth. It will be something to talk about. Not invasive, but watch the ground at the edges for ancillary growth, and keep them down.