One of two things...

Try gently heating it with a heat gun. You don't want to get the heat gun close enough to bubble it, only enough to make it pliable so that you can scrape it off.

The other thing would be to freeze it with liquid nitrogen and then pop it off with a sharp wood chisel. Not recommended as you can really do some damage to yourself if you get that stuff on you.

I don't know if this tool would be of any help for this particular application, but if it doesn't work you're only out twenty bucks.

Do a google search for 'putty chaser' and you'll find a number of vendors.

Here is a link that might be useful: putty chaser

The first time the carbide in the putty chaser touches the brick it will shatter, throw shrapnel everywhere, and may be out of balance enough to bend the shaft.

Try a heat gun.

Did you end up finding a solution? Our windows were caulked very poorly and need to be redone, but I really worry about tools that will remove the caulk damaging the window cladding or the soft sand-faced brick.