Goof off or goo gone may work on that or a wire brush used with the grain if thats a broom finish. Hopefully you have paint to touch up area when you get it off.

I had a similar issue. I power washed the concrete steps (multiple times), used goo gone with a scrapper and a wire brush. The scrapper worked a little better, but I did also use a wire brush. Took a ton of time but I eventually got it off. Good luck!