is that just the nature of granite?

That is not the nature of all granites, but it is the nature of those granites that have a lot of mica in them. Many granites do not have mica in them and are mirror-smooth. Yours must be beautiful. Nothing you can do about it.

Is there any residue on the granite? I had a fireplace refaced with Taj Mahal quartzite and I had to have the fabricator back to remove glue and other residue that was left from the fabrication process.

I would try a flat bladed razor in an inconspicuous area, to see if there is any residue that comes up on the blade.

Maybe use a different type of cloth!

It is what it is. You're not going to seal your way out of this.