There are probably hundreds of ground dwelling insects that they could be, so it may not be possible for anyone to give an accurate ID without a picture of the bugs themselves. I bet you could use a broom to sweep the sand/mortar back into the cracks, and might not even bother the bugs. There is a terrible decline in our pollinator populations, so anything we can do to encourage/protect them will benefit all of us. Farmers are seeing a decrease in the rate of pollination and then a decrease in yields of crops because of these declines. So, we are all urged to plant pollinator-friendly flowers and trees in our yards, and avoid use of any pesticides that might harm them unintentionally. There are even ways to provide nesting sites that can attract native bees that are efficient pollinators and also much more docile and less likely to sting, since they don't have a hive to protect. Look up "pollinators" and you'll find lots of info.

Martha

They look just like the ant hills that I see here. IME they don't cause damage, but you may find it a bit unsightly. If you sweep the dirt/sand/granite dust back between the cracks they will just start over.

I see very small wasps (I'm guessing that's what they are) going in and out. I have never seen an ant use them. The wasps are too fast for me to get a picture, although I will try again. I have swept the dirt back into the holes and they are very quickly opened up again.

Would wasps be going into ant holes to eat eggs or ants? I do not want to poison any pollinators.

I have the same thing. They are mason wasps.