I've had similar bugs for the past couple of years. Although my research has been inconclusive, I think they are spider mites. I pulled a curled leaf apart on Tuesday and a bug ran out from under the webbing. Unfortunately, it ran off the edge of the leaf and I lost it in the grass below. I was trying to keep it for a photo op. Oh well. It was as you described, light grey (almost white), about a 1/4 inch (maybe a bit larger) in length.

I combat them with a two part approach. When it rains, I take the garden hose and spary the underside of the leaves with a sharp spray. Secondly, I spray with Neem oil. Spray in the evening when its cooler. Make sure you get the top and bottom of the leaves as well as the limbs/trunk down to the base. Apply once a week for several weeks. If it rains during the period you're spraying, reapply a day or two after the rain.

I'm going to try orange oil as a method of pest control on the entire yard to battle mosquitos and ants as well.

1/4 inch sounds too large for a mite--most mites are less than a millimeter in length. I believe several species of jumping spider and sac spiders make small nests by rolling up the edge of a leaf. The "colony" is probably just independent young spiders that haven't traveled far from the nest. Since spiders are useful predators, I would probably leave them alone.

I'm pretty sure these things are NOT spiders. There were literally thousands of them in the colony and no signs they were trying to leave. Plus the leaf would start to look as if they had been feeding on it. I'm pretty sure these things are a pest of some sort. I just wish I knew what the heck they were. If they are spider mites, they're mutant, industrial sized spider mites. Maybe there is a variety of spider mite that gets that big. I suppose it is possible. They are an arachnid, not an insect though - 8 legs.