Looks like scale bugs

Yep scale.

Small worms sound like fungus gnat larvae, this is caused by over watering. That could also be why the leaves just pulled off. A good thing is the stem still looks healthy enough. Take it to your bathtub and put it in. Remove any dead leaves, this will make it easier for the next step. Spray the plant really well with a 50/50 mix of water and rubbing alcohol with a drop of liquid soap (not detergent), this will kill the scale. Be sure to get in between the leaves. You can then use a paper towel to wipe off the scale. Wait about 15 minutes and turn on your shower with warm water letting the water spray on the leaves then fill the tub. Submerge the pot completely in water and a couple of drops of liquid soap and the fungus gnat larvae will float to the top of the water, they can be scooped out then. Let the plant completely drain THEN stop watering too much. Only water when the soil is almost completely dry, as soon as the leaves start to droop water really well, letting the water drain off. Do not let it sit in water.

The "worms" don't sound like fungus gnat larvae to me, at all.