Is there any webbing? Stippling can indicate spider mites. The underside of one of the yellow leaves looked like it has tiny black dots, which could be the spider mites. Try neem oil, but not if it's 85º or above.

The white speckled leaves are a good indication that they are mites of some kind. "Brown and crumbly" sounds a lot like russet mites.

Thank you both! I haven't noticed any webbing (aside from a stray spider that is catching bugs on my tomato cage). It does get up into the 90s during the afternoon, although some days are a bit cooler. Can I still use the neem oil or is there something else I can use to rid my poor little plant of them?

Prune off the dead and dying leaves and seal them in a plastic bag and throw them away. The companion plants in the pot are problematic because mites can hide out there and move back onto the tomato. If you have other herb plants elsewhere or can buy replacements, I'd actually remove them. Because digging them out would damage the already seriously stressed tomato plants' roots, cut them down at the base at the soil line. Wash the plant thoroughly with a vigorous stream of water, paying special attention to the undersides of leaves. This will knock off some of the mites and reduce the population a bit to give your plant time for treatment to work.

There are a number of treatment options, and you may need to try more than one, as mites reproduce so quickly that they've developed resistance to different treatments in different regions. If you search on the forum, you will find some reports of different things people have tried that work for them. Neem is a good place to start if you're trying to stay organic.

Spray every surface of the plant in the evening, when it's cooler and the sun is weak. I would drench the soil with the treatment, too. You need to repeat regularly (I think it's every 5 days in warm weather for russet mites) because you need to get the new generations after they hatch from eggs, which treatments may not kill. Keep checking the plant, especially the undersides of leaves, with a hand lens (mites are too small to be seen with the naked eye) to monitor whether or not their numbers are increasing again and confirm the species of mite so you can look up its reproductive schedule and time your spraying accordingly.

If you have other tomato plants, spray them all now and consider bagging and trashing this plant to prevent spread.

Good luck, and I hope you can save your plant.

I am 80% certain that you have mites. If you have at least 20-20 vision, you can see them moving on the undersides of the leaves. Otherwise, you need a magnifying glass. At this stage, there is little that you can do except prolong the agony. Untreated, your plant will be dead in a few weeks.