Every year my chamomile thrive in the same spot, just getting larger every year. It grows in rock hard crushed concert, in full, hot sun, but does get some run off from a potted plant that drains too much automatic water daily. Maybe you're babying it too much.

With such a healthy, growing patch, it's too bad that I hate the taste of chamomile.

Is your climate a hot one, as well as a dry one? I tried growing a chamomile lawn here in the subtropics where the heat can be fierce, and only succeeded in getting cooked chamomile. It really doesn't like too much heat, and it is quite a thirsty plant. Why not try repotting it, using fresh potting mix. For one thing, you'll quickly learn if your chamomile is potbound and unable to take up nutrients and water because of squashed roots.