I never thought tomatoes are that hard to care for but maybe I'm doing too much...

Few months ago I decided to garden, starting with herbs. Sadly I already killed my peppermint plant by overwatering it and now it looks like I'm killing the tomatoes. Help!

I bought the little plant and put it in a large pot right away because I thought they are growing fast. Then the first tiny little green tomato started to form and I ran outside every morning to check on it,

watching it grow. I read I need to water them well and I did almost every evening. Then I noticed the smell. I drilled more drainage holes in the pot and stopped watering for a while to let it dry. Once the soil didn't feel wet anymore I started watering again every other day. Now the leaves are yellow and I don't know if it's because i overwatered or if the plant isn't getting enough water...help!

Should I repot into a smaller pot? What am I doing wrong? My plants are doing great but the herbs are a different story. I'm in Southern California and the plant gets more than 8 hours of full sun.