I'm going out on a limb here and wondering if the fig variety you have is a green fig like Green Ischia. Green figs are ripe when green. There is a chance that the brown figs you think are ripe are actually rotten green figs.

Of course, I'm a dummy so don't take my word for it. Wait to get a more knowledgable reply. Lol

You have a male tree aka caprifig, you can graft female varieties onto the male but the caprifigs branches will never be edible.

Haha. See, I told you not to listen to me. I knew nothing of a caprifig. Learn something new everyday.

I'm in Riverside County and have a few really good figs. Learn to graft. Then PM me and you can come and take some cuttings to turn your tree into a really tasty fig producing tree. OR, look for fig trees on sale this year and use those for grafting. You will have to follow me to PM me. I wish you success! Caprifigs require the wasp for pollination and it does exist in parts of So. CA, but not where I live. What part of So CA are you in?