There's a great chance that the tree hasn't been watered properly. For the most part I only go out there and water when I remember, and that can be quite infrequent.

The bigger issue in my original post (sorry if I didn't put emphasis on it as such) is what to do about the size and location of the tree.

Depending on the speed of tree growth, climate and your cultural practices, a selective canopy thinning may be carried out. In fact this is an annual practice in most Mediterranean countries. The choice of branches to eliminate is based on the concept that any space with in the canopy must be covered by only one branch. It is not wise to let surplus branches occupy the same aerial space. Anyway, thinning should not completely deplete any canopy sector. Suckers should only be kept if they occupy free spaces. After a few years they bear fruit, but ALL interior suckers must be cut out, A harmonious citrus tree grows to an almost round shape. Vegetation free canopy spaces should always be avoided since they reduce yeild. If your neighbor does not want your tree over their property, you can do what is called headging. You can find out more about "citrus tree hedging" in many place on the Internet. It's your tree and you can certainly prune your tree back anyway you wish. However, know that a citrus tree produces fruit on new wood, so you will delay a crop for a year. The best to you and this tree. - Silica

Personally, I pick all my oranges around New Years and don't store them on the tree. They attract rats & neighbors (no, really. People sneak in & take my oranges!). Plus, it gives the tree a break.

...Are you picking oranges that have sat ripe for 6 months or are you picking fruit that is still growing? Neither are going to taste great. Oranges are only in season a few months of the year. Pick & eat them then.

If your oranges are sad & flavorless when they're supposed to be ripe, make sure you're following the citrus care guidelines from your local citrus station. That will almost certainly fix your problem.

Before you go hacking on your tree, talk to your neighbors. Ask if they mind the tree. If they're like mine, they love the shade & the delicious oranges hanging in their yard that they didn't have to take care of. You might find your neighbors feel the same & want the tree to continue hanging into their yard.

Have you fertilized at all?

How about pruning? You need to get into the interior of the tree and cut all dead branches and others that will cause a problem. Open up the inside for light and air.