There are people more expert than I on this....and particularly for your area and I hope they will answer...but a couple things as starters:

1. It would help if this year you could take some leaf samples in to a local State Extension agent that could tell you exactly what the problem is for your tree.....it sounds like you have two problems, one related to the tree and one related to the fruit.....no exacting identification makes advise on sprays problematic.

2. In the Northwest, many organic guys are finding some happiness with Spinosad (Entrust, Bullseye, etc. are trade names).....it is a fermenting bacteria......but again, this is only if you know you have worms in your fruit (you said "bugs" but we need to know more).....I think the odds are good that Spinosad would help and you need to learn more about it and how to apply it, in my opinion. Local nursery would help.