First off a couple details- I live in Maine and I don't think the tree is potted in the ideal soil. What happened is the day before my wedding I was drunk, ate a mango, stuck the seed into a big pot of soil I had, then forgot about it for about a month. When I first noticed the tree it took me a few days to figure out what it could possibly be.

So, I know given that this is a tropical plant and I live in ME it will never thrive (or produce fruit for that matter), but I do have a bit of a sentimental attachment to it.
It stays indoors most of the time, and goes out when it's nice out. Strangely, the problems seem to be getting significantly worse since I started putting it outside (only on 50F+ days). It's hard to get a good picture of, but the brown spots in the middle of the leaf look like they're tiny veins turning brown.

Is there anything I can do for it?
Thanks in advance.