That will be fine. I plant most of my fruit trees - avocados included - about 3 ft from the fence line. I do have to prune the trees every year to keep them from overhanging the neighbor's yard too much.

Have you asked your neighbor if they want this? They might enjoy getting Avo's too and would save you labor (I enjoy my neighbors avo tree that grows into my yard)

I have an avocado about that same distance from a block wall, and I worry that it may cause problems in the future, because a grapefruit (similar distance, but several years older) has cracked the wall due to uplift from its roots.