Before you take down the fence, check behind it. You will likely find that the arbs have been shaded out by the fence and look pretty unsightly back there. I'd leave the fence.

I'm seeing a small informally paved area (natural flat stones?) big enough for a couple of chairs and a small table, irregularly shaped, with a stepping stone path leading out to the lawn. Ground cover and shrubs of varying heights that are interesting at different times and maybe a small ornamental tree to create a semi-private nook.

You don't say how much sun the area gets.

If the arborvitae look good, I would probably take the fence down. It would make them look like part of your garden instead of the neighbor's. See how nicely they frame the area that doesn't have the fence?

It does look like this area gets more shade than the other part of your yard. Why not have a 'secret garden' type feel there? A pergola (attached to the house) there, with lots of vines growing up and around it, and shade loving shrubs and perennials surrounding the area, with a stepping stone path leading to your chairs on a flat stone area (as saypoint suggested above).
Make sure you can see the backyard from your chairs (around the corner), so you can sit there and enjoy refreshments and still see your entire yard while being in seclusion.

I checked behind the fence. The neighbor's arborvitaes look a little scraggly however I think I can cover that with some dwarf shrubs.

I believe this is mostly a shaded area. Can you recommend a shade loving border of dwarf shrubs or perennials?

Azaleas, hydrangeas, ferns, hostas, mahonias, loripedilum. Dwarf hawthornes do fine for me in the shade, and dwarf hollies do good for me, too. Be sure to check how large some of these get as a few will get quite big.