First you should determine what you want to do in the garden area. It doesn't appear to be easily accessible from the apartment, so it is a destination rather than an extension of the living area.
Do you really have to turn sideways and scrape against the fence to get to the garden area around the tree?
If you want it as a sitting/relaxing area with privacy it looks like you are well on your way. You should create a sitting area near one corner so you can see the entire area and it will appear larger. I think that a small water as a focal point would be a good addition.
Do a search for small gardens and Japanese gardens for some design ideas.
Your sketch is fine. Please update it when you have ideas for how you want to use the space and you will get more input.

Thanks for the advice.

Re the path, since I'm a tenant, I can't change the major features such as the tree. I really wanted the garden to be a green al fresco dining area. There is a raised bed with stone borders that extends down the south side fence and wraps around to the west side fence. I wanted to plants these areas and leave the stone path in tact.

Next spring, I was thinking about moving some plants to better locations - rose bushes to north wall. If I can I would move the holly to the east side fence. I'm not really sure what to do with the dwarf alberta spruce. It doesn't get enough light where it is.

I'd also like to cut back the rose of sharon and the eastern hemlock (I think) whose branches spread over the east side fence.

Re path, I was thinking I would add turf around the tree and between the stones. I would also add more shade friendly pants - ferns, hostas - to green the space.

Re southwest corner - If I move the rose bush, I'll have to figure out what to do with this corner.