It depends on what you want out of the space. Flowers? Privacy from neighbors? I would add a couple of small trees for privacy underplanted with ground cover or shade loving perennials. I'd get rid of the rock border and unify the area.

I agree with adding trees for privacy and maybe some shrubs.

More pics of the rest of the space. I see two buildings, one matching brick and one block. Info on the yard and house access with patio or deck.

You mention that you’ll be making the changes slowly, but I would recommend making a plan for the whole space now, that you can implement over time. As others mention, more photos of the rest of the space (particularly looking back toward the house) would be helpful, as would some idea of your climate. Draw a map of your garden, with dimensions, and post it here. Where are you located? Do you know your gardening zone? How do you want to use the space? Do you want a seating area? Do you want to grow edibles? Flowers? How much time do you want to spend taking care of a garden? Think about how you want to use the space.

In the meantime, here are some initial thoughts:

• remove the collar around the base of your tree. The tree has outgrown it and it’s not needed by the tree.
• it looks like someone put a sort of trellis along the brick wall. Since the brick wall is more attractive than the cinder blocks, I would think about growing something either climbing on a trellis or something tall on the other side of the garden instead.
• as callirhoe123 mentioned above, I think your choices are being limited by the current stone border and even the path. Although it looks like it would be quite a project to remove the stones, I would plan for that over time. If you want to be able to enter the back area that you asked about, they are currently a tripping hazard.
• a good rule of thumb when planning a garden is to create pathways (or design raised beds) so that you can reach into your gardening space from the path and not stand on the soil that you’re working on. With this in mind, that area of soil would be subdivided or designed into smaller spaces that could be reached for weeding, picking flowers, deadheading, etc.
• that soil looks pretty compacted, and not much is growing there. Before you put in any plants that you care about, plan to spend some time and money preparing the soil with compost, etc. There is lots of information on line about how to do this.
• while privacy would be nice, it looks like much of your sun is coming from that direction, so you’ll have to balance privacy with a need for sun, unless you plant shade loving plants.
• Finally, your local garden center will be very helpful if you go in when they’re not too busy, especially if you bring photographs of the space. Your local extension service can help you learn about what your soil needs to be healthy.

Along the fence put a low story tree/shrub, maybe dogwoods. This will add all season interest. hide the fence and the neighbors beyond. Infront of the dogwoods add a lower shrub, or taller perennial flowers making a collection of one type by each dogwood. In front of the mid height plants, near the rock border add your favorite annuals. On the brick wall plant some native grasses, and continue with your favorite annuals near the border.

I would go with some shrubs

Here are some more photos of the back of the yard near the house. Also, near the garage in the back. I'm in New York City, zone 7.