I'm no expert on raised gardens, but have had a few. From your photo, it looks like the garden is only raised a few inches on this end, and I wouldn't hesitate to remove those timbers at all. The back portion looks to be much higher, though, and possibly located where the slope is fairly sharp. I think I'd leave a retaining wall on the higher portions just to be safe.

ETA: maybe photos that better show the highest parts would help.

I would never put a raised bed next to a house, it's an invitation to rot, water infiltration and insects. Many people plant shrubs as foundation plantings but fail to take into account how large they will become. I'm planning on putting in ground cover that blooms, easier maintenance.

As you can see in my photo comment, the foundation extends above the raised part by another foot. So that's 3 ft from ground level. Are you saying I can slope it away from the house without replacing the retaing wall?

I’d rip it all out pronto. Afterwards, if you have soil that’s more than 6” or so higher than the ground level (perhaps at the end toward the white fence that we can’t see well), I’d take a garden hoe and drag some of the dirt into the lawn. Unless it’s really high - like a foot higher. In that case I might substitute some interlocking landscape blocks to hold it for now, just in that space. (In the foreground of the picture, I don’t think you need anything at all.)

Then see how it looks next spring after some weathering.