That's a tough one, if you allow them to remain and nest there you can expect them to use your garden as a buffet.

Disturb them a little bit, and the mother might keep them away.

I agree with the previous post. You should disturb them, so that Mama Rabbit relocates the babies elsewhere. Otherwise, the time may come - rapidly - when the damage becomes severe, and you may not have merciful options.

Yesterday I found several of my gherkin vines chewed off near the ground, which is typical vole damage. I removed the landscape fabric from around the vines, saw the vole run for cover, and found its nest. There were four tiny mice buried under grass clippings. A vole will not relocate its young, or leave the garden unless trapped... so I quickly killed the young, and destroyed the nest. At some point, I will need to trap the adult, unless the semi-feral cats on the property get it first.

Well something got to the nest two days after I posted this. There were two dead babies and one laying there still breathing. It looked like something was digging around the nest and found them. I took the struggling one and the two others I found to a rehabber per their recommendation.

How sad, but nature is cruel :(. That was kind of you to save them :).

Baby rabbits are basically nature's snack food. Mother rabbits aren't particularly protective or attentive anyway. You could have just moved the nest. They have three broods a year at least. Not a high survival rate, but one is not needed considering how many bunnies are out there. Thankfully you had a rehab place nearby to help you with your conscience. I used to work with one. Rabbits are also very haphazard as to where and how they build their nests. So many sad folks who discovered a nest "accidentally." One does the best one can do . . .