Despite the fencing I'd say that looks like deer.

I think it looks like deer too. Since they are small and non-bearing try spraying them the with deer repellent like Bobbex Deer Repellent or whatever brand is sold in your area (usually has garlic and rotten eggs in the mix so it will smell bad). Or try hanging the stinky bars of soap in the tree. They provide some relieve.

Is the fence too close and just around the tree? If so try using a wider circle of fencing to keep the 4 legged thieves away. I can relate as they ate down apples I personally grafted 2 years in a row! Stinky deer repellent proved some relief for a few weeks. Lost two seasons worth of growth. After 6 years they are finally bearing some fruit. The deer still mow some of my lower branches.

I use Deer Off. It is basically mint oil. It won't completely stop the deer, but it greatly curbs their appetite for the leaves. I spray every two weeks during the peak growing season.