Looks like it might be a little deep...I'd pull back some mulch and possibly soil until you see a nice root flare. More sun would increase it's vigor...but if it truly gets 6-7 hours of hot afternoon soon, it should be good.

I also planted a ginkgo in the fall of 2019. It was the "Golden Globe" cultivar, which is supposedly has a more vigorous growth pattern. It is about the size of your tree as well. Like you, mine looked great the next summer (2020), big leaves that stood tall, nice dark foliage, etc. Since then (2021 and 2022), the foliage is smaller, paler, and looks a little droopy come late summer. It has maybe 3 branches that have put on some new growth of about 4-6", and the trunk is only slightly bigger in diameter.

My theory is the first summer after planting, it was still loaded up with the years of fertilizer and steroids it received being container grown in a nursery. After that, it settled into it's spot and started doing it's normal thing. For a ginkgo tree, the normal thing is painfully slow to establish and start putting on noticeable increases in size. Your zone will make some difference, I'm in zone 6, central Indiana. A warmer zone might show a little more growth the first few years, but not much. Later in life, a ginkgo will pick up the pace...at least I see that on 20-30 year old I see around my town. But that first 5-7 years takes a lot of patience.

Other than maybe being a little too deep, which can be remedied, I don't see anything out of the norm for your ginkgo.