Sounds like a good question for your local county extension agent. My particular area is low in phosphate, so my potatoes get a fairly heavy dose of Super Phosphate. Ronnigers, aka Potato Garden, has an amendment Acton that I'm going to try out this coming season. You might want to check it out yourself.

My area in southeast, Pa. has a hard pan clay soil. If you dig a hole and fill it with water. The water will still be there two weeks later.
When I started my garden thirteen years ago, I took a ten soil samples and combined them, dried the sample, than screen it through a piece of window screening.
Than I brought a test kit from Penn State Cooperative Extension Service. You can get a test kit in states that have agricultural colleges. My kit was $8, now it is $9. Follow the instructions and send it in. You will get back the results NPK plus information on amendments to the soil.
My soil I added 12 bails of peat moss, bar sand, lime, compost.Till it all in. Walla!

They need high fertility but not an abundance of N. Be careful on which kind of manure for N amount. I don't know what lobster matter is like. Most root crops need phosphorous more than Nitrogen.

I think that you may be wasting your time and money with the peat unless you add 4 bales of it, which is expensive and probably un-necessary, Same goes for the garden soil. If you haven't had a soil test, i'd say go with the composted manure and hope for the best. Next year add as much organic matter as you can find (leaves, compost, old straw, etc).
Potatoes are about the most forgiving crop you can grow when it comes to low ph, low nutrients and new ground. The worst that happens is you will have a less than massive yield. For that space, you still may have hundreds of pounds of potatoes!

Potatoes like a looser and sandy soil with a good amount of potassium. .....more like what lucifer58 did.

I am familiar with sweet potatoes. I always amend the rows with 50% sand I am getting a good results with sand. My soil is very heavy and sticky and I figured the root vegetable need loose soil. Also I add humus and chicken manure. The sand increases the drainage rate beside makes the soil fluffy for the potatoes to grow and expand, Before using sand I used to get skinny roots which were laughable.

I don't think your soil looks too bad, but it will take a few seasons to wake up. Meanwhile, I would incorporate a standard application of a balanced organic fertilizer and grow the potatoes under mulch the first season, then bring in your big guns soil amendments in the fall.