I have no direct experience but from the Amherst Bulletin June 2 of last year a story regarding the niece of a famous peony breeder who had trouble with her collection after she moved them into her new home in Amherst:

"Juglans nigra is notorious for causing inhibiting growth of tomatoes and some evergreens - and apparently tree peonies. So they cut down the tree, replanted the peonies in new soil and now they are thriving."

I had regular peonies near a walnut tree...they never thrived. Relocated them...and they are huge and bloom profusely. Just purchased a tree peony...and it's a safe distance away from it. (Or will be once it's planted)