"If you plant three or four years in the same place are you asking for trouble?"

Depends. Some people will just say 'yes' or 'maybe', but that really doesn't address the root of the (potential) problems. I've grown in the same area for several years, so have several other people on the forums.

"Would that be suficient to avoid a build up of nematodes or whatever disease organism it is that might make this inadvisable?"

Are you currently experiencing any problems growing in that area? If so, then my general advice would be to identify the specific problem, -then- come to a determination what the course of action would be.

If not, I don't see any reason why you can't keep growing there, with periodic additions of organic matter in the off-season so you don't deplete the area. Also remember that mulching can reduce splashback of fungal spores onto the plants and is a good general practice. So is keeping your garden clear of diseased plant materials (reduces pathogen load).

Are you currently experiencing any problems growing in that area?

No, this comes under the general heading of "preemptive strike". I don't want to get half way way through a season and have problems.
I use weed barrier and drip tape so I get no splash back. It seldom rains in my part of Washington State.
I have t-stakes and hog panels and I don't want to pull them up and move them.

The addition of organic matter will be one of your best defenses for keeping your soil healthy. Having lots of "good" bacteria in the soil will help combat the "bad stuff".They're are also things like beneficial nematodes that can be added to your garden area. Theses nematodes attack and kill root nematodes and some insect larvae.

Unfortunatly the best way to incorperate the organic matter may involve removing your stakes and fencing and tilling it in.

Good luck