'Three or four times a year.....' I think is the key to the problem. Groundsel, for example, can flower and seed in a few days. To reduce the seed bank weeding needs to be much more frequent. At least weekly - this sounds horrendous but it will gradually get easier and easier until you are just pulling a few weeds.

If you are putting down a growth inhibitor such as Preen and unwanted plants are appearing in a couple of weeks then the product is either defective or not being applied correctly.
I have had better, and much more effective, results using newspaper and a mulch material then with growth inhibitors.

I apply Preen exactly as detailed on the container.

I've even contacted Lebanon Seaboard Corporation (the maker of Preen) and gave them:

a list of the existing shrubs and groundcover
a list of the predominate weeds
my weeding technique
my soil-preparation technique
how I'm applying Preen
and how I do subsequent watering.

They said I'm doing everything correctly

(By the way, they did give me a refund since I was not satisfied with Preen's performance)

===

I'd like to have a groundcover but, if I use newspaper or black plastic, I can't.

Don't use newspaper or plastic, instead mulch heavily.

Don't use plastic, black, clear, brown, but with newspaper you could plant a ground cover, after those unwanted plants have been controlled. Newspaper, applied over unwanted plants deprives them of the sunlight the plants need to grow and then is converted, by the Soil Food Web, into humus in the soil.
Since you have been applying that product correctly then the product is defective.

This stuff doesn't work, just gives a false sense of security. If you will pull the weeds (which you are doing anyway) before they make seeds, fewer seeds will be dropped to sprout in the future. Also, instead of waiting until pulling unwanted sprouts is a time-consuming chore, snatch them randomly whenever you walk by, as part of admiring your pretty garden. Letting them grow until it's a big chore is just being mean to yourself (and using more of your time.) Keep your garden mulched with actual mulch or leaves to help maintain control. You'll still have sprouts, nothing can prevent that, but there will be fewer, and the ones you get will pull out more easily and completely.