A relatively recent explanation has it that Vitex was an anti-aphrodisiac for men*, but an aphrodisiac for women. Though this site says:

"In ancient Greek mythology the Chaste Tree was associated with the Greek goddess Hera who had domain over marital chastity. It is said she was born and nursed beneath a Chaste Tree on the Greek island of Samos.""

Duelling effects are not just postulated for Vitex. Awhile back I posted an online list of things cider vinegar was supposed to accomplish, which included helping just about any condition you could name. The author(s) claimed it was good for both weight loss and weight gain.

Some try to reconcile these contradictions by labeling certain herbs as "adaptogens", which comes across as a more science-y way of saying "Good for what ails you". ;)

*As to the monks, based on reports about what they were up to from ancient times on through Rasputin...if they were taking any anti-aphrodisiacs, they evidently didn't work so well.