Keep dirt and decomposing vegetable matter away from your exterior walls. It can lead to (a) termite infestation, (b) moisture intrusion, which will cause dry rot, (c) staining of stone or brick. Even stone and brick are both porous and permeable and thus allow moisture to pass if saturated. Also, moisture can cause some types of brick (depending upon curing and how they were formed) to become friable (it will crumble), which is costly to repair.

Older houses here in Oklahoma built of Guthrie red sandstone are also damaged by moisture....Guthrie sandstone is very soft for a stone and crumbles if subjected to wet.

Robert
OKC

P.S. Raised beds are nevertheless okay, just have a barrier (cinder block, stone, something) between your garden soil and the exterior wall of the house. Repairs for this type of wall damage can be hugely expensive; not to mention the catastrophe of having termites move in or dry rot.