The deposition of amyloid β (Aβ) protein is a consistent pathological hallmark of Alzheimer’s disease (AD) brains; therefore, inhibition of Aβ fibril formation and destabilization of pre-formed Aβ fibrils is an attractive therapeutic and preventive strategy in the development of disease-modifying drugs for AD. This study demonstrated that Paeonia suffruticosa, a traditional medicinal herb, not only inhibited fibril formation of both Aβ1–40 and Aβ1–42 but it also destabilized pre-formed Aβ fibrils in a concentration-dependent manner. Memory function was examined using the passive-avoidance task followed by measurement of Aβ burden in the brains of Tg2576 transgenic mice. The herb improved long-term memory impairment in the transgenic mice and inhibited the accumulation of Aβ in the brain. Three-dimensional HPLC analysis revealed that a water extract of the herb contained several different chemical compounds including 1,2,3,4,6-penta-O-galloyl-β-d-glucopyranose (PGG). No obvious adverse/toxic were found following treatment with PGG. As was observed with Paeonia suffruticosa, PGG alone inhibited Aβ fibril formation and destabilized pre-formed Aβ fibrils in vitro and in vivo. Our results suggest that both Paeonia suffruticosa and its active constituent PGG have strong inhibitory effects on formation of Aβ fibrils in vitro and in vivo. PGG is likely to be a safe and promising lead compound in the development of disease-modifying drugs to prevent and/or cure AD.