Memory impairment observed in patients with medial temporal lobe epilepsy (MTLE) is classically attributed to hippocampal atrophy. The contribution of extrahippocampal structures in shaping memory impairment in patients with MTLE is not yet completely understood, even though atrophy in MTLE extends beyond the hippocampus. We aimed to evaluate the neuropsychological profile of patients with MTLE focusing on memory, and to investigate whether gray matter concentration (GMC) distribution within and outside the medial portion of the temporal lobes would be associated with their neuropsychological performance. We performed a voxel based morphometry study of 36 consecutive patients with MTLE and unilateral hippocampal atrophy. We observed a significant simple regression between general and verbal memory performance based on Wechsler Memory Scale—Revised and the GMC of medial temporal and extratemporal structures in patients with left MTLE. We also performed a “regions of interest analysis” of the medial temporal lobe, and we observed that the GMC of the hippocampus, entorhinal, and perirhinal cortices were consistently associated with general and verbal memory performance in patients with MTLE. We also observed that the GMC of the cingulate and orbito-frontal cortex are independently associated with verbal and general memory performances. Our results suggest that general and verbal memory impairments in patients with left MTLE are associated with atrophy of the hippocampus, the entorhinal, and the perirhinal cortex. We also suggest that atrophy and dysfunction of limbic and frontal structures such as the cingulate and the orbito-frontal cortex contribute to memory impairment in MTLE. Hum Brain Mapp 2007. © 2007 Wiley-Liss, Inc.