Memory for context information (source memory) has been reported to rely on structures in the medial temporal lobe (MTL). Perirhinal cortex (anterior MTL) and parahippocampal cortex (posterior MTL) have distinct connectivity patterns with sensory neocortex, suggesting a possible modality-dependent organization of memory processes. The present study investigated the neural substrates of two different types of source information of newly encoded material using functional magnetic resonance imaging: auditory (speaker voice) and visual (texture and colour). Source judgements during retrieval were reliably above chance level for both modalities and performance did not differ between the auditory and visual condition. During encoding, activity predictive of subsequent source recollection was observed in the anterior hippocampus/parahippocampal gyrus, irrespective of source modality. During retrieval, on the other hand, a regional dissociation emerged: bilateral parahippocampal cortex discriminated between correct and incorrect auditory but not visual source judgements, whereas left perirhinal/entorhinal cortex showed the reverse pattern. These findings are consistent with recent lesion evidence of disrupted auditory but intact visual source memory following damage to the parahippocampal cortex. Results are discussed with respect to anatomical models of corticoparahippocampal connectivity and the functional organization of the MTL.