Research has demonstrated that the perirhinal cortex (PRC) represents complex object-level feature configurations, and participates in familiarity versus novelty discrimination. Barense et al. [(in press) Cerebral Cortex, 22:11, doi:10.1093/cercor/bhr347] postulated that, in addition, the PRC modulates part familiarity responses in lower-level visual areas. We used fMRI to measure activation in the PRC and V2 in response to silhouettes presented peripherally while participants maintained central fixation and performed an object recognition task. There were three types of silhouettes: Familiar Configurations portrayed real-world objects; Part-Rearranged Novel Configurations created by spatially rearranging the parts of the familiar configurations; and Control Novel Configurations in which both the configuration and the ensemble of parts comprising it were novel. For right visual field (RVF) presentation, BOLD responses revealed a significant linear trend in bilateral BA 35 of the PRC (highest activation for Familiar Configurations, lowest for Part-Rearranged Novel Configurations, with Control Novel Configurations in between). For left visual field (LVF) presentation, a significant linear trend was found in a different area (bilateral BA 38, temporal pole) in the opposite direction (Part-Rearranged Novel Configurations highest, Familiar Configurations lowest). These data confirm that the PRC is sensitive to the agreement in familiarity between the configuration level and the part level. As predicted, V2 activation mimicked that of the PRC: for RVF presentation, activity in V2 was significantly higher in the left hemisphere for Familiar Configurations than for Part-Rearranged Novel Configurations, and for LVF presentation, the opposite effect was found in right hemisphere V2. We attribute these patterns in V2 to feedback from the PRC because receptive fields in V2 encompass parts but not configurations. These results reveal two new aspects of PRC function: (1) it is sensitive to the congruency between the familiarity of object configurations and the parts comprising those configurations and (2) it likely modulates familiarity responses in visual area V2. © 2012 Wiley Periodicals, Inc.