Recent discussions of visuospatial working memory have suggested that this subsystem may incorporate a visual buffer which holds visuospatial information relatively passively. Empirical investigations of visual interference with information held within a visuospatial subsystem have yielded somewhat equivocal results. Nonetheless, evidence from Logie (1986) has indicated that visuospatial processing can be disrupted by passive exposure to irrelevant visual material in a manner analogous to the disruption of serial verbal recall by exposure to irrelevant speech. This paper reports two experiments which explore whether such irrelevant visual input is disruptive to storage of imaginal information in a primarily spatial task—the Brooks spatial matrix task. Experiment 1 shows that exposure to irrelevant visual input during encoding selectively disrupts performance on a spatial, but not a verbal, version of the task. The extent of such disruption is shown to be independent of the visual complexity of the material, its similarity to the to-be-remembered information, or a change in state, with a static white square pattern yielding equivalent disruption to that produced by changing matrix patterns. The second experiment indicates that this pattern of effects is robust, and that such disruption is evident at an equivalent level when the visual material is present only during a 20-second retention interval. These results are interpreted as evidence of obligatory access of external visual material to a passive visual buffer. Implications for the nature of a visuospatial subsystem in working memory are discussed.