SEARCH

SEARCH BY CITATION

Keywords:

  • Embodied cognition;
  • Concepts;
  • Perception;
  • Property verification;
  • Modality-switch effect

Abstract

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix

According to the Perceptual Symbols Theory of cognition (Barsalou, 1999), modality-specific simulations underlie the representation of concepts. A strong prediction of this view is that perceptual processing affects conceptual processing. In this study, participants performed a perceptual detection task and a conceptual property-verification task in alternation. Responses on the property-verification task were slower for those trials that were preceded by a perceptual trial in a different modality than for those that were preceded by a perceptual trial in the same modality. This finding of a modality-switch effect across perceptual processing and conceptual processing supports the hypothesis that perceptual and conceptual representations are partially based on the same systems.


1. Introduction

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix

A hallmark of human cognition is the ability to form internal representations of the external world. This ability is essential for a wide variety of cognitive operations, ranging from input-driven online processes including perception, categorization, and motor planning; to offline processes such as remembering, reasoning, planning, and producing and understanding language. Embodied views of cognition suggest that these internal representations are grounded in the systems of perception and action that are used to interact with the world (e.g., Barsalou, 1999; Glenberg, 1997; Grush, 2004; Pecher & Zwaan, 2005). According to the Perceptual Symbols Theory (Barsalou, 1999), the basic building blocks of conceptual representations are perceptual symbols. Perceptual symbols are analogue, multimodal (i.e., multiple sensory modalities are involved), and directly grounded in the brain's modality-specific systems. In this view, conceptual representations are formed by dynamically combining perceptual symbols into multimodal simulations. These simulations are partial reinstatements of the neural patterns that are created in actual experience with the world during perception, action, and interoception. To represent the concept COFFEE CUP,1 for example, the conceptual system simulates seeing, touching, or grasping an actual coffee cup, along with internal reactions that reflect goals, thirst, emotion, and so forth.

The Perceptual Symbols Theory is often contrasted with more traditional accounts of cognition that assume representations in an amodal symbolic format (Fodor, 1975). These include theories such as the physical symbol theory (Newell & Simon, 1972), semantic network theories (e.g., Collins & Quillian, 1969; Rapaport, 2002), exemplar models (e.g., Nosofsky, 1986), and propositional models (e.g., Kintsch, 1998). Whereas these theories do not explicitly claim that representations are amodal, they largely neglect the role of sensorimotor systems in representation. In these theories, a symbolic notation is used to represent concepts, which might imply that concepts are abstracted from sensorimotor experiences into amodal symbols. Such notations may be adequate for the purpose of the particular theory, but they constitute a problem for the question how these symbols are given meaning (Harnad, 1990; Johnson-Laird, Herrmann, & Chaffin, 1984; McNamara & Miller, 1989). The meaning of a symbol cannot be based solely on its relation to other symbols. For a system to have meaning, it needs to be grounded in real-world experience. The Perceptual Symbols Theory (Barsalou, 1999), therefore, assumes that the conceptual system is strongly intertwined with the perceptual and motor systems, and that concepts are represented by recreating patterns of activation that are associated with actual perception and action. This view may not necessarily contradict more traditional accounts, but it supplements the traditional view in a significant way.

An important assumption of the Perceptual Symbols Theory is that representations are highly flexible and dynamic (Barsalou, 1993). Concepts stored in memory consist of elaborated, multimodal knowledge structures, called simulators. Across different occasions, different subsets of a simulator are activated to construct specific simulations, making each simulation unique. The exact content of a particular simulation depends on the individual's experience with the simulated concept, as well as on situational factors such as current goals and task demands. For example, a concert pianist may form different simulations of PIANO in various contexts. When thinking about an upcoming performance, the simulation might include the piano's sound, along with the fine hand movements involved in playing the instrument. When planning to move the piano into a new apartment, however, the simulation might include the shape, size, and weight of the piano, along with the gross movements necessary for lifting and moving it.

The flexibility of concepts was first demonstrated by Barclay, Bransford, Franks, McCarrell, and Nitsch (1974). In a classic experiment, they presented sentences describing concepts in different contexts, such as “The man lifted the piano,” or “The man tuned the piano,” followed by a cued-recall task. The recall cue, “something heavy,” was more effective when participants had been presented with the “piano lifting” sentence; whereas the cue, “something with a nice sound,” was more effective when the participants had read the “piano tuning” sentence. Similar results have been obtained in other paradigms (e.g., Barsalou, 1982; Zeelenberg, Pecher, Shiffrin, & Raaijmakers, 2003). These results indicate that participants had activated partial, context-dependent representations of the concepts presented in the sentences.

Given the flexibility of concepts, a simulation may be dominated by one modality, with other modalities engaged to a lesser extent. If two representations are simulated by different modality-specific systems, then constructing them in turn will involve a switch of attention from one modality to another. It is expected that this switch of attention will cause a processing cost. In previous research, such a modality-switch cost has indeed been found, both in studies using a purely conceptual task (Pecher, Zeelenberg, & Barsalou, 2003) and in studies using a purely perceptual task (e.g., Spence, Nicholls, & Driver, 2001).

In the study of Spence et al. (2001), participants performed a simple perceptual decision task. On each trial, a visual, tactile, or auditory stimulus appeared left or right, with the modality of the stimulus varying randomly from trial to trial. Participants responded to the position of the stimulus as rapidly as possible (i.e., left or right). Reaction times and error rates were higher for trials that were preceded by a different-modality trial than for trials that were preceded by a same-modality trial, indicating that switching from one perceptual modality to another involved a processing cost.

A similar modality-switch effect was found by Pecher et al. (2003), using a property-verification task (also see Marques, 2006; Vermeulen, Niedenthal, & Luminet, 2007). In the study of Pecher et al. (2003), each target trial consisted of the presentation of a short sentence containing the name of a concept, the words “can be,” and a property from one of six modalities (vision, audition, taste, smell, touch, action; e.g., “BANANA can be YELLOW”). Participants were instructed to verify whether the property was true for the concept. Critically, one half of the target trials were preceded by a trial with a property from the same modality. The other target trials were preceded by a trial with a property from a different modality. For example, in a particular trial, participants verified the auditory property loud for BLENDER. In the previous trial, they had verified a property either on the same modality (e.g., LEAVESrustling) or on a different modality (e.g., CRANBERRIEStart). Analogous to the findings of Spence et al. (2001), a modality-switch effect was found. Properties were verified faster and more accurately in same-modality pairs than in different-modality pairs. This analogy between modality-specific perceptual and conceptual processing supports the theory that conceptual representations are grounded in the brain's modality-specific systems. If conceptual representations are formed by simulations of experience in the modality-specific systems of the brain, then analogous phenomena should be observed in perceptual and conceptual processing.

Although Pecher et al. (2003) showed that the modality-switch effect was not easily explained by associative priming between properties from the same modality, a critic might still argue that the modality-switch effect for conceptual representations could be the result of connections between symbols in an amodal system. Properties that are learned during perceptual processing could be transduced into amodal symbols. These symbols, although being amodal, might still be organized in a way that reflects their modality. This idea is not supported by a number of studies that show a direct interaction between language and perception. For example, Zwaan and colleagues have shown in various studies that during sentence comprehension, performance to pictures was facilitated if they matched the situation implied by the sentence (Stanfield & Zwaan, 2001; Zwaan & Madden, 2005; Zwaan, Stanfield, & Yaxley, 2002).

In all these studies, however, the match effect was due to overlap between language and perception in a meaningful way. Stanfield and Zwaan (2001), for example, showed faster recognition of pictures of objects if the object was in the same orientation as was implied by the sentence (e.g., a picture of a vertical pencil after the sentence, He put the pencil in the cup.). These effects have been interpreted as showing that language comprehenders run a sensorimotor simulation of the situation described by the sentence. There is a possibility, however, that these results are due to strategic use of imagery rather than automatic language comprehension processes. Because each sentence was followed by a picture of an object, participants in Stanfield and Zwaan's study may have used a strategy to form a mental image of the situation described in the sentence to recognize the picture. Such strategic use of imagery is different from the sensorimotor simulations that Barsalou (1999) proposed, because the latter are formed automatically and often without awareness. Whereas imagery might follow sentence comprehension, sensorimotor simulations constitute comprehension.

In this study, we aimed to find evidence for the involvement of sensorimotor systems in conceptual representations that could not be explained by amodal symbols or strategic imagery. We investigated the effect of a perceptual task on concept representation when no meaningful relation existed between the percept and concept and the task did not require comparison of the percept and concept. Specifically, we investigated if modality-switch effects would occur not only within perception or conception, but also across perceptual and conceptual tasks. Such a result would demonstrate that conceptual processing not only parallels perceptual processing in some respects, but that it is truly based on the same system.

This idea was explored in the present study in which the tasks used by Spence et al. (2001) and Pecher et al. (2003) were combined. On each trial, participants performed a perceptual detection task followed by a conceptual property-verification task. In the perceptual task, participants decided whether a simple perceptual stimulus (a light flash, a tone, or a vibration) appeared left or right. Thus, modality was irrelevant to the task. In the conceptual task, participants decided whether a property was true for a concept. The properties were modality specific and came from the same three modalities (i.e., vision, audition, and touch) as the perceptual stimuli. The property-verification task was used as the target task. It was expected that performance would be worse (longer reaction times, higher error rates, or both) on trials that were preceded by a perceptual stimulus on a different modality than on trials that were preceded by a stimulus on the same modality.

2. Method

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix

2.1. Participants

Eighty undergraduate students from Erasmus University Rotterdam participated in this experiment in return for course credit or a small monetary reward (4 euro). All participants were fluent in Dutch. Same versus different modality was manipulated within-subjects. Each participant received one of the four counterbalanced lists of trial pairs.

2.2. Apparatus and stimulus materials

2.2.1. Perceptual stimuli

Participants were seated in front of a computer screen. Speakers were placed on both sides of the computer screen for the presentation of the auditory stimuli. An auditory stimulus consisted of the presentation of white noise from one of the two speakers. A white circular LED was mounted on each speaker for the presentation of the visual stimuli. A visual stimulus consisted of the illumination of one of the two LEDs. A custom-made device for presentation of tactile stimuli was placed on the tabletop immediately in front of each speaker. Participants rested their index fingers on the tactile stimulators. A tactile stimulus consisted of a vibration of one of the two stimulators. Each perceptual stimulus was presented until the participant made a response or until 1,500 msec had passed. By presenting the stimuli in each modality from the same two locations, the potential for a spatial confound was eliminated.

The setup used in this experiment approximated the setup that was used by Spence et al. (2001). A pilot study was done to verify that Spence's perceptual-only modality-switch effect could be replicated with our setup. In this pilot experiment, 31 participants made left–right decisions to 216 target stimuli in each of the three modalities. One half of the target trials were preceded by a stimulus in the same modality, and the other targets were preceded by a stimulus in a different modality. A significant modality-switch effect was found, t(30) = 9.14, p < .001. Participants responded faster to trials that were preceded by a same-modality trial (M = 419 msec) than to trials that were preceded by a different-modality trial (M = 441 msec). Error scores did not significantly differ between the two conditions, t(30) = 1.33, p = .19.

2.2.2. Conceptual stimuli

A total of 72 auditory, visual, and tactile concept–property pairs was selected from a large sample of Dutch stimuli. Prior to the experiment, the items in this sample had been rated by 12 participants. The participants rated the degree to which the property could be perceived with any of the three senses (visual, auditory, tactile) on a 6-point scale ranging from 1 (cannot be perceived by this sense at all) to 6 (is perceived by this sense completely). The stimulus materials can be found in the Appendix.

The ratings provided by the participants were used to compute a measure of purity of the property that indicated to what extent the intended modality was exclusive in perceiving the property. This was done by subtracting the highest rating of the two irrelevant modalities from the score on the relevant modality. For example, if the tactile concept–property pair SOUP–hot had an average rating of 2 on the visual modality, 0 on the auditory modality, and 6 on the tactile modality, its score on the purity scale would be 4 (6 − 2). This scoring method was used to select only those items in which one modality was clearly dominant. On average, the visual concept–property pairs received a higher purity score (5.8) than the auditory (5.0) and tactile (4.8) items. This is probably due to the dominance of the visual sense above the other senses. Each concept–property pair was used only once in the experiment. However, because it was very difficult to find enough pure properties in the tactile modality, three of the tactile properties were repeated once, although paired with different concepts. The items that shared a property were always presented in different blocks, with a lag of one block (consisting of 36 pairs of trials) between them. In addition to the experimental trials, 72 false filler trials were created. Twelve filler items had a false auditory property, 12 had a false visual property, and 12 had a false tactile property. The additional 36 filler items had a false property that was not from one of these three modalities. As in the Pecher et al. (2003) study, many of the concepts and properties in filler trials were strongly associated to each other, but were actually related in an incorrect manner (e.g., “a cow drinks milk” or “a bed is sleepy”). This was done to prevent participants from basing their answer on a superficial word-association strategy rather than on deeper conceptual processing (Solomon & Barsalou, 2004).

2.3. Procedure

The experiment took place in a dimly lit room. Participants were seated in front of a computer. They placed each foot on one of two pedals, rested their hands on the table, and placed their index fingers on the tactile stimulators. The trials were presented pairwise, each trial pair consisting of a perceptual trial followed by a property-verification trial. The property-verification task was treated as the target task. Reaction times and error rates for those trials preceded by same-modality perceptual trials were compared to those preceded by different-modality perceptual trials.

A trial pair started with the presentation of a blank screen for 500 msec. Directly following the blank screen, a perceptual stimulus (in any of the 3 modalities) was presented to the left or to the right of the computer screen. Participants responded to the location of the perceptual stimulus as rapidly as possible by pressing down the foot pedal on the same side as the stimulus. Error feedback was given when the response was incorrect (“FOUT,” Dutch for error) or when no response was given after 1,500 msec (“TE LAAT,” Dutch for too slow). The feedback was presented in red uppercase letters for 1,000 msec in the center of the computer screen. The response or feedback was followed by a fixation cue (*****), presented for 500 msec in the center of the computer screen. Immediately after the fixation cue, a sentence containing the concept and the property (e.g., “a banana is yellow”) was presented in the middle of the screen in black letters. The target sentence was presented for 3,000 msec or until a response was given. Participants verified as quickly and accurately as possible whether the property was true of the concept, and responded by pressing down one of the foot pedals. One half of the participants were instructed to press the left foot pedal for a “true” response and to press the right foot pedal for a “false” response. The other half of the participants received the opposite response instruction. If the response was incorrect or not given within 3,000 msec after stimulus onset, feedback was given for 1,500 msec.

The experiment began with a general instruction and a practice block, consisting of 18 trial pairs. After the practice block, the participants continued with the first experimental block. In total, there were four experimental blocks. Each block consisted of 18 experimental trial pairs and 18 filler trial pairs. Each block had an equal number of trials in each modality, and an equal number of switch and no-switch trial pairs. After every block, participants could take a short break during which they received feedback about their performance in that block. If the average accuracy on the perceptual and conceptual trials was lower than 85%, participants were urged to try to be more accurate. If the accuracy was higher than 95%, they were complimented. The next block was initiated by pressing down one of the foot pedals.

3. Results

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix

The data from 5 participants were removed because they failed to reach an accuracy score of 70% on the conceptual trials. This left a total of 75 participants. Items that had an error rate higher than 40% were removed from the analysis. This resulted in the exclusion of 5 of the 72 items (1 visual, 2 auditory, and 2 tactile items). Correct reaction times to trials for which the response to the preceding perceptual trial was correct and that fell within two standard deviations from the participant's condition mean were used to compute mean reaction times. In total, 13% of the observations were removed due to errors on either the conceptual trial itself or on the preceding perceptual trial, and 2% of the observations were removed due to outlier reaction times. Mean reaction times for switch and no-switch trials were computed for each participant and then averaged across participants. Reaction times to switch trials were slower (M = 1,248 msec) than to no-switch trials (M = 1,226 msec). This modality-switch effect was significant, t(74) = 2.32, p = .012. The error rates did not differ between the switch and no-switch conditions, t(74) = 0.32, p > .75.

These results extend the findings of Pecher et al. (2003). Whereas Pecher et al. (2003) provided evidence that switching modalities within a conceptual task incurs a processing cost, this study shows that the modality-switch effect also occurs when switching from a perceptual task to a conceptual task. Therefore, conceptual processing is affected by perceptual processing.

4. Discussion

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix

Embodied views of cognition propose that the representation of concepts relies on the systems for perception, action, and interoception. To represent a concept, these systems run a (partial) simulation of experiencing a possible instance of it. If conceptual processing is based on perception, action, and interoception, then two predictions follow. First, phenomena that occur in perception, action, and interoception should have their equivalents in conceptual processing. Second, these two sets of processes should influence each other.

Consequently, two lines of research have followed, accumulating evidence for each of the predictions. The experiments of Pecher et al. (2003) addressed the first line of research. In that work, property verifications were slower and less accurate when the concept–property pair was preceded by a concept–property pair from a different modality than when it was preceded by a concept–property pair from the same modality. This result parallels the modality-switch effect observed in pure perceptual tasks (Spence et al., 2001), and as such suggests that conceptual processing is grounded in the modality-specific systems of perception and action.

The second prediction—perceptual processing interacts with conceptual processing—was addressed in the present study. It was demonstrated that the modality-switch effect does not only occur within conceptual processing, but also between perceptual processing and conceptual processing. The present study is not a mere extension of previous research; it is the first study that shows how pure perceptual processing (perceiving stimuli without any semantic meaning) can affect the activation of conceptual knowledge. This finding confirms the hypothesis that these two types of processing are partially based on the same systems. By showing that perceptual processing affects conceptual processing, this study takes its place in the second line of evidence for the embodied view of cognition.

An aspect of this study that deserves particular consideration is the nature of the perceptual task. The task was a simple stimulus-detection task using simple, non-symbolic percepts. In this regard, the task required only low-level perceptual processing. Previous studies have often used meaningful pictorial stimuli. For example, pictures were used to investigate the role of perceptual processing in conceptual representation (Pecher, Zanolie, & Zeelenberg, 2007) and language processing (Stanfield & Zwaan, 2001; Zwaan & Madden, 2005; Zwaan et al., 2002). It could be argued that processing picture stimuli involves a great deal of categorization and conceptualization, and that this kind of high-level perception in fact might partly overlap with conceptual processing. However, it would be much harder to make that case for the low-level stimuli employed in this task. The finding of a modality-switch effect between perceptual and conceptual processing suggests that low-level perceptual systems are involved in the construction of mental simulations.

Recently, a number of other studies, using percepts of motion, have provided evidence for the interaction between low-level perception and language comprehension. For example, Kaschak et al. (2004) and Kaschak, Zwaan, Aveyard, and Yaxley (2006) showed that perceiving visual or auditory motion in different directions (up, down, toward, away) affected verification of sentences that described motion in these directions (also see Richardson, Spivey, Barsalou, & McRae, 2003). Meteyard, Bahrami, and Vigliocco (2007) demonstrated an effect in the opposite direction. In their study, participants performed a vertical motion detection task while listening to verbs describing upward and downward movement. Performance in a motion detection task was impaired when participants listened to directionally incongruent verbs. Finally, in an experiment by Vermeulen, Corneille, Budke, and Niedenthal (2007), participants verified visual and auditory properties while memorizing one or three perceptual stimuli from the same or different modality. Property verifications were impaired by a high memory load in the same modality, but not by a low memory load, indicating that perceptual simulations are subject to modality-specific resource limitations. Taken together, these studies are in line with our research in showing that perception and cognition influence each other at different levels of processing.

Whereas in previous demonstrations of the modality-switch effect using the property-verification task all stimuli required conceptual processing (Marques, 2006; Pecher et al., 2003; Vermeulen et al., 2007), in this study the perceptual stimuli did not require conceptual processing, thereby showing more directly that perceptual and conceptual processing are intimately related. It is unlikely that the perceptual stimuli in this study gave rise to higher level conceptual representations, for example, by activating the categories lights, sounds, and vibrations.2 Modality was irrelevant in the perceptual detection task (or in the conceptual task, for that matter). The participants' task was to respond to the location of the percepts, not to their modality. If some kind of categorization of the stimuli would have occurred, it is more likely that participants categorized them by location (the relevant feature) than by modality. The fact that a modality-switch effect occurred despite the irrelevance of modality in either the perceptual task or the conceptual task, suggests that it reflects a true switch between different modality-specific perceptual systems, rather than within the same conceptual system.

The results from this study are consistent with the idea that selective activation of a modality-specific brain underlies the representation of concepts and properties in different modalities. This is confirmed by a growing number of neuroimaging studies showing that modality-specific brain areas are active during conceptual processing (for reviews, see Martin, 2001, 2007; Martin & Chao, 2001). Specific evidence for modality-specific activation during property verification comes from a recent study by Goldberg, Perfetti, and Schneider (2006). This study applied functional magnetic resonance imaging to scan participants during a property-verification task, using properties from four different sensory modalities (color, sound, touch, and taste). Across modalities, the property-verification task activated specific brain regions associated with perception and encoding of sensory experiences in the relevant modality. This led the authors to conclude that the retrieval of perceptual knowledge relies on the modality-specific brain regions that are associated with experiencing instances of the represented concepts. The results from our study corroborate this conclusion. Moreover, they suggest that the observed activations of modality-specific brain regions did not occur as a mere corollary of representation in an amodal symbol system, but that they themselves reflect the actual representing.

Note
  • 1

    Throughout this article, we use uppercase italics to represent concepts, lowercase italics to represent properties, and quotes for words and sentences.

  • 2

    We thank Daniel Casasanto for raising this point.

Acknowledgments

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix

This research was supported by a VIDI grant from the Netherlands Organization for Scientific Research (NWO) to Diane Pecher. We thank Gerrit Jan de Bie for his technical support and Carol Madden for helpful suggestions on earlier versions of this manuscript.

References

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix
  • Barclay, J. R., Bransford, J. D., Franks, J. J., McCarrell, N. S., Nitsch, K. Comprehension and semantic flexibility. Journal of Verbal Learning and Verbal Behavior. 1974 13 471481.
  • Barsalou, L. W. Context-independent and context-dependent information in concepts. Memory & Cognition 1982 10 8293.
  • Barsalou, L. W. Flexibility, structure, and linguistic vagary in concepts: Manifestations of a compositional system of perceptual symbols In Collins, A. C., Gathercole, S. E., Conway, M. A. (Eds.), Theories of memory 1993 Hillsdal, NJ: Lawrence Erlbaum Associates, Inc 29101.
  • Barsalou, L. W. Perceptual symbol systems. Behavioral and Brain Sciences. 1999 22 577660.
  • Collins, A. M., Quillian, M. R. Retrieval time from semantic memory. Journal of Verbal Learning and Verbal Memory. 1969 8 240247.
  • Fodor, J. A. The language of thought 1975 Cambridg, MA: Harvard University Press.
  • Glenberg, A. M. What memory is for. Behavioral and Brain Sciences. 1997 20 155.
  • Goldberg, R. F., Perfetti, C. A., Schneider, W. Perceptual knowledge retrieval activates sensory brain regions. The Journal of Neuroscience. 2006 26 49174921.
  • Grush, R. The emulation theory of representation: Motor control, imagery, and perception. Behavioral and Brain Sciences. 2004 27 377442.
  • Harnad, S. The symbol grounding problem. Physica D 1990 42 335346.
  • Johnson-Laird, P. N., Herrmann, D. J., Chaffin, R. Only connections: A critique of semantic networks. Psychological Bulletin 1984 96 292315.
  • Kaschak, M. P., Madden, C. J., Therriault, D. J., Yaxley, R. H., Aveyard, M. E., Blanchard, A. A., Zwaan, R. A. Perception of motion affects language processing. Cognition 2005 94 B79B89.
  • Kaschak, M. P., Zwaan, R. A., Aveyard, M. E., Yaxley, R. H. Perception of auditory motion affects language processing. Cognitive Science. 2006 30 733744.
  • Kintsch, W. The representation of knowledge in minds and machines. International Journal of Psychology. 1998 33 411420.
  • Marques, J. M. Specialization and semantic organization: Evidence for multiple-semantics linked to sensory modalities. Memory & Cognition. 2006 34 6067.
  • Martin, A. Functional neuroimaging of semantic memory In Cabeza, R., Kingstone, A. (Eds.), Handbook of functional neuroimaging of cognition 2001 Cambridg, MA: MIT Press 153186.
  • Martin, A. The representation of object concepts in the brain. Annual Review of Psychology. 2007 58 2545.
  • Martin, A., Chao, L. Semantic memory and the brain: Structure and process. Current Opinion in Neurobiology. 2001 11 194201.
  • McNamara, T. P., Miller, D. L. Attributes of theories of meaning. Psychological Bulletin. 1989 106 355376.
  • Meteyard, L., Bahrami, B., Vigliocco, G. Comprehension of motion verbs affects the detection of visual motion: Evidence from psychophysics. Psychological Science. 2007 18 10071013.
    Direct Link:
  • Newell, A., Simon, H. A. Human problem solving 1972 Englewood Cliff, NJ: Prentice-Hall.
  • Nosofsky, R. M. Attention, similarity, and the identification–categorization relationship. Journal of Experimental Psychology: General. 1986 115 3957.
  • Pecher, D., Zanolie, K., Zeelenberg, R. Verifying visual properties in sentence verification facilitates picture recognition memory. Experimental Psychology. 2007 54 173179.
  • Pecher, D., Zeelenberg, R., Barsalou, L. W. Verifying different-modality properties for concepts produces switching costs. Psychological Science. 2003 14 119124.
    Direct Link:
  • Pecher, D., Zwaan, R. A. Grounding cognition: The role of perception and action in memory, language, and thinking 2005 Cambridge, England: Cambridge University Press.
  • Rapaport, W. J. Holism, conceptual-role semantics, and syntactic semantics. Minds and Machines 2002 12 359.
  • Richardson, D. C., Spivey, M. J., Barsalou, L. W., McRae, K. Spatial representations activated during real-time comprehension of verbs. Cognitive Science 2003 27 767780.
  • Solomon, K. O., Barsalou, L. W. Perceptual simulation in property verification. Memory & Cognition. 2004 32 244259.
  • Spence, C., Nicholls, M. E. R., Driver, J. The cost of expecting events in the wrong sensory modality. Perception & Psychophysics. 2001 63 330336.
  • Stanfield, R. A., Zwaan, R. A. The effect of implied orientation derived from verbal context on picture recognition. Psychological Science 2001 12 153156.
    Direct Link:
  • Vermeulen, N., Corneille, O., Budke, S., Niedenthal, P. M. Sensory load incurs conceptual processing costs 2007 Manuscript submitted for publication.
  • Vermeulen, N., Niedenthal, P. M., Luminet, O. Switching between sensory and affective systems incurs processing costs. Cognitive Science. 2007 31 183192.
  • Zeelenberg, R., Pecher, D., Shiffrin, R. M., Raaijmakers, J. G. W. Semantic context effects and priming in word association. Psychonomic Bulletin & Review 2003 10 653660.
  • Zwaan, R. A., Madden, C. J. Embodied sentence comprehension In Pecher, D., Zwaan, R. A. (Eds.), The grounding of cognition: The role of perception and action in memory, language, and thinking 2005 Cambridge England: Cambridge University Press 224245.
  • Zwaan, R. A., Stanfield, R. A., Yaxley, R. H. Language comprehenders mentally represent the shapes of objects. Psychological Science 2002 13 168171.
    Direct Link:

Appendix

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Method
  5. 3. Results
  6. 4. Discussion
  7. Acknowledgments
  8. References
  9. Appendix

List of stimulus materials

DutchEnglishaAuditorybTactileVisualPurityc
  • a

    Because of language differences, English translations do not always perfectly overlap in meaning with the Dutch stimulus materials.

  • b

    Modality rating ranges from 1 (cannot be perceived by this sense at all) to 6 (is perceived by this sense completely).

  • c

    Purity score = rating relevant modality – highest rating of irrelevant modalities.

Auditory stimuli     
   een bij gonstA bee buzzes5.730.270.005.45
   een blokfluit klinkt hoogA flute is high-pitched5.670.000.005.67
   een brommer ronktA scooter hums5.921.750.754.17
   een fietsbel rinkeltA bicycle bell rings6.000.830.335.17
   een kerkorgel galmtA church organ clangs5.750.750.335.00
   een krekel tjirptA cricket chirps5.920.000.675.25
   een saxofoon schaltA saxophone blares5.830.750.085.08
   een scheepshoorn klinkt laagA ship's horn is low-pitched5.831.170.004.67
   een sirene loeitA siren wails5.920.250.085.67
   een stationshal is rumoerigA station hall is noisy5.670.921.334.33
   een tram knarstA tram grinds5.730.640.095.09
   een triangel tinkeltA triangle jingles5.180.730.274.45
   een trompet klinkt schelA trumpet sounds shrill5.920.920.085.00
   een typemachine rateltA typewriter rattles5.670.831.174.50
   een vlieg zoemtA fly buzzes5.920.250.335.58
   een vliegtuig is luidruchtigAn airplane is loud6.001.080.084.92
   een vrachtwagen toetertA truck honks5.820.450.185.36
   een wekker tiktAn alarm clock ticks5.911.000.734.91
   herfstbladeren ritselenAutumn leaves rustle5.911.551.364.36
   kreupelhout kraaktBrushwood crackles5.911.180.364.73
   naaldhakken tikkenHigh heels tap5.750.670.675.08
   onweer dondertThunder rumbles6.001.550.094.45
   pannen kletterenPans clang6.000.550.915.09
   spoorbomen rinkelenA railroad crossing rings5.920.171.004.92
   Average 5.830.760.454.95
   Tactile stimuli     
   badwater is lauwBath water is lukewarm0.005.910.275.64
   een dweil is klamA rag is moist0.005.921.424.50
   een gloeilamp is loeiheetA light bulb is very hot0.006.001.005.00
   een grot is kilA cave is chilly0.005.360.824.55
   een gum is stroefAn eraser is rough0.085.581.334.25
   een knikker is keihardA marble is rock hard0.365.821.644.18
   een kraan is loeiheetA faucet is hot0.005.820.555.27
   een mier kriebeltAn ant tickles0.186.000.555.45
   een muggenbult jeuktA mosquito bite itches0.005.640.645.00
   een muntje is hardA coin is hard0.506.001.584.42
   een sjaal kriebeltA shawl itches0.095.911.094.82
   een sneeuwbal is koudA snowball is cold0.005.920.505.42
   een snoepje is plakkerigA candy is sticky0.275.821.734.09
   een strijkijzer is heetAn iron is hot0.276.001.184.82
   een suikerspin is kleverigCotton candy is sticky0.005.922.003.92
   een theepot is warmA teapot is warm0.185.910.455.45
   een tosti is warmToast is warm0.256.001.334.67
   een veertje kieteltA feather tickles0.006.000.915.09
   een waterval is koelA waterfall is cool0.005.250.674.58
   hagel is koudHail is cold0.005.910.095.82
   jodium priktIodine stings0.005.730.095.64
   regen is frisRain is fresh0.095.360.275.09
   vingers tintelenFingers tingle0.005.550.455.09
   zand kan schurenSand can grind2.175.671.833.50
   Average 0.195.790.934.84
Visual stimuli     
   boter is geligButter is yellowish0.000.005.925.92
   broccoli is groenBroccoli is green0.000.005.925.92
   chocola is donkerbruinChocolate is dark brown0.000.005.915.91
   een aubergine is donkerpaarsAn eggplant is dark purple0.000.005.925.92
   een binnenband is matzwartAn inner tube is black0.000.005.915.91
   een cassettebandje is zwartA cassette tape is black0.000.005.925.92
   een diamant glinstertA diamond glistens0.000.005.915.91
   een eekhoorn is roodbruinA squirrel is red-brown0.000.005.825.82
   een ijsklontje is doorzichtigAn ice cube is transparant0.000.005.825.82
   een kelder is donkerA cellar is dark0.000.086.005.92
   een kwal is doorschijnendA jellyfish is translucent0.000.175.925.75
   een luipaard is gespikkeldA leopard is spotted0.000.005.915.91
   een orka is zwart-witAn orca is black-and-white0.000.005.915.91
   een pepermuntje is witPeppermint is white0.000.005.925.92
   een schaakbord is gebloktA chessboard is checkered0.000.086.005.92
   een scheermesje is zilverkleurigA razorblade is silver0.000.005.925.92
   een tennisbal is geelA tennisball is yellow0.000.005.915.91
   een walnoot is bruinA walnut is brown0.000.005.925.92
   een wesp is gestreeptA wasp is striped0.000.005.825.82
   een zwembad is azuurblauwA swimming pool is azure blue0.000.005.925.92
   ham is rozeHam is pink0.000.005.925.92
   honing is goudgeelHoney is golden-yellow0.000.005.915.91
   mayonaise is lichtgeelMayonaise is light yellow0.000.005.925.92
   spinazie is donkergroenApinach is dark green0.000.006.006.00
   Average 0.000.015.915.90