Perspectives on multisensory experience and cognitive development in infants with cochlear implants


Mary K. Fagan, Department of Communication Science and Disorders, 303 Lewis Hall, University of Missouri, Columbia, MO 65211, USA. E-mail:


Infants learn about their environment through sensory exploration, acquiring knowledge that is important for cognitive development. However, little is known about the sensory exploration of infants with profound hearing loss before or after they receive cochlear implants. This paper reviews aspects of sensory perception and cognitive development in hearing infants, discusses the implications of delayed access to auditory information for multisensory perception and cognitive development in infants who use cochlear implants, and suggests several new directions for research addressing multisensory exploration and cognitive development in infants with cochlear implants.

Early cognitive development relies heavily on the sensory information infants acquire as they observe, explore and interact with their environment (Piaget, 1952). This paper reviews several selected aspects of sensory perception and cognitive development in hearing infants and discusses the implications of delayed access to auditory information for multisensory perception and cognitive development in infants with profound hearing loss who receive cochlear implants. Additionally, we suggest several new directions for research addressing substantial gaps in our understanding of multisensory exploration and cognitive development in infants with cochlear implants.

Sensory information and cognitive development

Infants learn about their complex surroundings from the sensory information they receive (Bushnell & Boudreau, 1993; Gibson, 1979; Oakes & Tellinghuisen, 1994; Piaget, 1952; Ruff, 1986). The fundamental information that infants obtain through sensory perception and exploration of their environment contributes to the learning and development of important cognitive concepts (see Rakison & Woodward, 2008). As examples, visual perceptual information shapes infants’ cognitive representations of motion, depth, and event sequencing; manual activities and tactile exploration are rich sources of information about shape, texture, consistency, object properties, and the development of object representations (Perone, Madole, Ross-Sheehy, Carey & Oakes, 2008; Ruff, 1984, 1986); and auditory signals from the linguistic community contribute to word learning and joint attention (Mandel, Jusczyk & Pisoni, 1995). The unique contribution of sensory stimuli from each of these sensory modalities maximizes perceptual learning and cognitive development.

In processing events under natural circumstances, infants are typically exposed to several streams of sensory information from multiple modalities simultaneously, rather than information constrained to a single modality (Gaver, 1993; Gibson, 1979; Lewkowicz & Kraebel, 2004; Lickliter & Bahrick, 2004). They make use of the concurrent and often redundant sensory information they receive, combining and integrating stimuli from multiple modalities (Lewkowicz & Kraebel, 2004). From an early age, infants demonstrate a remarkable ability to process and integrate sensory experiences to form cross-modal associations between various forms of sensory stimuli. During the first weeks of life, for example, one-month-old infants demonstrate visual recognition of shapes and object properties explored only orally (Gibson & Walker, 1984; Meltzoff & Borton, 1979). At 4.5 months, infants demonstrate that they recognize incongruencies between auditory and visual properties of spoken vowel productions, looking longer toward speakers articulating vowels that match the auditory stimuli they hear than to those that do not (Kuhl & Meltzoff, 1982, 1984; Patterson & Werker, 1999). Thus, young infants can integrate and recall relevant multisensory information in processing and recognizing objects and events that generate multiple forms of simultaneous sensory stimulation.

Relations between sensory perception and behavior

Traditional investigations of multisensory perception focus on behavioral responses to sensory stimuli presented under controlled experimental conditions (see Lewkowicz and Kraebel, 2004). However, infants themselves increasingly engage in self-directed multisensory exploration of their own environment as their growing motor competence allows. Self-directed exploratory activities and the multi-modal perceptual opportunities they create provide infants with important additional information about the world within their reach. For example, 5-month-old infants often engage in repetitive exploratory behaviors with objects, alternately putting objects in their mouths for oral exploration and bringing them into view for visual inspection. These repetitive inspections appear to facilitate infants’ ability to compare visual and tactile information and develop cross-modal links between sensory experiences in both modalities (Rochat, 1989; Ruff, Saltarelli, Capozzoli & Dubiner, 1992).

Moreover, potential opportunities to generate and experience multiple forms of sensory stimulation appear to directly affect infants’ selection of exploratory behavior patterns (Gibson & Walker, 1984). For example, infants engaged in more frequent and varied manipulatory behaviors when exploring objects in a lighted room that permitted both visual and tactile sensory feedback than in a darkened room that permitted only tactile feedback. Thus, self-generated exploratory activities play an important role in expanding infants’ perceptual experiences and fueling their cognitive growth (Gaver, 1993; Gibson & Walker, 1984; Piaget, 1952).

Infants’ perceptual experiences contribute not only to the development of a variety of important cognitive concepts, but also to a broad range of observable behaviors that reflect what infants know and what they are in the process of learning (Adolph, 2008; Adolph, Tamis-LeMonda, Ishak, Karasik & Lobo, 2008; Boysson-Bardies, Halle, Sagart, and Durand, 1989). Clear and direct effects of perceptual experience on specific behaviors have been identified and measured in interesting and novel ways. For example, Adolph and her colleagues have shown that perceptual experience from a locomotor perspective directly influences infants’ behavioral response to risky slopes (Adolph et al., 2008). That is, the amount and type of locomotor experience (i.e., crawling vs. walking) 12-month-old infants had accrued affected their attempts to navigate steep and shallow slopes. Infants who were experienced crawlers (but did not yet walk) refused to crawl down the steep slopes; however, new walkers attempted to walk down risky slopes even when actively discouraged by their mothers from doing so (Adolph et al., 2008). Thus, the novice walkers showed little evidence of having acquired (or transferred from their crawling experience) the perceptual ability to assess risk from the perspective of their newly acquired status as a walker. As further evidence of the strong relationship between locomotor perceptual experience and behavior, relatively experienced walkers were less likely to attempt the risky slopes than were inexperienced walkers (Adolph et al., 2008). These studies of locomotor behavior show that infants must learn to incorporate new perceptual information with multisensory information accumulated through prior perceptual experience to understand their environment and refine their behavior (Adolph, 2008; Adolph et al., 2008; Lewkowicz and Kraebel, 2004).

auditory perception and its influence on behavior

Somewhat analogously, infants with cochlear implants must learn to incorporate the new auditory information they receive from their implants with multisensory information accumulated through prior perceptual experience without their implants. Early auditory perceptual experience, the focus of our discussion of sensory perception and cognitive development in infants with hearing loss, has an unmistakable influence on infant behavior. We briefly review well-documented evidence of the influence of auditory experience on behavior in hearing infants and then discuss the implications for infants with cochlear implants.

The influence of auditory perceptual experience on behavior is especially evident in the early vocal behaviors of hearing infants. Relations between hearing and the onset and quality of canonical babbling (summarized below) are well documented; however, auditory perceptual experience also influences vocal productions in other ways. For example, 10-month-old infants from French, Swedish, and Arabic communities consistently produced the types of consonants and vowels that were specific to the language they heard in their communities (Boysson-Bardies et al., 1989). Moreover, French adults listening to audiotapes of French, Arabic, and Cantonese infants correctly identified the French babies from their vocalizations alone (Boysson-Bardies, Sagart & Durand, 1984). Thus, early auditory experience with their native language has been shown to directly influence the vocal behaviors of infants from very different language communities.

For hearing infants, auditory perception is important not only for the development of linguistic awareness and language learning (e.g., Mandel et al., 1995; Saffran, Aslin & Newport, 1996), but also for sound localization, voice recognition, and establishing predictive relationships between sounds, objects, and events (Perone et al., 2008; Shinskey, 2008). For example, as early as 6 months of age, sound cues facilitated infants’ search for objects in the dark (Shinskey, 2008). Objects and the sounds that objects produce are especially salient to infants (Perone et al., 2008). Infants selectively attended to object affordances for sound production during object play, manipulating, exploring, and mouthing objects that produced sound more often than objects that did not (Fenson, Kagan, Kearsley & Zelazo, 1976; McCall, 1974; Palmer, 1989). Further, infants extended their interest in auditory play with objects to the effects of object mouthing on simultaneous vocalizations (Fagan & Iverson, 2007). As a result, consonants produced during mouthing were more varied than those produced in non-mouthing vocalizations.

Together these studies have shown that observing the exploratory activities in which infants engage and the contexts in which they occur can uncover behavioral evidence of the sensory information infants seek and the stimuli to which they attend. The actions of hearing infants consistently highlight their interest in and attention to auditory information -- auditory information found in their language communities, auditory feedback created by their actions with objects, and auditory variations in their own vocalizations (Fagan & Iverson, 2007; Lansink, Mintz & Richards, 2000; Palmer, 1989; Perone et al., 2008; Ruff, 1986; Stark, 1980). Auditory signals convey complex information about temporal patterns, events, the characteristics of a sound source, and the medium in which it travels (Gaver, 1993). Attention to the various sounds and auditory effects in their environment, and to the auditory consequences of their own actions, provides hearing infants with the multisensory experiences they need to establish cause and effect relationships, understand links between perception and action, and acquire new concepts that depend on sound perception (Piaget, 1952; Ruff et al., 1992).

Effects of hearing loss on behavior and development

Infants with hearing loss differ in several important ways from hearing infants in the extent to which they perceive and benefit from auditory information. However, for infants with profound hearing loss, most auditory information is inaccessible. Thus, whereas hearing infants receive auditory and visual information during speech perception, profoundly deaf infants do not experience the auditory signal. Therefore, when presented with spoken speech perception tasks, children and adults with profound hearing loss must rely on information encoded solely in the visual modality. Moreover, it is widely accepted that profound hearing loss significantly impacts early vocal productions and spoken language development. Infants with profound hearing loss have limited consonant repertoires, absent or delayed canonical babbling, little if any repetitive babbling, and subsequent delays in spoken vocabulary development even after cochlear implantation (Fagan, Pisoni, Horn & Dillon, 2007; Kent, Osberger, Netsell & Hustedde, 1987; Koopmans-van Beinum, Clement & van den Dikkenberg-Pot, 2001; Oller & Eilers, 1988; Thal, DesJardin & Eisenberg, 2007).

Whereas profound hearing loss has a marked effect on infant vocalizations and spoken language development, deficits in auditory perception are also likely to affect a broad spectrum of behaviors and cognitive processes beyond speech development. For example, infants with profound hearing loss may differ from hearing infants in the types of object-related behaviors in which they engage and the sensory stimuli to which they attend. However, little is currently known about the broad effects of reduced auditory perceptual acuity on infant behavior and development. Therefore, new research investigating sensory exploration and cognitive development in infants with hearing loss is important both for understanding infant behavior and for understanding changes in developmental patterns when sensory input is altered during early development.

Lickliter and Bahrick (2004) suggested that experiencing uncoupled multimodal sensory input (e.g., auditory and visual), as happens when the auditory modality is inaccessible, results in altered neural development, perceptual organization and developmental trajectories. In support, Campbell and MacSweeney (2004) found different patterns of brain activation for deaf and hearing adults during speech perception. In deaf adults who used sign language, auditory cortex, typically associated with spoken language processing in hearing individuals, became active during processing of nonlinguistic visual movements. Other research studies have found similar evidence of differences in cortical activation and neural organization for spoken versus signed language processing in deaf adults (Newman, Bavelier, Corina, Jezzard & Neville, 2002).

Language learning with cochlear implants

One of the results of growing numbers of newborn hearing screening programs is that many infants identified with profound hearing loss now receive cochlear implants at an early age. When implant surgery is performed during infancy or early childhood, central auditory pathways often begin to function soon after cochlear implantation (Sharma, Dorman & Spahr, 2002). In general, the auditory cortical responses of children with cochlear implants resemble those of hearing children when children receive a cochlear implant before 3 years of age. In contrast, the cortical responses of later-implanted individuals differ substantially from those of hearing children and adults. Later-implanted individuals show evidence of marked central auditory system reorganization and atypical patterns of cortical activation beyond the auditory cortex (Doucet, Bergeron, Lassonde, Ferron & Lepore, 2006; Gilley, Sharma & Dorman, 2008).

The auditory signal provided by cochlear implants is highly degraded, varying in quality in relation to the number of electrodes inserted and other factors. However, many children perform quite well with the electrical signal they receive from their cochlear implants, often continuing to improve with length of implant use (Schauwers, Gillis, Daemers et al., 2004; Schorr, Fox, van Wassenhove & Knudsen, 2005). Although cochlear implants help deaf individuals to perceive sound, implant surgery alone cannot restore deficits in auditory experience that accrue during the pre-implant period. These deficits in auditory experience often continue to affect infant development even after cochlear implant surgery. For example, with little early bimodal speech perception experience, many children with cochlear implants fail to show bimodal fusion in processing incongruent audio-visual stimuli. Instead, they rely on their early visual experience, often weighting visual cues more heavily than auditory cues in making perceptual decisions on speech perception tasks (Bergeson & Pisoni, 2004; Bergeson, Pisoni & Davis, 2005; Rouger, Fraysse, Deguine & Barone, 2008; Schorr et al., 2005).

Deficits in early auditory experience can also affect spoken vocabulary development following cochlear implantation. In deaf children with cochlear implants, receptive and expressive vocabulary scores are often below average even after several years of implant use (Blamey, Sarant, Paatsch et al., 2001; Connor, Craig, Raudenbush, Heavner & Zwolan, 2006; Connor, Hieber, Arts & Zwolan, 2000; El-Hakim, Levasseur, Papsin et al., 2001; Fagan et al., 2007; Kirk, Miyamoto, Ying, Perdew & Zuganelis, 2000; Thal et al., 2007). Thus, research that investigates learning and development in infants with cochlear implants is important for understanding both the developmental effects of delayed access to auditory information and the changes that occur when novel sensory stimulation is introduced with cochlear implants.

New directions in research

New research is needed to address questions regarding relations between hearing loss and infant behavior as well as changes in behavior and development following access to auditory information through cochlear implantation. In addition to addressing questions about behavior and development in relation to cochlear implant use, new research will stimulate testable hypotheses and theoretical discussions about the role of auditory information and perceptual experience in infants’ learning and cognitive development. In order to advance these research goals, new methodologies appropriate for behavioral research with young infants must also be developed. Future methodological needs include research paradigms appropriate for investigating multisensory exploration, sensory integration, and the influence of exploration and sensory stimulation on behavior and cognitive development. Research focused on sensory experience must also address questions concerning how infants with profound hearing loss perceive and interact with their environment in the absence of auditory information, including the effects of hearing loss on attention and exploration (see Bavelier, Dye & Hauser, 2006; Boutla, Supalla, Newport & Bavelier, 2004; Dye, Baril & Bavelier, 2007).

Research underway in our laboratory is currently investigating a measure of infant vocalization recently used in a study of hearing infants (Fagan, 2009) but not previously used in assessing the vocalizations of infants with hearing loss. This measure documents the number of sound and syllable repetitions per vocalization that infants produce in their spontaneous vocalizations. In research with hearing infants (Fagan, 2009), this measure of repetition reached a peak between 6 and 12 months, the established time of similar peaks in number of repetitions produced in object exploration (i.e., shaking and banging; Locke, Bekken, McMinn-Larson & Wein, 1995). These results led to the hypothesis that both measures of repetition (i.e., vocal repetition and repetitive behaviors with objects) may reflect infants’ general cognitive interest in sound exploration and production. We are currently testing this hypothesis in research with infants with profound hearing loss before and after cochlear implantation.

Novel research has recently shown that cortical pathways important for speech and language development were formed during this established time of peak repetition between 6 and 12 months (Imada, Zhang, Cheour, Taulu, Ahonen & Kuhl, 2006). Imada et al. (2006) measured infants’ brain activation patterns using magnetoencephalography and found that neural connections between cortical areas responsible for syllable perception and syllable production were formed during the second half of the first year. Therefore, repetitive vocal and exploratory behaviors motivated by infants’ interest in sound exploration may provide infants with redundant sensory experiences that contribute not only to establishing fundamental links between action and perception but also to neural connections between cortical areas critical to speech and spoken language development. Whereas hearing infants form neural pathways between cortical areas responsible for syllable perception and syllable production between 6 and 12 months (Imada et al., 2006), the timing and formation of similar pathways in infants who receive cochlear implants is currently unknown. Important auditory pathways often begin to function soon after cochlear implantation (Sharma et al., 2002); however, the timing of links to cortical areas responsible for speech production has not been established.

Recently, Lickliter and Bahrick (2004) suggested that simultaneous presentation of temporally synchronous information in two or more sensory modalities (e.g., auditory-visual), selectively recruits infant attention and facilitates perceptual learning and organization. For hearing infants, temporally synchronous information in auditory and visual modalities is both prevalent and developmentally important. However, infants with profound hearing loss process many, if not all, auditory-visual events through the visual modality alone. Unimodal sensory processing and encoding of these events and the development of bimodal processing after cochlear implantation is not well understood. Whereas multisensory redundancy and temporally concurrent information facilitate early auditory-visual learning (Lickliter & Bahrick, 2004; Lewkowicz & Kraebel, 2004), infants with profound hearing loss who do not receive concurrent auditory information but later receive cochlear implants face a more challenging task in learning to integrate and attend to bimodal auditory-visual cues. Sensory perception in infants with profound hearing loss requires substantial new research, including research that addresses the possible benefits that may accrue from unimodal sensory perception and those that accrue from cochlear implantation.

Because spoken language and cognitive development depend on perceiving and understanding both speech and non-speech sounds (e.g., Mandel et al., 1995; Saffran et al., 1996), future research should investigate linguistic and non-linguistic auditory stimuli. Selected auditory attention and experience with both forms of stimuli prepares hearing infants to make sense of the sounds and words they encounter in their environment (Bradley-Johnson, Friedrich & Wyrembelski, 1981; Rochat, 1989; Ruff, 1984). Thus, studying infants’ attention to auditory stimuli will provide important information about their perceptual experiences, their interest in sounds, and the sensory focus of their attention and learning. Because infants’ actions can show where they direct their attention and what they perceive (Lansink et al., 2000; Oakes & Tellinghuisen, 1994; Palmer, 1989; Perone et al., 2008; Ruff, 1986), studying the actions of infants with hearing loss will help to identify the sensory focus of their learning and attention. For example, noting active periods of vocal repetition and sound exploration in infants who receive cochlear implants may shed light on fundamental processes underlying the development of auditory attention and awareness after cochlear implantation.

In summary, early sensory and cognitive development in infants with hearing loss, and the behavioral and neurophysiological processes underlying development, are not well understood. However, future research investigating sensory behavior in infants with hearing loss and changes in behavior following cochlear implantation will advance cognitive developmental theory and contribute to new methods for investigating the developmental effects of profound hearing loss.


This research was supported by NIH-NIDCD Grants T32 DC00012 and R01 DC 000064 to Indiana University.