A Rational Analysis of the Acquisition of Multisensory Representations
Version of Record online: 5 DEC 2011
Copyright © 2011 Cognitive Science Society, Inc.
Volume 36, Issue 2, pages 305–332, March 2012
How to Cite
Yildirim, I. and Jacobs, R. A. (2012), A Rational Analysis of the Acquisition of Multisensory Representations. Cognitive Science, 36: 305–332. doi: 10.1111/j.1551-6709.2011.01216.x
- Issue online: 5 MAR 2012
- Version of Record online: 5 DEC 2011
- Received 21 January 2011; received in revised form 18 May 2011; accepted 23 May 2011
- Multisensory perception;
- Bayesian modeling;
- Rational analysis
How do people learn multisensory, or amodal, representations, and what consequences do these representations have for perceptual performance? We address this question by performing a rational analysis of the problem of learning multisensory representations. This analysis makes use of a Bayesian nonparametric model that acquires latent multisensory features that optimally explain the unisensory features arising in individual sensory modalities. The model qualitatively accounts for several important aspects of multisensory perception: (a) it integrates information from multiple sensory sources in such a way that it leads to superior performances in, for example, categorization tasks; (b) its performances suggest that multisensory training leads to better learning than unisensory training, even when testing is conducted in unisensory conditions; (c) its multisensory representations are modality invariant; and (d) it predicts ‘‘missing” sensory representations in modalities when the input to those modalities is absent. Our rational analysis indicates that all of these aspects emerge as part of the optimal solution to the problem of learning to represent complex multisensory environments.