Cue Integration in Categorical Tasks: Insights from Audio-Visual Speech Perception
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Previous cue integration studies have examined continuous perceptual dimensions (e.g., size) and have shown that human cue integration is well described by a normative model in which cues are weighted in proportion to their sensory reliability, as estimated from single-cue performance. However, this normative model may not be applicable to categorical perceptual dimensions (e.g., phonemes). In tasks defined over categorical perceptual dimensions, optimal cue weights should depend not only on the sensory variance affecting the perception of each cue but also on the environmental variance inherent in each task-relevant category. Here, we present a computational and experimental investigation of cue integration in a categorical audio-visual (articulatory) speech perception task. Our results show that human performance during audio-visual phonemic labeling is qualitatively consistent with the behavior of a Bayes-optimal observer. Specifically, we show that the participants in our task are sensitive, on a trial-by-trial basis, to the sensory uncertainty associated with the auditory and visual cues, during phonemic categorization. In addition, we show that while sensory uncertainty is a significant factor in determining cue weights, it is not the only one and participants' performance is consistent with an optimal model in which environmental, within category variability also plays a role in determining cue weights. Furthermore, we show that in our task, the sensory variability affecting the visual modality during cue-combination is not well estimated from single-cue performance, but can be estimated from multi-cue performance. The findings and computational principles described here represent a principled first step towards characterizing the mechanisms underlying human cue integration in categorical tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Humans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844
Control Methods Used in a Study of the Vowels
Peterson, G. E., & Barney, H. L. (1952) · The Journal of the Acoustical Society of America · doi:10.1121/1.1906875 · Cited by 2,837 · cited 1×
“It is well known that vowel productions differ considerably both within and between talkers [44] and this variability may be greater than the production variance for the consonants in the /ba/ and /da/ syllables.”
BackgroundThe Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · cited 1×
“modality or due to noise or variability in the environment [21].”
BackgroundSelf-citationVisual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583
The Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936 · cited 1×
“This is an assumption that has been made by most previous studies of cue integration [20,24,25,27,28,30,31].”
ResultMerging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906 · cited 2×
“Under certain assumptions [1,22,23], a statistically optimal mechanism for combining multiple uncertain cues is equivalent to using a linear combination rule where the estimate from each cue is weighted by its relative uncertainty.”
BackgroundCausal Inference in Multisensory Perception
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B., & Shams, L. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000943 · Cited by 1,069 · cited 1×
“A large body of prior work [39–41] has shown that when there are large cue conflicts in a cue integration task, participants would need to solve the ‘‘cue source’’ problem in addition to the cue integration problem.”
BackgroundIntegration of Proprioceptive and Visual Position-Information: An Experimentally Supported Model
van Beers, R. J., Sittig, A. C., & Gon, J. J. D. V. D. (1999) · Journal of Neurophysiology · doi:10.1152/jn.1999.81.3.1355 · Cited by 742
Combining Sensory Information: Mandatory Fusion Within, but Not Between, Senses
Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002) · Science · doi:10.1126/science.1075396 · Cited by 486
Visual speech contributes to phonetic learning in 6-month-old infants
Teinonen, T., Aslin, R. N., Alku, P., & Csibra, G. (2008) · Cognition · doi:10.1016/j.cognition.2008.05.009 · Cited by 287 · cited 1×
“The influence of visual information on the perception of auditory information has also been observed in children [11–13], across languages [14–16] and even in pre-linguistic infants [17– 19].”
BackgroundSelf-citation
Cited by
What do we mean by prediction in language comprehension?
Kuperberg, G. R., & Jaeger, T. F. (2016) · Language, Cognition and Neuroscience · doi:10.1080/23273798.2015.1102299 · Cited by 1,087
“This is also true of language processing (e.g. Bejjanki et al., 2011; Chater, Crocker, & Pickering, 1998; Clayards, Tanenhaus, Aslin, & Jacobs, 2008; Feldman et al., 2009; Kleinschmidt & Jaeger, 2015; Levy, 2008; Norris, 2006; Norris & McQueen, 2008; see also Crocker & Brants, 2000; Hale, 2001;…”
Robust speech perception: Recognize the familiar, generalize to the similar, and adapt to the novel.
Kleinschmidt, D. F., & Jaeger, T. F. (2015) · Psychological Review · doi:10.1037/a0038695 · Cited by 635
Neural Coding of Uncertainty and Probability
Ma, W. J., & Jazayeri, M. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071013-014017 · Cited by 290
“Analogous conclusions were reached in a comparable auditory task (Maiworm et al. 2011), a spatial reasoning task in natural scenes (D'Antona et al. 2013), an auditory-visual causal inference task (Kording et al. 2007), and speech perception studies (Ma et al. 2009, Bejjanki et al. 2011).”
Background(2011) · Cited by 38
Metadata
- Authors
- Bejjanki, V. R., Clayards, M., Knill, D. C., & Aslin, R. N.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-05-26
- Volume
- 6
- Issue
- 5
- Pages
- e19812
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2135220764
- Cited by
- 59
- References
- 13
- FWCI
- 2.37
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Cue Integration in Categorical Tasks: Insights from Audio-Visual Speech Perception. https://app.cochat.ai/papers/10.1371/journal.pone.0019812
- The paper
- Bejjanki, V. R., Clayards, M., Knill, D. C., & Aslin, R. N. (2011). Cue Integration in Categorical Tasks: Insights from Audio-Visual Speech Perception. PLoS ONE. https://doi.org/10.1371/journal.pone.0019812
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.