Lip-Reading Aids Word Recognition Most in Moderate Noise: A Bayesian Explanation Using High-Dimensional Feature Space
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Watching a speaker's facial movements can dramatically enhance our ability to comprehend words, especially in noisy environments. From a general doctrine of combining information from different sensory modalities (the principle of inverse effectiveness), one would expect that the visual signals would be most effective at the highest levels of auditory noise. In contrast, we find, in accord with a recent paper, that visual information improves performance more at intermediate levels of auditory noise than at the highest levels, and we show that a novel visual stimulus containing only temporal information does the same. We present a Bayesian model of optimal cue integration that can explain these conflicts. In this model, words are regarded as points in a multidimensional space and word recognition is a probabilistic inference process. When the dimensionality of the feature space is low, the Bayesian model predicts inverse effectiveness; when the dimensionality is high, the enhancement is maximal at intermediate auditory noise levels. When the auditory and visual stimuli differ slightly in high noise, the model makes a counterintuitive prediction: as sound quality increases, the proportion of reported words corresponding to the visual stimulus should first increase and then decrease. We confirm this prediction in a behavioral experiment. We conclude that auditory-visual speech perception obeys the same notion of optimality previously observed only for simple multisensory stimuli.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Humans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · cited 1×
“In contrast, Bayes-optimal cue integration can be linked to physiology in a rigorous way, using the formalism of probabilistic population codes [81,82,83,84].”
MethodVisual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · cited 3×
“Therefore, with the vocabulary sizes used in earlier studies (e.g. at most 256 words in [8]), it is not surprising that inverse effectiveness was observed.”
MethodResultThe Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936 · cited 2×
“Numerous studies have shown that humans are nearly Bayes-optimal in combining simple perceptual cues, even in the presence of a small conflict between the cues [16,17,18,46,47,48,49,50,51,52], sensorimotor integration [53,54], and other forms of cue combination [55,56,57,58,59,60].”
BackgroundBayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457 · cited 1×
“In contrast, Bayes-optimal cue integration can be linked to physiology in a rigorous way, using the formalism of probabilistic population codes [81,82,83,84].”
MethodSelf-citationCausal Inference in Multisensory Perception
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B., & Shams, L. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000943 · Cited by 1,069 · cited 2×
“Numerous studies have shown that humans are nearly Bayes-optimal in combining simple perceptual cues, even in the presence of a small conflict between the cues [16,17,18,46,47,48,49,50,51,52], sensorimotor integration [53,54], and other forms of cue combination [55,56,57,58,59,60].”
BackgroundSelf-citationIntegration of Proprioceptive and Visual Position-Information: An Experimentally Supported Model
van Beers, R. J., Sittig, A. C., & Gon, J. J. D. V. D. (1999) · Journal of Neurophysiology · doi:10.1152/jn.1999.81.3.1355 · Cited by 742 · cited 2×
“Numerous studies have shown that humans are nearly Bayes-optimal in combining simple perceptual cues, even in the presence of a small conflict between the cues [16,17,18,46,47,48,49,50,51,52], sensorimotor integration [53,54], and other forms of cue combination [55,56,57,58,59,60].”
BackgroundCombining Sensory Information: Mandatory Fusion Within, but Not Between, Senses
Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002) · Science · doi:10.1126/science.1075396 · Cited by 486
Evaluating the Operations Underlying Multisensory Integration in the Cat Superior Colliculus
Stanford, T. R., Quessy, S., & Stein, B. E. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.5095-04.2005 · Cited by 314
Cited by
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620
Multisensory Processing in Review: from Physiology to Behaviour
Alais, D., Newell, F., & Mamassian, P. (2010) · Seeing and Perceiving · doi:10.1163/187847510x488603 · Cited by 324
Neural Coding of Uncertainty and Probability
Ma, W. J., & Jazayeri, M. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071013-014017 · Cited by 290
“Analogous conclusions were reached in a comparable auditory task (Maiworm et al. 2011), a spatial reasoning task in natural scenes (D'Antona et al. 2013), an auditory-visual causal inference task (Kording et al. 2007), and speech perception studies (Ma et al. 2009, Bejjanki et al. 2011).”
BackgroundSelf-citationFoxe, J. J., Molholm, S., Del Bene, V. A., Frey, H.-P., Russo, N. N., Blanco, D., Saint-Amour, D., & Ross, L. A. (2015) · Cerebral Cortex · doi:10.1093/cercor/bht213 · Cited by 263
Self-citationBrandwein, A. B., Foxe, J. J., Butler, J. S., Russo, N. N., Altschuler, T. S., Gomes, H., & Molholm, S. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhs109 · Cited by 236
“…inputs from multiple sensory modalities to oftentimes dramatically influence perception and performance (Wallace et al. 2004; Ross et al. 2007a; Ma et al. 2009; Fiebelkorn et al. 2011), dysfunctional patterns of brain connectivity, as proposed for ASD, should lead to deficits in the…”
BackgroundSelf-citation
Metadata
- Authors
- Ma, W. J., Zhou, X., Ross, L. A., Foxe, J. J., & Parra, L. C.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2009-03-04
- Volume
- 4
- Issue
- 3
- Pages
- e4638
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2088079940
- Cited by
- 189
- References
- 14
- FWCI
- 13.66
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Lip-Reading Aids Word Recognition Most in Moderate Noise: A Bayesian Explanation Using High-Dimensional Feature Space. https://app.cochat.ai/papers/10.1371/journal.pone.0004638
- The paper
- Ma, W. J., Zhou, X., Ross, L. A., Foxe, J. J., & Parra, L. C. (2009). Lip-Reading Aids Word Recognition Most in Moderate Noise: A Bayesian Explanation Using High-Dimensional Feature Space. PLoS ONE. https://doi.org/10.1371/journal.pone.0004638
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.