Causal inference of asynchronous audiovisual speech
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
During speech perception, humans integrate auditory information from the voice with visual information from the face. This multisensory integration increases perceptual precision, but only if the two cues come from the same talker; this requirement has been largely ignored by current models of speech perception. We describe a generative model of multisensory speech perception that includes this critical step of determining the likelihood that the voice and face information have a common cause. A key feature of the model is that it is based on a principled analysis of how an observer should solve this causal inference problem using the asynchrony between two cues and the reliability of the cues. This allows the model to make predictions about the behavior of subjects performing a synchrony judgment task, predictive power that does not exist in other approaches, such as post-hoc fitting of Gaussian curves to behavioral data. We tested the model predictions against the performance of 37 subjects performing a synchrony judgment task viewing audiovisual speech under a variety of manipulations, including varying asynchronies, intelligibility, and visual cue reliability. The causal inference model outperformed the Gaussian model across two experiments, providing a better fit to the behavioral data with fewer parameters. Because the causal inference model is derived from a principled understanding of the task, model parameters are directly interpretable in terms of stimulus and subject properties.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · cited 1×
“When there is a single talker, integrating the auditory and visual speech information increases perceptual accuracy (Sumby and Pollack, 1954; Rosenblum et al., 1996; Schwartz et al., 2004; Ma et al., 2009); most computational work on audiovisual integration of speech has focused on this condition…”
BackgroundCausal Inference in Multisensory Perception
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B., & Shams, L. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000943 · Cited by 1,069 · cited 6×
“The CIMS model affords a quantitative grounding for multisensory speech perception that recognizes the fundamental role of causal inference in general multisensory integration (Kording et al., 2007; Sato et al., 2007; Schutz and Kubovy, 2009; Shams and Beierholm, 2010; Buehner, 2012).”
BackgroundMethodInfluentialSelf-citationUnraveling multisensory integration: patchy organization within human STS multisensory cortex
Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004) · Nature Neuroscience · doi:10.1038/nn1333 · Cited by 513 · cited 2×
“For instance, audiovisual spatial localization likely occurs in the parietal lobe (Zatorre et al., 2002) while multisensory speech perception is thought to occur in the superior temporal sulcus (Beauchamp et al., 2004).”
BackgroundSelf-citationAn extended multisensory temporal binding window in autism spectrum disorders
Foss-Feig, J. H., Kwakye, L. D., Cascio, C. J., Burnette, C. P., Kadivar, H., Stone, W. L., & Wallace, M. T. (2010) · Experimental Brain Research · doi:10.1007/s00221-010-2240-4 · Cited by 411
Causal inference in perception
Shams, L., & Beierholm, U. R. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.07.001 · Cited by 403
Stevenson, R. A., Zemtsov, R. K., & Wallace, M. T. (2012) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/a0027339 · Cited by 269 · cited 1×
“…stimuli (congruent audiovisual words with high or low visual reliability and intelligibility), subjects with high sensory noise might show differences from subjects with low sensory noise across a range of multi-sensory speech tasks, such as perception of the McGurk effect (Stevenson et al., 2012).”
Background
Cited by
Robertson, C. E., & Baron-Cohen, S. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2017.112 · Cited by 1,157
Behavioral, Neural, and Computational Principles of Bodily Self-Consciousness
Blanke, O., Slater, M., & Serino, A. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.029 · Cited by 698
Baum, S. H., Stevenson, R. A., & Wallace, M. T. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.007 · Cited by 370
Perception of Body Ownership Is Driven by Bayesian Sensory Inference
Samad, M., Chung, A. J., & Shams, L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0117178 · Cited by 315
“A Bayesian causal inference model [13–18] has been shown to successfully account for a variety of human multisensory perceptual phenomena, and a recent human fMRI study has provided further support for the brain carrying out this type of computation [19].”
BackgroundNeural Coding of Uncertainty and Probability
Ma, W. J., & Jazayeri, M. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071013-014017 · Cited by 290
“…and varying reliability has been extended to multiple-item categorical tasks, namely visual search (Ma et al. 2011), change detection (Keshvari et al. 2012), oddity detection (Hillis et al. 2002; as modeled by Hospedales & Vijayakumar 2009), and simultaneity judgment (Magnotti et al. 2013).”
BackgroundSelf-citation
Metadata
- Authors
- Magnotti, J. F., Ma, W. J., & Beauchamp, M. S.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2013
- Volume
- 4
- Pages
- 798-798
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W1993930228
- Cited by
- 96
- References
- 6
- FWCI
- 7.00
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.