The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
The Natural Statistics of Audiovisual Speech
Open accessAbstract
Humans, like other animals, are exposed to a continuous stream of signals, which are dynamic, multimodal, extended, and time varying in nature. This complex input space must be transduced and sampled by our sensory systems and transmitted to the brain where it can guide the selection of appropriate actions. To simplify this process, it's been suggested that the brain exploits statistical regularities in the stimulus space. Tests of this idea have largely been confined to unimodal signals and natural scenes. One important class of multisensory signals for which a quantitative input space characterization is unavailable is human speech. We do not understand what signals our brain has to actively piece together from an audiovisual speech stream to arrive at a percept versus what is already embedded in the signal structure of the stream itself. In essence, we do not have a clear understanding of the natural statistics of audiovisual speech. In the present study, we identified the following major statistical features of audiovisual speech. First, we observed robust correlations and close temporal correspondence between the area of the mouth opening and the acoustic envelope. Second, we found the strongest correlation between the area of the mouth opening and vocal tract resonances. Third, we observed that both area of the mouth opening and the voice envelope are temporally modulated in the 2-7 Hz frequency range. Finally, we show that the timing of mouth movements relative to the onset of the voice is consistently between 100 and 300 ms. We interpret these data in the context of recent neural theories of speech which suggest that speech communication is a reciprocally coupled, multisensory event, whereby the outputs of the signaler are matched to the neural processes of the receiver.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Neuronal Oscillations in Cortical Networks
Buzsáki, G., & Draguhn, A. (2004) · Science · doi:10.1126/science.1099745 · Cited by 6,576
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Speech Recognition with Primarily Temporal Cues
Shannon, R. V., Zeng, F.-G., Kamath, V., Wygonski, J., & Ekelid, M. (1995) · Science · doi:10.1126/science.270.5234.303 · Cited by 3,024
High Gamma Power Is Phase-Locked to Theta Oscillations in Human Neocortex
Canolty, R. T., Edwards, E., Dalal, S. S., Soltani, M., Nagarajan, S. S., Kirsch, H. E., Berger, M. S., Barbaro, N. M., & Knight, R. T. (2006) · Science · doi:10.1126/science.1128115 · Cited by 2,650
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,508
Natural Image Statistics and Neural Representation
Simoncelli, E. P., & Olshausen, B. A. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.1193 · Cited by 2,460
Is neocortex essentially multisensory?
GHAZANFAR, A., & SCHROEDER, C. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.04.008 · Cited by 1,463
Self-citationPoeppel, D. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00107-3 · Cited by 1,315
(1991) · Choice Reviews Online · doi:10.5860/choice.29-0605 · Cited by 1,284
Lakatos, P., Shah, A. S., Knuth, K. H., Ulbert, I., Karmos, G., & Schroeder, C. E. (2005) · Journal of Neurophysiology · doi:10.1152/jn.00263.2005 · Cited by 1,237
Pfeifer, R., & Scheier, C. (1999) · The MIT Press eBooks · doi:10.7551/mitpress/6979.001.0001 · Cited by 1,155
An audio-visual corpus for speech perception and automatic speech recognition
Cooke, M., Barker, J., Cunningham, S., & Shao, X. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2229005 · Cited by 1,144
Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex
Luo, H., & Poeppel, D. (2007) · Neuron · doi:10.1016/j.neuron.2007.06.004 · Cited by 995
(2004) · Medical Entomology and Zoology · Cited by 978
Neuronal oscillations and multisensory interaction in primary auditory cortex
Lakatos, P., Chen, C.-M., O'Connell, M. N., Mills, A., & Schroeder, C. E. (2007) · Neuron · doi:10.1016/j.neuron.2006.12.011 · Cited by 961
Analysis of Dynamic Brain Imaging Data
Mitra, P. P., & Pesaran, B. (1999) · Biophysical Journal · doi:10.1016/s0006-3495(99)77236-x · Cited by 955
Analysis of Dynamic Brain Imaging Data
Mitra, P. P., & Pesaran, B. (1998) · Cold Spring Harbor Laboratory Institutional Repository (Cold Spring Harbor Laboratory) · Cited by 898
Chimaeric sounds reveal dichotomies in auditory perception
Smith, Z. M., Delgutte, B., & Oxenham, A. J. (2002) · Nature · doi:10.1038/416087a · Cited by 873
Visual speech speeds up the neural processing of auditory speech
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0408949102 · Cited by 854
Perceived order in different sense modalities.
Hirsh, I. J., & Sherrick, C. E. (1961) · Journal of Experimental Psychology · doi:10.1037/h0045283 · Cited by 697
A Hierarchy of Time-Scales and the Brain
Kiebel, S. J., Daunizeau, J., & Friston, K. J. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000209 · Cited by 694 · cited 1×
“Finally, sensory and motor systems will be coupled so that stable features of the brain, body, and/or environment can be exploited to simplify perception and action [2,4,5].”
BackgroundTemporal window of integration in auditory-visual speech perception
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.01.001 · Cited by 621
The Development of Embodied Cognition: Six Lessons from Babies
Smith, L., & Gasser, M. (2005) · Artificial Life · doi:10.1162/1064546053278973 · Cited by 597 · cited 4×
“Second, multiple overlapping and time-locked sensory systems enable the agent to act in the world [5].”
BackgroundInfluentialEffect of reducing slow temporal modulations on speech reception
Drullman, R., Festen, J. M., & Plomp, R. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.409836 · Cited by 589
Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex
Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., & Logothetis, N. K. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.0799-05.2005 · Cited by 544
Self-citationSpeech comprehension is correlated with temporal response patterns recorded from auditory cortex
Ahissar, E., Nagarajan, S., Ahissar, M., Protopapas, A., Mahncke, H., & Merzenich, M. M. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.201400998 · Cited by 538
The use of visible speech cues for improving auditory detection of spoken sentences
Grant, K. W., & Seitz, P.-F. (2000) · The Journal of the Acoustical Society of America · doi:10.1121/1.1288668 · Cited by 521
Modulation spectra of natural sounds and ethological theories of auditory processing
Singh, N. C., & Theunissen, F. E. (2003) · The Journal of the Acoustical Society of America · doi:10.1121/1.1624067 · Cited by 521
Visual Modulation of Neurons in Auditory Cortex
Kayser, C., Petkov, C. I., & Logothetis, N. K. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhm187 · Cited by 509
Neuronal oscillations and visual amplification of speech
Schroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S., & Puce, A. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.01.002 · Cited by 498
Visual Prosody and Speech Intelligibility: Head Movement Improves Auditory Speech Perception
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502010.x · Cited by 482
Endogenous Cortical Rhythms Determine Cerebral Specialization for Speech Perception and Production
Giraud, A.-L., Kleinschmidt, A., Poeppel, D., Lund, T. E., Frackowiak, R. S. J., & Laufs, H. (2007) · Neuron · doi:10.1016/j.neuron.2007.09.038 · Cited by 443
Some preliminaries to a comprehensive account of audio-visual speech perception.
Summerfield, A. Q. (1987) · Cited by 421
Quantitative association of vocal-tract and facial behavior
Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00048-x · Cited by 420
Sampling Properties of the Spectrum and Coherency of Sequences of Action Potentials
Jarvis, M. R., & Mitra, P. P. (2001) · Neural Computation · doi:10.1162/089976601300014312 · Cited by 396
The Detection of Auditory Visual Desynchrony
Dixon, N. F., & Spitz, L. (1980) · Perception · doi:10.1068/p090719 · Cited by 390
Speech Physiology, Speech Perception, and Acoustic Phonetics
Lieberman, P., & Blumstein, S. E. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165952 · Cited by 376
Seeing speech: visual information from lip movements modifies activity in the human auditory cortex
Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S.-T., & Simola, J. (1991) · Neuroscience Letters · doi:10.1016/0304-3940(91)90914-f · Cited by 367
Temporal constraints on the McGurk effect
Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996) · Perception & Psychophysics · doi:10.3758/bf03206811 · Cited by 358
Lipreading and audio-visual speech perception
Summerfield, Q. (1992) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1992.0009 · Cited by 358
Temporal properties of spontaneous speech—a syllable-centric perspective
Greenberg, S., Carvey, H., Hitchcock, L., & Chang, S. (2003) · Journal of Phonetics · doi:10.1016/j.wocn.2003.09.005 · Cited by 337
(2000) · Kybernetes · doi:10.1108/k.2000.29.9_10.1333.4 · Cited by 318
Implicit multisensory associations influence voice recognition
von Kriegstein, K., & Giraud, A.-L. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040326 · Cited by 309
Mapping Information Flow in Sensorimotor Networks
Lungarella, M., & Sporns, O. (2006) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0020144 · Cited by 293
The processing of audio-visual speech: empirical and neural bases
Campbell, R. (2008) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2007.2155 · Cited by 282
Linking facial animation, head motion and speech acoustics
Yehia, H. C., Kuratate, T., & Vatikiotis-Bateson, E. (2002) · Journal of Phonetics · doi:10.1006/jpho.2002.0165 · Cited by 270
Temporal envelope and fine structure cues for speech intelligibility
Drullman, R. (1995) · The Journal of the Acoustical Society of America · doi:10.1121/1.413112 · Cited by 255
Auditory Analysis and Perception of Speech
(1975) · Elsevier eBooks · doi:10.1016/b978-0-12-248550-3.x5001-6 · Cited by 236
Eye movement of perceivers during audiovisualspeech perception
Vatikiotis-Bateson, E., Eigsti, I.-M., Yano, S., & Munhall, K. G. (1998) · Perception & Psychophysics · doi:10.3758/bf03211929 · Cited by 236
Cognitive restoration of reversed speech
Saberi, K., & Perrott, D. R. (1999) · Nature · doi:10.1038/19652 · Cited by 234
Temporal envelope and fine structure cues for speech intelligibility
Drullman, R. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.408825 · Cited by 213
An audiovisual test of kinematic primitives for visual speech perception.
Rosenblum, L. D., & Saldaña, H. M. (1996) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.22.2.318 · Cited by 204
Transient Cortical Excitation at the Onset of Visual Fixation
Rajkai, C., Lakatos, P., Chen, C.-M., Pincze, Z., Karmos, G., & Schroeder, C. E. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhm046 · Cited by 202
Ghazanfar, A. A., Chandrasekaran, C., & Logothetis, N. K. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.0541-08.2008 · Cited by 200
Self-citationThe DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM
Garofolo, J. S. (2025) · TIB Data Manager · doi:10.57702/2gqdxziu · Cited by 194
Speech perception as a multimodal phenomenon
Rosenblum, L. D. (2008) · Current Directions in Psychological Science · doi:10.1111/j.1467-8721.2008.00615.x · Cited by 193 · cited 3×
“This common modulation has been previously suggested as a possible amodal metric of speech [15,53].”
BackgroundResultInfluential`Putting the Face to the Voice'
Kamachi, M., Hill, H., Lander, K., & Vatikiotis-Bateson, E. (2003) · Current Biology · doi:10.1016/j.cub.2003.09.005 · Cited by 183
Eye Position Affects Activity in Primary Auditory Cortex of Primates
Werner-Reiss, U., Kelly, K. A., Trause, A. S., Underhill, A. M., & Groh, J. M. (2003) · Current Biology · doi:10.1016/s0960-9822(03)00168-4 · Cited by 176
Simulation of talking faces in the human brain improves auditory speech recognition
von Kriegstein, K., Dogan, Ö., Grüter, M., Giraud, A.-L., Kell, C. A., Grüter, T., Kleinschmidt, A., & Kiebel, S. J. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0710826105 · Cited by 160
Point-Light Facial Displays Enhance Comprehension of Speech in Noise
Rosenblum, L. D., Johnson, J. A., & Saldaña, H. M. (1996) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3906.1159 · Cited by 156
Lansing, C. R., & McConkie, G. W. (2003) · Perception & Psychophysics · doi:10.3758/bf03194581 · Cited by 133
Primacy of Multimodal Speech Perception
Rosenblum, L. D. (2008) · Blackwell Publishing Ltd eBooks · doi:10.1002/9780470757024.ch3 · Cited by 118
Integration of Bimodal Looming Signals through Neuronal Coherence in the Temporal Lobe
Maier, J. X., Chandrasekaran, C., & Ghazanfar, A. A. (2008) · Current Biology · doi:10.1016/j.cub.2008.05.043 · Cited by 117
Self-citationTiming and Laminar Profile of Eye-Position Effects on Auditory Responses in Primate Auditory Cortex
Fu, K.-M. G., Shah, A. S., O’Connell, M. N., McGinnis, T., Eckholdt, H., Lakatos, P., Smiley, J. F., & Schroeder, C. E. (2004) · Journal of Neurophysiology · doi:10.1152/jn.01228.2003 · Cited by 102
Looming Biases in Monkey Auditory Cortex
Maier, J. X., & Ghazanfar, A. A. (2007) · Journal of Neuroscience · doi:10.1523/jneurosci.0330-07.2007 · Cited by 102
Self-citationChandrasekaran, C., & Ghazanfar, A. A. (2008) · Journal of Neurophysiology · doi:10.1152/jn.90843.2008 · Cited by 95
Self-citationThe moving face during speech communication
de Gelder, B., Jean, V., & Bachoud‐Lévi, A. (2013) · Routledge eBooks · doi:10.4324/9780203098752-14 · Cited by 92
Spatial and Temporal Constraints on Audiovisual Speech Perception
Munhall, K. G., & Vatikiotis‐Bateson, E. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/3422.003.0015 · Cited by 72
Investigating the audio–visual speech detection advantage
Kim, J., & Davis, C. (2004) · Speech Communication · doi:10.1016/j.specom.2004.09.008 · Cited by 58
Eye movements of monkey observers viewing vocalizing conspecifics
Ghazanfar, A. A., Nielsen, K. J., & Logothetis, N. K. (2006) · Cognition · doi:10.1016/j.cognition.2005.12.007 · Cited by 57
Self-citationX-ray microbeam speech production database
Westbury, J. R., Milenkovic, P., Weismer, G. G., & Kent, R. D. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.2029064 · Cited by 52
Sodoyer, D., Schwartz, J., Laurent Girin, B., Klinkisch, J., & Jutten, C. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702207015 · Cited by 46
A study of lip movements during spontaneous dialog and its application to voice activity detection
Sodoyer, D., Rivet, B., Laurent Girin, B., Savariaux, C., Schwartz, J., & Jutten, C. (2009) · The Journal of the Acoustical Society of America · doi:10.1121/1.3050257 · Cited by 38
Analysis of temporal patterns of communication signals
Pollack, G. S. (2001) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(01)00277-x · Cited by 32
Effect of Video Frame Rate on Subjects’ Ability to Shadow One of Two Competing Verbal Passages
Vitkovitch, M., & BARBER, P. J. (1994) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3705.1204 · Cited by 30
The multisensory roles for auditory cortex in primate vocal communication
Ghazanfar, A. A. (2009) · Hearing Research · doi:10.1016/j.heares.2009.04.003 · Cited by 28
Self-citationBrain-wave representation of words by superposition of a few sine waves
Suppes, P., & Han, B. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.140228397 · Cited by 26
Paving the Way Forward: Integrating the Senses through Phase-Resetting of Cortical Oscillations
Ghazanfar, A. A., & Chandrasekaran, C. (2007) · Neuron · doi:10.1016/j.neuron.2007.01.003 · Cited by 23
Self-citation
Cited by
An integrated theory of language production and comprehension
Pickering, M. J., & Garrod, S. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12001495 · Cited by 1,569
Brain-to-brain coupling: a mechanism for creating and sharing a social world
Hasson, U., Ghazanfar, A. A., Galantucci, B., Garrod, S., & Keysers, C. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.12.007 · Cited by 1,140
Self-citationA temporal sampling framework for developmental dyslexia
Goswami, U. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.10.001 · Cited by 912
Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”
Zion Golumbic, E. M., Ding, N., Bickel, S., Lakatos, P., Schevon, C. A., McKhann, G. M., Goodman, R. R., Emerson, R., Mehta, A. D., Simon, J. Z., et al. (2013) · Neuron · doi:10.1016/j.neuron.2012.12.037 · Cited by 909
Chronux: A platform for analyzing neural signals
Bokil, H., Andrews, P., Kulkarni, J. E., Mehta, S., & Mitra, P. P. (2010) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2010.06.020 · Cited by 865
Metadata
- Authors
- Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., & Ghazanfar, A. A.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2009-07-17
- Volume
- 5
- Issue
- 7
- Pages
- e1000436
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2099940393
- Cited by
- 630
- References
- 78
- FWCI
- 20.09
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.