Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 101–200 of 1,005
A review of speech-based bimodal recognition
Chibelushi, C. C., Deravi, F., & Mason, J. S. D. (2002) · IEEE Transactions on Multimedia · doi:10.1109/6046.985551 · Cited by 222
“Lipreading is particularly beneficial when the listener suffers from impaired hearing or when the acoustic signal is degraded [115], [117].”
BackgroundStein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., & Rowland, B. A. (2009) · Experimental Brain Research · doi:10.1007/s00221-009-1880-8 · Cited by 220
“…intensity, and/or likelihood of responding to a stimulus in the first modality, and does so in a manner proportional to its original effectiveness (e.g., Sumby and Pollack 1954; Stein et al. 1996; Lovelace et al. 2003; Gillmeister and Eimer 2007; Odgaard et al. 2003; Arieh and Marks 2008).”
BackgroundAuditory-visual fusion in speech perception in children with cochlear implants
Schorr, E. A., Fox, N. A., van Wassenhove, V., & Knudsen, E. I. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0508862102 · Cited by 220
Facial expressions linked to monkey calls
Ghazanfar, A. A., & Logothetis, N. K. (2003) · Nature · doi:10.1038/423937a · Cited by 216
Integration of acoustic and visual speech signals using neural networks
Yuhas, B. P., Goldstein, M. H., & Sejnowski, T. J. (1989) · IEEE Communications Magazine · doi:10.1109/35.41402 · Cited by 209
Factors influencing audiovisual fission and fusion illusions
Andersen, T. S., Tiippana, K., & Sams, M. (2004) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.06.004 · Cited by 212
“As an example, speech perception is facilitated by viewing a congruent articulating face [2,3,7,14].”
BackgroundThe neural basis of multisensory integration in the midbrain: Its organization and maturation
Stein, B. E., Stanford, T. R., & Rowland, B. A. (2009) · Hearing Research · doi:10.1016/j.heares.2009.03.012 · Cited by 209
“…1996; Frens and Van Opstal, 1995; Hughes et al., 1994; Stein et al., 1989; Marks, 2004; Newell, 2004; Woods and Recanzone, 2004b; Shams et al., 2004; Wallace et al., 1996; Sumby and Pollack, 1954; Massaro, 2004; Ghazanfar et al., 2005; Partan, 2004; Bernstein et al., 2004; Sugihara et al., 2006).”
Crosse, M. J., Butler, J. S., & Lalor, E. C. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1829-15.2015 · Cited by 208
“Future work examining this using a speech-in-noise paradigm should prove more informative given the well-established benefits of multisensory speech in adverse hearing conditions (Sumby and Pollack, 1954; Ross et al., 2007a).”
BackgroundThe role of speech-related arm/hand gestures in word retrieval
Krauss, R. M., & Hadar, U. (1999) · Gesture, Speech, and Sign · doi:10.1093/acprof:oso/9780198524519.003.0006 · Cited by 206
“Rogers also did not adequately control for the 'speech-reading effect'—the well established finding that seeing a speaker's lips contributes to the intelligibility of speech (Sumby and Pollack 1954).”
BackgroundAn audiovisual test of kinematic primitives for visual speech perception.
Rosenblum, L. D., & Saldaña, H. M. (1996) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.22.2.318 · Cited by 204
Aero-tactile integration in speech perception
Gick, B., & Derrick, D. (2009) · Nature · doi:10.1038/nature08572 · Cited by 196
“023) (that is, perceivers identified unaspirated stops slightly more readily across all experiments) and place (F(1,63) 5 6.”
InfluentialBilingualism Modulates Infants’ Selective Attention to the Mouth of a Talking Face
Pons, F., Bosch, L., & Lewkowicz, D. J. (2015) · Psychological Science · doi:10.1177/0956797614568320 · Cited by 195
“…Davis, 2005), people who are deprived of vision during infancy because of congenital cataracts exhibit deficits in audiovisual speech integration as adults (Putzar, Goerendt, Lange, Rösler, & Röder, 2007), and adults comprehend audiovisual speech better than auditory speech (Sumby & Pollack, 1954).”
BackgroundWu, Y.-H., Stangl, E., Chipara, O., Hasan, S. S., Welhaven, A., & Oleson, J. (2018) · Ear & Hearing · doi:10.1097/aud.0000000000000486 · Cited by 193
Measures of auditory–visual integration in nonsense syllables and sentences
Grant, K. W., & Seitz, P. F. (1998) · The Journal of the Acoustical Society of America · doi:10.1121/1.423751 · Cited by 193
Rapid Recalibration to Audiovisual Asynchrony
Van der Burg, E., Alais, D., & Cass, J. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.1182-13.2013 · Cited by 191
“A highly relevant example would be the optimization of speech comprehension (Sumby and Pollack, 1954), which is optimal when visual and auditory signals are perceived as simultaneous.”
BackgroundAudiovisual prosody and feeling of knowing
Swerts, M., & Krahmer, E. (2005) · Journal of Memory and Language · doi:10.1016/j.jml.2005.02.003 · Cited by 190
“A number of studies have shown that visual cues are beneWcial for the understandability of speech (e.g., Jordan & Sergeant, 2000; Sumby & Pollack, 1954).”
BackgroundNeural processes underlying perceptual enhancement by visual speech gestures
Callan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2003) · NeuroReport · doi:10.1097/00001756-200312020-00016 · Cited by 190
“In contrast, concordant audio and visual speech information improves intelligibility especially under degraded auditory conditions [2].”
BackgroundMa, W. J., Zhou, X., Ross, L. A., Foxe, J. J., & Parra, L. C. (2009) · PLoS ONE · doi:10.1371/journal.pone.0004638 · Cited by 189
“Therefore, with the vocabulary sizes used in earlier studies (e.g. at most 256 words in [8]), it is not surprising that inverse effectiveness was observed.”
MethodResultSomatosensory function in speech perception
Ito, T., Tiede, M., & Ostry, D. J. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0810063106 · Cited by 187
“The contribution of visual inputs in particular is well documented (6, 7).”
BackgroundRoss, L. A., Saint-Amour, D., Leavitt, V. M., Molholm, S., Javitt, D. C., & Foxe, J. J. (2007) · Schizophrenia Research · doi:10.1016/j.schres.2007.08.008 · Cited by 186
“…is essential for everyday communication, as seeing a speaker's face substantially facilitates the recognition of spoken words, especially under noisy listening conditions (e.g. Erber, 1969; Grant and Seitz, 2000; Munhall et al., 2004a; O'Neill, 1954; Ross et al., 2007; Sumby and Pollack, 1954).”
BackgroundHow does gaze to faces support face-to-face interaction? A review and perspective
Hessels, R. S. (2020) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01715-w · Cited by 184
Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration
Navarra, J., Vatakis, A., Zampini, M., Soto-Faraco, S., Humphreys, W., & Spence, C. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.07.009 · Cited by 185
“Note, however, that the speech signal, presented at a speech-tonoise ratio of +3 dB, was intelligible (see [37]).”
BackgroundAuditory speech detection in noise enhanced by lipreading
Bernstein, L. E., Auer, E. T., & Takayanagi, S. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.011 · Cited by 185
“Under noisy conditions as well, intelligibility is enhanced when the talker can be seen: Seeing the talker can be functionally equivalent to an increase in the acoustic signal-to-noise ratio (Sumby and Pollack, 1954).”
BackgroundAudio-visual deep learning for noise robust speech recognition
Huang, J., & Kingsbury, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639140 · Cited by 179
“Motivated by the bimodality (auditory and visual) of human speech perception [1] and by the need for robust automatic speech recognition (ASR) in noisy environments, significant research effort has been directed into the study of audio-visual speech recognition (AVSR).”
BackgroundHCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
Kortum, P. T. (2008) · Cited by 242
Blank, H., & Davis, M. H. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002577 · Cited by 181
Infants' Use of Synchronized Visual Information to Separate Streams of Speech
Hollich, G., Newman, R. S., & Jusczyk, P. W. (2005) · Child Development · doi:10.1111/j.1467-8624.2005.00866.x · Cited by 180
“For adults, visual information can provide an additional means of directing attention to a speech stream in the presence of white noise (Sumby & Pollack, 1954).”
BackgroundListening with eye and hand: Cross-modal contributions to speech perception.
Fowler, C. A., & Dekle, D. J. (1991) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.17.3.816 · Cited by 183
Audiovisual Integration and Lipreading Abilities of Older Adults with Normal and Impaired Hearing
Tye-Murray, N., Sommers, M. S., & Spehar, B. (2007) · Ear & Hearing · doi:10.1097/aud.0b013e31812f7185 · Cited by 177
Impacts of face coverings on communication: an indirect impact of COVID-19
Saunders, G. H., Jackson, I. R., & Visram, A. S. (2021) · International Journal of Audiology · doi:10.1080/14992027.2020.1851401 · Cited by 178
“Such cues are used by everyone to supplement incoming speech information (Grant and Seitz 2000; Sumby and Pollack 1954), although the benefit gained is highly variable across individuals (MacLeod and Summerfield 1990).”
BackgroundEffects of Talker Variability on Perceptual Learning of Dialects
Clopper, C. G., & Pisoni, D. B. (2004) · Language and Speech · doi:10.1177/00238309040470030101 · Cited by 175
“In addition, the effects of set size on speech perception and speech intelligibility grow vanishingly small as the size of the sets increases (Sumby & Pollack, 1954).”
BackgroundKlucharev, V., Möttönen, R., & Sams, M. (2003) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2003.09.004 · Cited by 174
“Seeing speech improves speech recognition especially in a noisy environment [39].”
BackgroundNath, A. R., & Beauchamp, M. S. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4853-10.2011 · Cited by 171
“A number of behavioral studies show that the presence of information from both the auditory and visual modalities aids the perception of noisy speech (Sumby and Pollack, 1954; Hauser, 1996; Kryter, 1996; Grant and Seitz, 2000; MacDonald et al., 2000; Shahin and Miller, 2009).”
BackgroundCrossmodal Integration in the Identification of Consonant Segments
Braida, L. D. (1991) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640749108400991 · Cited by 171
Audiovisual processing in children with and without autism spectrum disorders
Mongillo, E. A., Irwin, J. R., Whalen, D. H., Klaiman, C., Carter, A. S., & Schultz, R. T. (2008) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-007-0521-y · Cited by 170
“Typically developing individuals seem to be influenced perceptually by both the auditory and visual components of speech (Massaro 1984; Sumby and Pollack 1954 ).”
BackgroundBinding of sights and sounds: Age-related changes in multisensory temporal processing
Hillock, A. R., Powers, A. R., & Wallace, M. T. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2010.11.041 · Cited by 170
“…studies, and include enhancements in signal detection, speeded motor responses, and improved speech in noise performance (Frens, Van Opstal, & Van der Willigen, 1995; Grant & Seitz, 2000; Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994; Lovelace, Stein, & Wallace, 2003; Sumby & Pollack, 1954).”
BackgroundVisual speech recognition for multiple languages in the wild
Ma, P., Petridis, S., & Pantic, M. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00550-z · Cited by 169
“In addition, it has been shown that the addition of visual speech information to a word recognition task performed by normal hearing adults is equivalent to increasing the signal-to-noise ratio (SNR) by 15 dB compared to audio-only recognition [20].”
BackgroundInfant perception of audio-visual speech synchrony
Lewkowicz, D. J. (2010) · Developmental Psychology · doi:10.1037/a0015579 · Cited by 168
“Studies comparing comprehension of auditory-only speech versus audiovisual speech have shown that comprehension of the latter is significantly better (Sumby & Pollack, 1954; Sum- merfield, 1979).”
BackgroundChildren’s perception of speech in multitalker babble
Fallon, M., Trehub, S. E., & Schneider, B. A. (2000) · The Journal of the Acoustical Society of America · doi:10.1121/1.1323233 · Cited by 168
Sekiyama, K. (1997) · Perception & Psychophysics · doi:10.3758/bf03206849 · Cited by 167
Zion Golumbic, E. M., Poeppel, D., & Schroeder, C. E. (2012) · Brain and Language · doi:10.1016/j.bandl.2011.12.010 · Cited by 167
“It is known that congruent visual input substantially improves speech comprehension, particularly under difficult listening conditions (Campanella & Belin, 2007; Campbell, 2008; Sumby & Pollack, 1954; Summerfield, 1992).”
Direct Structural Connections between Voice- and Face-Recognition Areas
Blank, H., Anwander, A., & von Kriegstein, K. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.2091-11.2011 · Cited by 165
“For example, speech recognition may benefit from the integration of fast-varying dynamic visual and auditory information (Sumby and Pollack, 1954).”
BackgroundAlbright, T. D. (2012) · Neuron · doi:10.1016/j.neuron.2012.04.001 · Cited by 162
“Along the same lines, Sumby and Pollack (1954) showed that visibility of a speaker’s lips improves auditory word recognition, particularly when spokenwords are embedded in auditory noise.”
Hazan, V., Sennema, A., Iba, M., & Faulkner, A. (2005) · Speech Communication · doi:10.1016/j.specom.2005.04.007 · Cited by 161
“Visual cues add to the multiplicity of cues that is so crucial for making speech robust to environmental degradation, and are indeed particularly useful for native listeners in difficult listening conditions (e.g., Sumby and Pollack, 1954).”
BackgroundSimulation of talking faces in the human brain improves auditory speech recognition
von Kriegstein, K., Dogan, Ö., Grüter, M., Giraud, A.-L., Kell, C. A., Grüter, T., Kleinschmidt, A., & Kiebel, S. J. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0710826105 · Cited by 160
“053]; a type of learning and group interaction [F(1,32) 4.”
BackgroundMethodInfluentialCrosse, M. J., Di Liberto, G. M., & Lalor, E. C. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.1396-16.2016 · Cited by 161
“It has long been established that the behavioral benefits of audiovisual (AV) speech are more apparent in acoustic conditions in which intelligibility is reduced (Sumby and Pollack, 1954; Erber, 1975; Grant and Seitz, 2000; Bernstein et al., 2004; Ross et al., 2007).”
BackgroundThe cascading influence of multisensory processing on speech perception in autism
Stevenson, R. A., Segers, M., Ncube, B. L., Black, K. R., Bebko, J. M., Ferber, S., & Barense, M. D. (2018) · Autism · doi:10.1177/1362361317704413 · Cited by 161
“Integrating sensory inputs across modalities subsequently provides behavioral benefits, most notably improvements in speech perception, particularly in noisy environments (Sumby and Pollack, 1954).”
BackgroundAtilgan, H., Town, S. M., Wood, K. C., Jones, G. P., Maddox, R. K., Lee, A. K. C., & Bizley, J. K. (2018) · Neuron · doi:10.1016/j.neuron.2017.12.034 · Cited by 160
“Being able to see a talker’s mouth provides listeners with information about the rhythm and amplitude of the speech waveform that may help listeners by cueing them to pay attention to the auditory envelope (Peelle and Sommers, 2015) as well as by providing cues to the place of articulation that can disambiguate differ…”
BackgroundProcessing of audiovisual speech in Broca's area
Ojanen, V., Möttönen, R., Pekkola, J., Jääskeläinen, I. P., Joensuu, R., Autti, T. H., & Sams, M. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.001 · Cited by 178
“Seeing speaker’s articulatory gestures improves identification of acoustic speech stimuli, especially in noisy conditions (Sumby and Pollack, 1954).”
BackgroundSpatial statistics of gaze fixations during dynamic face processing
Buchan, J. N., Paré, M., & Munhall, K. G. (2007) · Social Neuroscience · doi:10.1080/17470910601043644 · Cited by 159
“It has long been known that seeing a talker’s face enhances speech intelligibility in noisy acoustic environments (Sumby & Pollack, 1954).”
BackgroundPoint-Light Facial Displays Enhance Comprehension of Speech in Noise
Rosenblum, L. D., Johnson, J. A., & Saldaña, H. M. (1996) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jshr.3906.1159 · Cited by 156
“, Erber, 1969; MacLeod & Summerfield, 1987; 1990; Middleweerd & Plomp, 1987; Sumby and Pollack, 1954). This research has revealed that visual speech information can enhance speech in noise so that it functionally improves signal to noise ratios (SNRs) as much as 22 dB (e.g., Sumby & Pollack, 1954). These studies have…”
BackgroundBrain activity during audiovisual speech perception: An fMRI study of the McGurk effect
Jones, J. A., & Callan, D. E. (2003) · Neuroreport · doi:10.1097/00001756-200306110-00006 · Cited by 160
Holle, H., Obleser, J., Rueschemeyer, S.-A., & Gunter, T. C. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.08.058 · Cited by 153
“Notably, Willems et al. (2009) insights into differential roles of left IFG and pSTS/MTG were largely based on the use of semantically incongruent combinations of speech and hand movements.”
Effects of Phonetic Context on Audio-Visual Intelligibility of French
Benoı̂t, C., Mohamadi, T., & Kandel, S. (1994) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3705.1195 · Cited by 163
Development of Audiovisual Comprehension Skills in Prelingually Deaf Children With Cochlear Implants
Bergeson, T. R., Pisoni, D. B., & Davis, R. A. O. (2005) · Ear and Hearing · doi:10.1097/00003446-200504000-00004 · Cited by 152
“In normal-hearing adult listeners, the gain in performance from combined AV information is superadditive in nature (Sumby & Pollack, 1954).”
BackgroundInfluentialPark, H., Kayser, C., Thut, G., & Groß, J. (2016) · eLife · doi:10.7554/elife.14521 · Cited by 173
“Even for untrained observers and in the presence of auditory signals, lip movements can be beneficial for speech comprehension when the acoustic signal is degraded (Peelle and Sommers, 2015; Sumby and Pollack, 1954; van Wassenhove et al., 2005; Zion-Golumbic and Schroeder, 2012).”
BackgroundInfluentialThe role of facial movements in emotion recognition
Krumhuber, E. G., Skora, L. I., Hill, H. C. H., & Lander, K. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00172-1 · Cited by 194
“…and can even change, speech recognition with distinctions that are difficult to hear like place of articulation (McGurk & MacDonal, 1976; Sumby & Pollack, 1954; Summerfield, 1987), similar facilitative and biasing effects are found for emotion recognition (Aubergé & Cathiard, 2003; De…”
Processing of changes in visual speech in the human auditory cortex
Möttönen, R., Krause, C. M., Tiippana, K. M., & Sams, M. E. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00053-8 · Cited by 171
Evidence for causal top-down frontal contributions to predictive processes in speech perception
Cope, T. E., Sohoglu, E., Sedley, W., Patterson, K., Jones, P. S., Wiggins, J., Dawson, C., Grube, M., Carlyon, R. P., Griffiths, T. D., et al. (2017) · Nature Communications · doi:10.1038/s41467-017-01958-7 · Cited by 150
Multisensory Integration as a Window into Orderly and Disrupted Cognition and Communication
Wallace, M. T., Woynaroski, T. G., & Stevenson, R. A. (2020) · Annual Review of Psychology · doi:10.1146/annurev-psych-010419-051112 · Cited by 149
“The gains in comprehension typically seen for combined audiovisual speech (Sumby & Pollack 1954) have been shown to be diminished in autistic individuals.”
BackgroundSuperior temporal activation in response to dynamic audio-visual emotional cues
Robins, D. L., Hunyadi, E., & Schultz, R. T. (2009) · Brain and Cognition · doi:10.1016/j.bandc.2008.08.007 · Cited by 149
“…perception of two moving bars as colliding or passing through one another depending on the timing of an auditory burst (Bushara et al., 2003), lip reading (G. A. Calvert, Michael J. Brammer and Susan D. Iversen, 1998; Sumby & Pollack, 1955) or reading written text (Frost, Repp, & Katz, 1989).”
BackgroundMultisensory perceptual learning and sensory substitution
Proulx, M. J., Brown, D. J., Pasqualotto, A., & Meijer, P. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2012.11.017 · Cited by 148
The development of multisensory speech perception continues into the late childhood years
Ross, L. A., Molholm, S., Blanco, D., Gomez‐Ramirez, M., Saint‐Amour, D., & Foxe, J. J. (2011) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07685.x · Cited by 147
“paper by Sumby & Pollack (1954), adjusts AV gain by the room for improvement left for AV performance, which becomes drastically limited with increasing intelligibility.”
BackgroundIntersensory Perception at Birth: Newborns Match Nonhuman Primate Faces and Voices
Lewkowicz, D. J., Leo, I., & Simion, F. (2010) · Infancy · doi:10.1111/j.1532-7078.2009.00005.x · Cited by 160
“2004; Partan & Marler, 1999; Rowe, 1999; Stein & Meredith, 1993; Sumby & Pollack, 1954;”
Visual influences on auditory spatial learning
King, A. J. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0230 · Cited by 146
“For instance, it is frequently the case when listening to someone’s voice that we also see their lips moving, which, particularly in noisy situations, can improve speech intelligibility (Sumby & Pollack 1954).”
Moving image theory: ecological considerations
(2005) · Choice Reviews Online · doi:10.5860/choice.43-0207 · Cited by 147
McGurk effects in cochlear-implanted deaf subjects
Rouger, J., Fraysse, B., Déguine, O., & Barone, P. (2007) · Brain Research · doi:10.1016/j.brainres.2007.10.049 · Cited by 170
“…deaf subjects Julien Rougera, Bernard Fraysseb, Olivier Deguinea,b, Pascal Baronea,⁎ aCentre de Recherche Cerveau et Cognition, UMR CNRS 5549, Faculté de Médecine de Rangueil, 31062 Toulouse Cedex 9, France bService ORL, CHU Toulouse-Purpan, 1 place du Docteur Baylac, 31059 Toulouse…”
BackgroundThe associations between multisensory temporal processing and symptoms of schizophrenia
Stevenson, R. A., Park, S., Cochran, C., McIntosh, L. G., Noel, J.-P., Barense, M. D., Ferber, S., & Wallace, M. T. (2017) · Schizophrenia Research · doi:10.1016/j.schres.2016.09.035 · Cited by 144
“…speech signals with their associated visual cues, affording substantial tern Ontario, Department of behavioral and perceptual benefits, ranging from faster response times (Raab, 1962) to improved speech perception (Sumby and Pollack, 1954) in healthy participants but not asmuch in SCZpatients.”
Visual attention modulates audiovisual speech perception
Tiippana, K. M., Andersen, T., & Sams, M. E. (2004) · The European Journal of Cognitive Psychology · doi:10.1080/09541440340000268 · Cited by 154
“This occurs not only in noisy conditions (Erber, 1969; Sumby & Pollack, 1954), but also when the auditory speech signal is clear.”
BackgroundNeural pathways for visual speech perception
Bernstein, L. E., & Liebenthal, E. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00386 · Cited by 166
“The basic finding in normal-hearing adults that vision can compensate for hearing under noisy conditions was reported by mid-twentieth century (Sumby and Pollack, 1954).”
BackgroundEmbodied conversational agents in computer assisted language learning
Wik, P., & Hjalmarsson, A. (2009) · Speech Communication · doi:10.1016/j.specom.2009.05.006 · Cited by 168
“It is well established that visual information supports speech perception (Sumby & Pollack, 1954).”
BackgroundLachs, L., Pisoni, D. B., & Kirk, K. I. (2001) · Ear and Hearing · doi:10.1097/00003446-200106000-00007 · Cited by 160
Activity in perceptual classification networks as a basis for human subjective time perception
Roseboom, W., Fountas, Z., Nikiforou, K., Bhowmik, D., Shanahan, M., & Seth, A. K. (2019) · Nature Communications · doi:10.1038/s41467-018-08194-7 · Cited by 163
“This additional level of redundancy in estimation would likely improve performance for scenarios that include both visual and auditory information, as has been shown in other cases where redundant cues from different modalities are combined 23,52 – 54 .”
BackgroundDevelopmental changes in visual and auditory contributions to speech perception
Massaro, D. W., Thompson, L. A., Barron, B., & Laren, E. (1986) · Journal of Experimental Child Psychology · doi:10.1016/0022-0965(86)90053-6 · Cited by 144
Impact of language on development of auditory‐visual speech perception
Sekiyama, K., & Burnham, D. K. (2008) · Developmental Science · doi:10.1111/j.1467-7687.2008.00677.x · Cited by 159
“Participants were asked to identify syllables in audiovisual (with congruent or discrepant auditory and visual components), audio-only, and video-only presentations at various signal-to-noise levels.”
BackgroundChildren's Perception of Visual and Auditory Speech
Massaro, D. W. (1984) · Child Development · doi:10.1111/j.1467-8624.1984.tb00420.x · Cited by 143
Early experience and multisensory perceptual narrowing
Lewkowicz, D. J. (2014) · Developmental Psychobiology · doi:10.1002/dev.21197 · Cited by 138
“Seeing as well as hearing speech is known to increase its salience and comprehension (Rosenblum, Johnson, & Saldana, 1996; Sumby & Pollack, 1954; Summerfield, 1979).”
BackgroundAuditory and Auditory-Visual Perception of Clear and Conversational Speech
Helfer, K. S. (1997) · Journal of Speech Language and Hearing Research · doi:10.1044/jslhr.4002.432 · Cited by 141
Saint‐Amour, D., Sanctis, P. D., Molholm, S., Ritter, W., & Foxe, J. J. (2006) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.03.036 · Cited by 160
“For example, it is remarkable just how much speech recognition can be accentuated by the visual cues provided by mouth articulation under noisy environmental conditions (Ross, Saint-Amour, Leavitt, Javitt, & Foxe, in press; see also Sumby & Pollack, 1954).”
BackgroundDetection and Resolution of Audio-Visual Incompatibility in the Perception of Vowels
Summerfield, Q., & McGrath, M. (1984) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748408401503 · Cited by 140
Priming and sentence context support listening to noise-vocoded speech by younger and older adults.
Sheldon, S., Pichora-Fuller, M., & Schneider, B. A. (2008) · Journal of the Acoustical Society of America · doi:10.1121/1.2783762 · Cited by 142
“…situational cues, such as priming with a semantically related sentence e.g., Gagné et al., 2002 , presenting visual speech for speech reading e.g., Sumby and Pollack, 1954 , presenting written text or clear speech as feedback e.g., Davis et al., 2005 , spatially separating concurrent sounds e.g.,…”
Memorable Audiovisual Narratives Synchronize Sensory and Supramodal Neural Responses
Cohen, S. S., & Parra, L. C. (2016) · eneuro · doi:10.1523/eneuro.0203-16.2016 · Cited by 138
“Coupling of semantically linked audio and visual information has been shown to augment overall comprehension of the material (Sumby and Pollack, 1954).”
BackgroundAuditory-Visual Speech Perception and Aging
Cienkowski, K. M., & Carney, A. E. (2002) · Ear and Hearing · doi:10.1097/00003446-200210000-00006 · Cited by 146
“Sumby and Pollack (1954) demonstrated that the addition of visual cues could improve the perception of speech by the equivalent o fa5t o 18 dBincrease in the signal-to-noise (S/N) ratio; that is, an effective change of up to 60% in word recognition was observed depending on the materials used.”
BackgroundCalvert, G. A., Brammer, M. J., & Iversen, S. D. (1998) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(98)01189-9 · Cited by 171
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Makino, T., Liao, H., Assael, Y., Shillingford, B., Garcia, B., Braga, O., & Siohan, O. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9004036 · Cited by 129
Glick, H., & Sharma, A. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00093 · Cited by 167
Riseborough, M. G. (1981) · Journal of Nonverbal Behavior · doi:10.1007/bf00986134 · Cited by 135
Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception
Hubbard, A. L., Wilson, S. M., Callan, D. E., & Dapretto, M. (2009) · Human Brain Mapping · doi:10.1002/hbm.20565 · Cited by 133
“Behavioral effects related to visual speech are similar to those observed for gesture, as concordant visual speech can aid speech perception [Sumby and Pollack, 1954] whereas discordant visual speech can alter auditory perception [McGurk and MacDonald, 1976].”
BackgroundGuttman, O. T., Lazzara, E. H., Keebler, J. R., Webster, K. L. W., Gisick, L. M., & Baker, A. L. (2021) · Journal of Patient Safety · doi:10.1097/pts.0000000000000541 · Cited by 68
“These barriers are of particular concern in the context of significant background noise, when a spoken message can be masked by background noise.(54) The operating room environment presents a particular challenge within healthcare given the use of opaque drapes, surgical masks that cover a provider’s face below the ey…”
BackgroundDrijvers, L., & Özyürek, A. (2016) · Journal of Speech Language and Hearing Research · doi:10.1044/2016_jslhr-h-16-0101 · Cited by 165
Navarra, J., & Soto‐Faraco, S. (2005) · Psychological Research · doi:10.1007/s00426-005-0031-5 · Cited by 173
“For example, previous research has established that the presence of correlated visual articulatory information can greatly enhance the comprehension of spoken messages when the auditory signal is acoustically degraded ( Sumby, & Pollack, 1954 ).”
BackgroundContext effects in word identification
Kintsch, W., & Mross, E. F. (1985) · Journal of Memory and Language · doi:10.1016/0749-596x(85)90032-4 · Cited by 131
Audio-visual enhancement of speech in noise
Laurent Girin, B., Schwartz, J., & Feng, G. (2001) · The Journal of the Acoustical Society of America · doi:10.1121/1.1358887 · Cited by 130
Survey on automatic lip-reading in the era of deep learning
Fernandez-Lopez, A., & Sukno, F. M. (2018) · Image and Vision Computing · doi:10.1016/j.imavis.2018.07.002 · Cited by 148
“the visual channel becomes more important in noisy environments) [5], [6], [7], [8].”
BackgroundEnhanced Visual Speech Perception in Individuals With Early-Onset Hearing Impairment
Auer, E. T., & Bernstein, L. E. (2007) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2007/080 · Cited by 150
Orienting Attention to Points in Time Improves Stimulus Processing Both within and across Modalities
Lange, K., & Röder, B. (2006) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2006.18.5.715 · Cited by 128
“For example, speech comprehension is improved when the speaker’s lip movements are visible, particularly in noisy environments ( Sumby & Pollack, 1954 ).”
BackgroundAudio–visual speech perception is special
Tuomainen, J., Andersen, T., Tiippana, K. M., & Sams, M. E. (2004) · Cognition · doi:10.1016/j.cognition.2004.10.004 · Cited by 146
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12552 · Cited by 126
“On the other hand, it has been shown that viewers are very sensi-c tive to discrepancies between the auditory signal and the visual appearance of the face and lips during speech [SP54].”
BackgroundInfluentialDigital noise reduction: Outcomes from laboratory and field studies
Bentler, R. A., Wu, Y.-H., Kettel, J., & Hurtig, R. R. (2008) · International Journal of Audiology · doi:10.1080/14992020802033091 · Cited by 152
The role of visual speech cues in reducing energetic and informational masking
Helfer, K. S., & Freyman, R. L. (2005) · The Journal of the Acoustical Society of America · doi:10.1121/1.1836832 · Cited by 126
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.