Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 601–700 of 1,005
EFFECTS OF DIFFERENT TYPES OF FACE COVERINGS ON SPEECH ACOUSTICS AND INTELLIGIBILITY
Llamas, C. C., Harrison, P., Donnelly, D., & Watt, D. J. L. (2014) · Cited by 41
O’Sullivan, A. E., Lim, C. Y., & Lalor, E. C. (2019) · European Journal of Neuroscience · doi:10.1111/ejn.14425 · Cited by 42
“…aid speech comprehension (Crosse, Butler, & Lalor, 2015; Reisberg, Mclean, & Goldfield, 1987), a benefit which is even more evident in suboptimal listening conditions (Crosse, Di Liberto, & Lalor, 2016; Grant & Seitz, 2000; Ross, Saint‐Amour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954).”
BackgroundFernandez-Lopez, A., Martínez, O., & Sukno, F. M. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.34 · Cited by 51
Zerr, M., Freihorst, C., Schütz, H., Sinke, C., Müller, A., Bleich, S., Münte, T. F., & Szycik, G. R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02489 · Cited by 43
Alaerts, K., Swinnen, S. P., & Wenderoth, N. (2009) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2009.05.006 · Cited by 50
Effects of Amplification and Speechreading on Consonant Recognition by Persons with Impaired Hearing
Walden, B. E., Grant, K. W., & Cord, M. T. (2001) · Ear and Hearing · doi:10.1097/00003446-200108000-00007 · Cited by 41
“…could be limited by ceiling effects for some consonants, these data are expressed as the amount of improvement resulting from each source of information relative to the improvement possible from the baseline condition (i.e., 100 2 baseline score) for each consonant (Sumby & Pollack, 1954).”
BackgroundTop-down and bottom-up modulation of audiovisual integration in speech
Colin, C., Radeau, M., & Deltenre, P. (2005) · The European Journal of Cognitive Psychology · doi:10.1080/09541440440000168 · Cited by 41
Cross-Modal Source Information and Spoken Word Recognition.
Lachs, L., & Pisoni, D. B. (2004) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.30.2.378 · Cited by 53
“In combination, auditory and visual information about speech can produce improved speech recognition performance (Sumby & Pollack, 1954).”
BackgroundCascading and Multisensory Influences on Speech Perception Development
Choi, D., Black, A. K., & Werker, J. F. (2018) · Mind Brain and Education · doi:10.1111/mbe.12162 · Cited by 43
Baart, M. (2016) · Psychophysiology · doi:10.1111/psyp.12683 · Cited by 41
Stevenson, R. A., Sun, S. Z., Hazlett, N., Cant, J. S., Barense, M. D., & Ferber, S. (2016) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-016-2711-y · Cited by 53
“Besides being able to determine who an individual is at the sight of their face, typical face perception includes processing of emotional expressions (Ekman 1972), contributions to speech perception through the facial articulations of language (Sumby and Pollack 1954), and directed attention via the perception of othe…”
BackgroundCecere, R., Groß, J., Willis, A., & Thut, G. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.2926-16.2017 · Cited by 41
“In visual-to-auditory (VA) interactions such as visual enhancement of auditory speech comprehension (Sumby and Pollack, 1954; Munhall et al., 2004; van Wassenhove et al., 2005), anticipatory visual information can facilitate the tracking of the dynamic auditory speech signal by providing useful predictive cues about t…”
BackgroundHolmes, E., Folkeard, P., Johnsrude, I. S., & Scollie, S. D. (2018) · International Journal of Audiology · doi:10.1080/14992027.2018.1432901 · Cited by 39
“Listeners with normal hearing can use non-acoustic cues—such as congruent visual lip movement information (e.g. Sumby and Pollack 1954), prior knowledge of talker characteristics (e.g. Johnsrude et al. 2013), and contextual information (e.g. Davis et al. 2011; Dubno, Ahlstrom, and Horwitz 2000)—to…”
Eyes Wide Open: Pupillary Response to a Foreign Accent Varying in Intelligibility
Porretta, V., & Tucker, B. V. (2019) · Frontiers in Communication · doi:10.3389/fcomm.2019.00008 · Cited by 30
“…the presence of various types of noise (both energetic, in which the mask causes the nature of the target signal to become imperceptible, and informational, in which the mask competes with the target signal) generally leads to diminished intelligibility (see, for example, Sumby and Pollack, 1954).”
BackgroundMagnotti, J. F., Mallick, D. B., Feng, G. Y., Zhou, B., Zhou, W., & Beauchamp, M. S. (2015) · Experimental Brain Research · doi:10.1007/s00221-015-4324-7 · Cited by 44
“2005) and accuracy (Sumby and Pollack 1954) of speech perception.”
RetractedRamirez, J., & Mann, V. A. (2005) · The Journal of the Acoustical Society of America · doi:10.1121/1.1940509 · Cited by 40
Ten Years at the Multisensory Forum: Musings on the Evolution of a Field
Foxe, J. J., & Molholm, S. (2009) · Brain Topography · doi:10.1007/s10548-009-0102-9 · Cited by 50
“From this, we surmise that during ‘‘normal’’ speech perception, where we consider our perceptions to be veridical, visual inputs meaningfully impact the speech that we hear, and this is very clearly observed when humans try to recognize speech under noisy environmental conditions (Sumby and Pollack 1954; Ross et al. 2…”
BackgroundCecere, R., Groß, J., & Thut, G. (2016) · European Journal of Neuroscience · doi:10.1111/ejn.13242 · Cited by 43
“Several studies have investigated the time window over which audiovisual events are integrated, also called the temporal binding window, and revealed asymmetries depending on the order of audiovisual input (i.e. the leading sense).”
BackgroundJoint audio-visual speech processing for recognition and enhancement.
Potamianos, G., Neti, C., & Deligne, S. (2003) · AVSP · Cited by 44
Large-vocabulary audio-visual speech recognition by machines and humans
Potamianos, G., Neti, C., Iyengar, G., & Helmuth, E. (2001) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2001-294 · Cited by 44
“To our knowledge, human audio-visual LVCSR has also been studied in [15] at a single SNR, whereas effective SNR gains have been reported to be 16 dB for word recognition, in [8], and 11 dB forwhitenoise and sentences of limited syntactical variety, in [16].”
BackgroundA text-to-audiovisual-speech synthesizer for French
Goff, B. L., & Benoı̂t, C. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607232 · Cited by 40
“Visible speech is particularly effective when the auditory speech is degraded, because of noise, bandwidth filtering, or hearingimpairment, as shown for long in English (Sumby & Pollack, 1954[18]; Summerfield et al.”
BackgroundKauramäki, J., Jääskeläinen, I. P., Hari, R., Möttönen, R., Rauschecker, J. P., & Sams, M. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.1950-09.2010 · Cited by 47
“For example, watching the lips of a speaker enhances speech perception in noisy conditions (Sumby and Pollack, 1954).”
BackgroundBernstein, L. E., Jordan, N., Auer, E. T., & Eberhardt, S. P. (2022) · American Journal of Audiology · doi:10.1044/2021_aja-21-00112 · Cited by 53
“The intelligibility of noisy and/or degraded speech has long been known to increase when the talker can be seen as well as heard (i.e., Erber, 1969; Middelweerd & Plomp, 1987; Sumby & Pollack, 1954).”
BackgroundAudiovisual benefit for recognition of speech presented with single-talker noise in older listeners
Jesse, A., & Janse, E. (2012) · Language and Cognitive Processes · doi:10.1080/01690965.2011.620335 · Cited by 47
“Listeners commonly recognise speech better when they can see and hear a speaker talk compared to when they only hear the speaker talk (e.g., Arnold & Hill, 2001; MacLeod & Summerfield, 1987; Reisberg, McLean, & Goldfield, 1987; Sumby & Pollack, 1954).”
BackgroundThe behavioral relevance of multisensory neural response interactions
Sperdin, H. F., Cappe, C., & Murray, M. M. (2010) · Frontiers in Neuroscience · doi:10.3389/neuro.01.009.2010 · Cited by 48
“Frontiers in Neuroscience May 2010 | Volume 4 | Issue 1 | 9”
BackgroundAudio-Visual Detection Benefits in the Rat
Kayser, S. J., & Kayser, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0045677 · Cited by 46
“These include faster reactions and better detection rates in multisensory compared to unisensory conditions [1,2], the more rapid accumulation of information in time [3], or the facilitation of higher-level object processing [4].”
BackgroundDrijvers, L., & Özyürek, A. (2019) · Language and Speech · doi:10.1177/0023830919831311 · Cited by 49
Zhang, S. S., Lei, M., Ma, B., & Xie, L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682566 · Cited by 38
“Back to 1954, the visual modality benefits to speech intelligibility in noise has been quantified by Sumby and Pollack in [6].”
BackgroundMultisensory Interactions in the Real World
Soto‐Faraco, S., Kvasova, D., Biau, E., Ikumi, N., Ruzzoli, M., Fernández, L. M., & Cuello, M. T. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108578738 · Cited by 49
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis
Yang, K., Markovic, D., Krenn, S., Agrawal, V., & Richard, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00805 · Cited by 60
“The ability of humans to isolate a target speaker from a noisy environment [41,57] has inspired extensive study into computational approaches for speech separation and enhancement.”
BackgroundInter-signal interaction and uncertain information in anuran multimodal signals
Taylor, R. C., Klein, B. A., & Ryan, M. J. (2011) · Current Zoology · doi:10.1093/czoolo/57.2.153 · Cited by 39
“For example, visual cues can increase comprehension and alter perception of human speech (Sumby and Pollack, 1954; McGurk and MacDonald, 1976).”
BackgroundTest–retest reliability in fMRI of language: Group and task effects
CHEN, E., & Small, S. L. (2006) · Brain and Language · doi:10.1016/j.bandl.2006.04.015 · Cited by 46
“…and hands, should lead participants to better performance overall, including processes for attention, working memory, and comprehension (Goldin-Meadow, Kim, & Singer, 1999a; Goldin-Meadow & Sandhofer, 1999b; Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001; Sumby & Pollack, 1954; Summerfield, 1992).”
The lateralization of lip-read sounds: A first look
Campbell, R. (1986) · Brain and Cognition · doi:10.1016/0278-2626(86)90059-x · Cited by 39
Kinematics-based synthesis of realistic talking faces
Kuratate, T., Yehia, H. C., & Vatikiotis‐Bateson, E. (1998) · Cited by 39
Zhu, Q., Zhou, L., Zhang, Z., Liu, S., Jiao, B., Zhang, J., Dai, L.-R., Jiang, D., Li, J., & Wei, F. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3275873 · Cited by 68
“On the one hand, speech production is accompanied by lip movements, where both audio and visual information facilitate speech understanding process of human [22].”
BackgroundFan, C., Zhang, H., Ni, Q., Zhang, J., Tao, J., Zhou, J., Yi, J., Lv, Z., & Wu, X. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102946 · Cited by 56
Recognition of amodal language identity emerges in infancy
Lewkowicz, D. J., & Pons, F. (2013) · International Journal of Behavioral Development · doi:10.1177/0165025412467582 · Cited by 39
“…redundancy of everyday speech, and because of this we not only perceive our interlocutors as unified amodal entities (Fowler, 2004; Rosenblum, 2008; Sumby & Pollack, 1954) but even recognize their specific amodal identity (Kamachi, Hill, Lander, & Vatikiotis-Bateson, 2003; Lachs & Pisoni, 2004).”
BackgroundA survey on QoE-oriented wireless resources scheduling
Sousa, I., Queluz, M. P., & Rodrigues, A. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2020.102594 · Cited by 46
“For instance, the authors of [7] showed that if supplementary visual observation of the speaker’s facial and lip movements are also utilized in addition to the oral speech, humans can tolerate higher noise interference levels than if no visual factors were utilized.”
BackgroundConcurrent talking in immersive virtual reality: on the dominance of visual speech cues
González-Franco, M., Maselli, A., Florencio, D. A. F., Smolyanskiy, N., & Zhang, Z. (2017) · Scientific Reports · doi:10.1038/s41598-017-04201-x · Cited by 41
“Lip-reading is well known to enhance auditory speech perception in noisy environments 16 , but it also leads to a set of cross-modal distortions in auditory perception.”
BackgroundCortical tracking of speech in noise accounts for reading strategies in children
Destoky, F., Bertels, J., Niesen, M., Wens, V., Ghinst, M. V., Leybaert, J., Lallier, M., Ince, R. A. A., Groß, J., Tiège, X. D., et al. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000840 · Cited by 40
“Finally, to enhance SiN processing, humans also benefit from visual information of the speaker’s articulatory mouth movements [61,62].”
BackgroundGaze Patterns and Audiovisual Speech Enhancement
Yi, A., Wong, W., & Eizenman, M. (2012) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2012/10-0288 · Cited by 38
A study of lip movements during spontaneous dialog and its application to voice activity detection
Sodoyer, D., Rivet, B., Laurent Girin, B., Savariaux, C., Schwartz, J., & Jutten, C. (2009) · The Journal of the Acoustical Society of America · doi:10.1121/1.3050257 · Cited by 38
“Many studies have shown that the visual modality improves the intelligibility of speech in noise when switching from the “audio only” condition to the “audio + speaker’s face” condition (Sumby and Pollack, 1954; Erber, 1975; Benoît et al ., 1994; Robert-Ribes et al. , 1998).”
Noise alters beta-band activity in superior temporal cortex during audiovisual speech processing
Schepers, I. M., Schneider, T. R., Hipp, J. F., Engel, A. K., & Senkowski, D. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.11.066 · Cited by 42
“In line with previous studies investigating audiovisual speech and language processing (Bernstein et al., 2004; Ross et al., 2007a; Sumby and Pollack, 1954), the syllable detection rate was enhanced and the response speed was shorter when visual speech was presented in addition to auditory speech.”
BackgroundResultMotherese by Eye and Ear: Infants Perceive Visual Prosody in Point-Line Displays of Talking Heads
Kitamura, C., Guellaï, B., & Kim, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0111467 · Cited by 47
“speech in noisy [1] and quiet environments [2].”
BackgroundBrancazio, L., & Miller, J. L. (2005) · Perception & Psychophysics · doi:10.3758/bf03193531 · Cited by 38
“Furthermore, the overall incidence of the McGurk effect for a given stimulus typically varies considerably across individuals (Brancazio, Miller, & Pare, 1999; Carney, Clement, & Cienkowski, 1999; MacDonald, Andersen, & Bachmann, 2000).”
BackgroundMultisensory Facilitation of Behavior in Monkeys: Effects of Stimulus Intensity
Cappe, C., Murray, M. M., Barone, P., & Rouiller, E. M. (2010) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21423 · Cited by 45
“…detection (e.g., Cappe, Thut, Romei, & Murray, 2009; Romei, Murray, Cappe, & Thut, 2009; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Stein, London, Wilkinson, & Price, 1996; Sumby & Pollack, 1954), the ability to integrate cues across multiple senses is likely advantageous for survival.”
Read my lips: Visual speech influences word processing in infants
Weatherhead, D., & White, K. S. (2017) · Cognition · doi:10.1016/j.cognition.2017.01.002 · Cited by 42
“One source of information that contributes to the robustness of this process in adults is visual articulatory information (Sumby & Pollack, 1954).”
BackgroundPekkarinen, E., Salmivalli, A. J., & Suonpää, J. T. K. (1990) · Scandinavian Audiology · doi:10.3109/01050399009070749 · Cited by 38
Schelinski, S., Riedel, P., & von Kriegstein, K. (2014) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2014.09.031 · Cited by 37
“The findings imply that successful processing of visual speech is not only a key requirement for robust face-to-face communication (Schwartz et al., 2004; Sumby and Pollack, 1954; Wayne and Johnsrude, 2012), but also in S.”
BackgroundMechanisms of enhancing visual–speech recognition by prior auditory information
Blank, H., & von Kriegstein, K. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.09.047 · Cited by 42
“For example, if we are in a noisy environment, such as a crowded pub, the information from the moving face helps us “hear” what is said (Ross et al., 2007; Sumby and Pollack, 1954).”
BackgroundHertrich, I., Dietrich, S., Moos, A., Trouvain, J., & Ackermann, H. (2009) · Neurocase · doi:10.1080/13554790802709054 · Cited by 44
“Under adequate conditions, lip reading, e.g., enhances the intelligibility of spoken language in noisy environments (Sumby & Pollack, 1954) or elicits auditory illusions such as the McGurk effect (MacDonald & McGurk, 1978).”
BackgroundCharbonneau, G., Véronneau, M. H., Boudrias-Fournier, C., Leporé, F., & Collignon, O. (2013) · Journal of Vision · doi:10.1167/13.12.20 · Cited by 45
“The respective reliability of distinct sensory inputs strongly influences how they are merged in a single coherent percept (Sumby & Pollack, 1954).”
BackgroundAudio-visual speech perception in adult readers with dyslexia: an fMRI study
Rüsseler, J., Ye, Z., Gerth, I., Szycik, G. R., & Münte, T. F. (2017) · Brain Imaging and Behavior · doi:10.1007/s11682-017-9694-y · Cited by 38
“For example, seeing articulatory gestures has long been known to improve intelligibility of speech especially in noisy environments (Sumby and Pollack 1954).”
BackgroundInfluentialPerceptual training enhances temporal acuity for multisensory speech
Niear, M. A. D., Gupta, P. B., Baum, S. H., & Wallace, M. T. (2017) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2017.10.016 · Cited by 40
“…everyday lives when conversing as the integration of auditory and visual cues enhances our comprehension of speech stimuli (Grant & Seitz, 2000; Schwartz, Berthommier, & Savariaux, 2004), particularly in noisy environments (Ross, SaintAmour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954).”
BackgroundRapid recalibration to audiovisual asynchrony follows the physical—not the perceived—temporal order
der Burg, E. V., Alais, D., & Cass, J. (2018) · Attention Perception & Psychophysics · doi:10.3758/s13414-018-1540-9 · Cited by 38
Shavit-Cohen, K., & Golumbic, E. Z. (2019) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2019.00386 · Cited by 34
Audiovisual Enhancement of Speech Perception in Noise by School-Age Children Who Are Hard of Hearing
Lalonde, K., & McCreery, R. W. (2020) · Ear and Hearing · doi:10.1097/aud.0000000000000830 · Cited by 37
Audio-visual speech recognition using an infrared headset
Huang, J., Potamianos, G., Connell, J. H., & Neti, C. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.007 · Cited by 41
“Motivated by the bimodality (auditory and visual) of human speech perception (Sumby and Pollack, 1954) and by the need for robust automatic speech recognition (ASR) in noisy environments, significant research effort has recently been directed into the study of audio-visual automatic speech”
Background‘Seeing tunes.’ The role of visual gestures in tune interpretation
Borràs-Comes, J., & Prieto, P. (2011) · Laboratory Phonology Journal of the Association for Laboratory Phonology · doi:10.1515/labphon.2011.013 · Cited by 35
Acoustic Cues to Place of Articulation and the McGurk Effect
Green, K. P., & Norrix, L. W. (1997) · Journal of Speech Language and Hearing Research · doi:10.1044/jslhr.4003.646 · Cited by 37
Laterality in visual speech perception.
Smeele, P. M. T., Massaro, D. W., Cohen, M. M., & Sittig, A. C. (1998) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.24.4.1232 · Cited by 37
“The influence of visual speech occurs in a wide variety of conditions: in noisy environments (Binnie et al., 1974; Breeuwer & Plomp, 1984; Sumby & Pollack, 1954 ), with highly complex sentences (Reisberg, McLean, & Goldfield, 1987), with conflicting auditory and visual speech (Green & Kuhl, 1989; Massaro, 1987; McGurk…”
BackgroundGrant, K. W., Tufts, J. B., & Greenberg, S. (2007) · The Journal of the Acoustical Society of America · doi:10.1121/1.2405859 · Cited by 37
Looking where you listen: visual cues and auditory attention
Reisberg, D. (1978) · Acta Psychologica · doi:10.1016/0001-6918(78)90007-0 · Cited by 37
Hadamard Product in Deep Learning: Introduction, Advances and Challenges
Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 59
“Early studies showed that the interaction between auditory and visual information is non-linear rather than purely additive [117], [118].”
Visibility of speech articulation enhances auditory phonetic convergence
Dias, J. W., & Rosenblum, L. D. (2015) · Attention Perception & Psychophysics · doi:10.3758/s13414-015-0982-6 · Cited by 41
“Another possibility is that despite the lack of a reduction in phonetic identification, noise may still have impacted the information and processes used in the recovery of phonetic information (e.g., French & Steinberg, 1947; G. A. Miller et al., 1951; Sumby & Pollack, 1954).”
Interaction between auditory and motor systems in speech perception
Wu, Z., Chen, M.-L., Wu, X. H., & Li, L. (2014) · Neuroscience Bulletin · doi:10.1007/s12264-013-1428-6 · Cited by 42
“Also, another study focused on the role of articulatory organ movement in a noisy environment([7]).”
BackgroundOn the neurobiological investigation of language understanding in context
Small, S. L., & Nusbaum, H. C. (2003) · Brain and Language · doi:10.1016/s0093-934x(03)00344-4 · Cited by 38
“So perhaps it should not be surprising that visual information showing movements of the mouth and lips during talking enhances speech comprehension, even though we often think of speech perception as being infallible based on the acoustic signal alone (Sumby & Pollack, 1954; Summerfield, 1992).”
BackgroundAndres, A., Cardy, J. O., & Joanisse, M. F. (2010) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2010.09.012 · Cited by 37
“For example, Sumby and Pollack (1954) found that speech perception in noise was significantly improved when participants were able to see the corresponding lip movements.”
BackgroundDetermining the influence of Gaussian blurring on inversion effects with talking faces
Thomas, S. M., & Jordan, T. R. (2002) · Perception & Psychophysics · doi:10.3758/bf03196797 · Cited by 36
“A great deal of research suggests that the influence of visual speech on auditory speech recognition increases when the signal-tonoise ratio of the auditory signal decreases (e.g., Jordan & Sergeant, 1998; MacLeod & Summerfield, 1987, 1990; Middleweerd & Plomp, 1987; Sumby & Pollack, 1954 ).”
BackgroundPisoni, D. B. (1996) · Language and Cognitive Processes · doi:10.1080/016909696387097 · Cited by 36
“Other Comments Speech intelligibility tests have also been used to study multi-modal integration of auditory and visual information (Sumby & Pollack, 1954) and to place the problem of speech perception within the larger context of event perception and recent developments in ecological psychology…”
BackgroundMethodMeronen, A., Tiippana, K. M., Westerholm, J., & Ahonen, T. P. S. (2012) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2012/11-0270 · Cited by 48
Bernstein, L. E., Lu, Z., & Jiang, J. (2008) · Brain Research · doi:10.1016/j.brainres.2008.04.018 · Cited by 37
“In addition, word intelligibility—that is, ease of word recognition—varies across unisensory versus multisensory conditions (Sumby and Pollack, 1954), and intelligibility affects cortical activity levels (Scott et al.”
InfluentialVisual speech primes open-set recognition of spoken words
Buchwald, A. B., Winters, S. J., & Pisoni, D. B. (2008) · Language and Cognitive Processes · doi:10.1080/01690960802536357 · Cited by 41
“…and extends previous results in the literature indicating that speech perception is not limited to the auditory modality (e.g., Bernstein, 2005; Massaro, 1987, 1998; Sumby & Pollack, 1954), and that visual speech can prime perception of auditory speech produced by the same event (Kim et al., 2004).”
BackgroundResultBurkholder, R. A., & Pisoni, D. B. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179873.003.0014 · Cited by 42
Pardo, J. S., & Remez, R. E. (2006) · Elsevier eBooks · doi:10.1016/b978-012369374-7/50008-0 · Cited by 43
Multiexpert Automatic Speech Recognition Using Acoustic and Myoelectric Signals
Chan, A. D. C., Englehart, K. B., Hudgins, B. S., & Lovely, D. F. (2006) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2006.870224 · Cited by 36
“Research has shown that speech intelligibility improves when both of these sources of information are used, especially under acoustically noisy conditions [14], [15].”
BackgroundAudio-to-visual conversion for multimedia communication
Rao, R. R., Chen, T., & Mersereau, R. M. (1998) · IEEE Transactions on Industrial Electronics · doi:10.1109/41.661300 · Cited by 36
“One early study [ 7 ] found that human speech recognition performance degrades less rapidly in the prescence of increasing noise when viewing the speaker.”
BackgroundLexical Competition and Reduction in Speech: A Preliminary Report 1
WRIGHT, R. A. (1997) · Cited by 36
Spence, C. J. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn214 · Cited by 44
Nahorna, O., Berthommier, F., & Schwartz, J.-L. (2015) · The Journal of the Acoustical Society of America · doi:10.1121/1.4904536 · Cited by 35
“This framework provided the basis for explaining the results of the two main paradigms 55 for the study of audiovisual interactions: speech perception in noisy conditions, in which 56 the visual input enhances the intelligibility of auditory input degraded by acoustic noise 57 (Sumby and Pollack, 1954; Erber, 1969; Be…”
BackgroundDegrading phonetic information affects matching of audiovisual speech in adults, but not in infants
Baart, M., Vroomen, J., Shaw, K. E., & Bortfeld, H. (2013) · Cognition · doi:10.1016/j.cognition.2013.09.006 · Cited by 41
“…that both the adult and infant brain combine auditory and visual speech into a single percept (e.g., Burnham & Dodd, 2004; Burnham & Sekiyama, 2004; Kuhl & Meltzoff, 1982, 1984; McGurk & MacDonald, 1976; Patterson & Werker, 1999, 2003; Rosenblum, Schmuckler, & Johnson, 1997; Sumby & Pollack, 1954).”
Reetzke, R., Lam, B. P. W., Xie, Z., Sheng, L., & Chandrasekaran, B. (2016) · PLoS ONE · doi:10.1371/journal.pone.0168048 · Cited by 38
Improving Lip-reading Performance for Robust Audiovisual Speech Recognition using DNNs
Thangthai, K., Harvey, R., Cox, S., & Theobald, B.-J. (2015) · UEA Digital Repository (University of East Anglia) · Cited by 35
Adeel, A., Ahmad, J. E., Larijani, H., & Hussain, A. (2019) · Cognitive Computation · doi:10.1007/s12559-019-09653-z · Cited by 42
“The correlation between the visible properties of the articulatory organs (e.g. lips, teeth, tongue) and speech reception has been previously shown in numerous behavioural studies [11–14].”
BackgroundHadley, L. V., Whitmer, W. M., Brimijoin, W. O., & Naylor, G. (2020) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01821-9 · Cited by 44
Pragmatics of conversation and communication in noisy settings
McKellin, W. H., Shahin, K. N., Hodgson, M. R., Jamieson, J. R., & Pichora-Fuller, M. K. (2007) · Journal of Pragmatics · doi:10.1016/j.pragma.2006.11.012 · Cited by 35
Visual and audiovisual speech perception with color and gray-scale facial images
Jordan, T. R., Mccotter, M. V., & Thomas, S. M. (2000) · Perception & Psychophysics · doi:10.3758/bf03212141 · Cited by 35
Lip movement synthesis from speech based on hidden Markov models
Yamamoto, E., Nakamura, S., & Shikano, K. (1998) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670941 · Cited by 145
“PII: S 0 1 6 7 - 6 3 9 3 ( 9 8 ) 0 0 0 5 4 - 5 noise (Sumby and Pollack, 1954; Neely, 1956; Erber, 1969).”
Audiovisual spoken word recognition by children with cochlear implants
Kirk, K. I., Hay-McCutcheon, M. J., Holt, R. F., Gao, S., Qi, R. A., & Gerlain, B. L. (2007) · Hearing Balance and Communication · doi:10.1080/16513860701673892 · Cited by 35
“Sumby and Pollack (3) were the first researchers to document that the addition of visual speech cues to the auditory speech signal yielded substantial improvements in speech perception for listeners with normal hearing, especially in adverse listening conditions such as noise.”
BackgroundThe teleface project multi-modal speech-communication for the hearing impaired
Beskow, J., Dahlquist, M., Granström, B., Lundeberg, M., Spens, K.-E., & Öhman, T. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-531 · Cited by 35
“The amount of visual information from lip reading has been described as a function o f the signal-to-noise level [ 1 ,2].”
BackgroundSpeech sounds alter facial skin sensation
Ito, T., & Ostry, D. J. (2011) · Journal of Neurophysiology · doi:10.1152/jn.00029.2011 · Cited by 38
A possible neurophysiological correlate of audiovisual binding and unbinding in speech perception
Ganesh, A. C., Berthommier, F., Vilain, C., Sato, M., & Schwartz, J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01340 · Cited by 34
“…subjects concerns the role of lip reading during speech comprehension, with a gain in the AV modality in respect to the audio-only modality particularly in adverse listening conditions (e.g., Sumby and Pollack, 1954; Erber, 1971; Benoît et al., 1994; Grant and Seitz, 2000; Bernstein et al., 2004b).”
BackgroundAudiovisual Non-Verbal Dynamic Faces Elicit Converging fMRI and ERP Responses
Brefczynski‐Lewis, J. A., Lowitszch, S., Parsons, M. W., Lemieux, S. K., & Puce, A. (2009) · Brain Topography · doi:10.1007/s10548-009-0093-6 · Cited by 37
“explanation in the light of the behavioral facilitation effects observed in many studies (Bolognini et al. 2005; Gondan et al. 2005; Grant and Walden 1996; Miller 1982; Sumby and Pollack 1954), although all three explanations are plausible and cannot be differentiated in the current Fig.”
BackgroundMethodInfluentialBarutchu, A., Crewther, S. G., Kiely, P. M., Murphy, M., & Crewther, D. P. (2007) · The European Journal of Cognitive Psychology · doi:10.1080/09541440601125623 · Cited by 38
“It is well accepted that visual information, such as seeing a speaker's lips move, can facilitate auditory speech perception particularly in noisy environments (eg, Binnie, Montgomery, & Jackson, 1974 ; Sumby & Pollack, 1954 ).”
An Assessment of Behavioral Dynamic Information Processing Measures in Audiovisual Speech Perception
Altieri, N. A., & Townsend, J. T. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00238 · Cited by 35
“A vocabulary size of eight monosyllabic words was used in this study – one of the vocabulary sizes used by Sumby and Pollack (1954).”
BackgroundMethodResultInfluentialKelly, S., Bailey, A., & Hirata, Y. (2017) · Collabra: Psychology · doi:10.1525/collabra.76 · Cited by 34
“This make sense given research on native language processing showing that people rely on visual information more when they struggle to process auditory information (Sumby & Pollack, 1954; and specifically for hand gesture, Drijvers & Özyürek, 2016; Obermeier, Dolk, & Gunter, 2012).”
BackgroundAudiovisual speech perception: A new approach and implications for clinical populations
IRWIN, J. R., & DiBlasi, L. (2017) · Language and Linguistics Compass · doi:10.1111/lnc3.12237 · Cited by 41
“Over 60 years ago, Sumby and Pollack (1954) reported that visible articulatory information assists listeners in the identification of speech in auditory noise, creating a “visual gain” over auditory speech alone (also see Erber, 1975; Grant & Seitz, 2000; MacLeod & Summerfield, 1987; Payton,…”
BackgroundEffects of aging on audio-visual speech integration
Huyse, A., Leybaert, J., & Berthommier, F. (2014) · The Journal of the Acoustical Society of America · doi:10.1121/1.4894685 · Cited by 34
Perception of audiovisual speech synchrony for native and non-native language
Navarra, J., Alsius, A., Velasco, I., Soto‐Faraco, S., & Spence, C. J. (2010) · Brain Research · doi:10.1016/j.brainres.2010.01.059 · Cited by 41
Auditory–visual speech integration by adults with and without language-learning disabilities
Norrix, L. W., Plante, E., & Vance, R. (2005) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2005.05.003 · Cited by 35
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.