Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 401–500 of 1,005
Biometric Applications Related to Human Beings: There Is Life beyond Security
Faundez-Zanuy, M., Hussain, A., Mekyska, J., Sesa-Nogueras, E., Monte-Moreno, E., Esposito, A., Chetouani, M., Garre-Olmo, J., Abel, A., Smekal, Z., et al. (2013) · Cognitive Computation · doi:10.1007/s12559-012-9169-9 · Cited by 66
“works in 1954 [87] and subsequent developments such as the McGurk effect [56].”
BackgroundNeural and behavioral correlates of auditory categorization
Russ, B. E., Lee, Y. S., & Cohen, Y. E. (2007) · Hearing Research · doi:10.1016/j.heares.2006.10.010 · Cited by 72
“For instance, in humans, speech perception is enhanced when auditory information is combined with visual information (Calvert et al., 1997; McGurk and MacDonald, 1976; Sumby and Pollack, 1954).”
BackgroundPerceptual assimilation of lexical tone: The roles of language experience and visual information
Reid, A., Burnham, D. K., Kasisopa, B., Reilly, R., Attina, V., Rattanasone, N. X., & Best, C. T. (2014) · Attention Perception & Psychophysics · doi:10.3758/s13414-014-0791-3 · Cited by 68
“Visual speech information is known to benefit speech perception, both under difficult listening conditions (e.g., Sumby & Pollack, 1954) and in completely undegraded (McGurk & McDonald, 1976) conditions (see Campbell, Dodd, & Burnham, 1998, and Bailly, Perrier, & Vatikiotis-Bateson, 2012, for a…”
BackgroundStevenson, R. A., Bushmakin, M. A., Kim, S., Wallace, M. T., Puce, A., & James, T. W. (2012) · Brain Topography · doi:10.1007/s10548-012-0220-7 · Cited by 73
“Visual speech increases the ability of a listener to perceive an utterance correctly (Sumby and Pollack 1954), increases the speed at which a listener perceives an utterance (Soto-Faraco et al.”
BackgroundLanguage familiarity modulates relative attention to the eyes and mouth of a talker
Barenholtz, E., Mavica, L., & Lewkowicz, D. J. (2015) · Cognition · doi:10.1016/j.cognition.2015.11.013 · Cited by 71
“Indeed, studies show that concurrent access to redundant audible and visible speech cues enhances speech perception under noisy conditions (Middelweerd & Plomp, 1987; Rosenblum, 2008; Rosenblum, Johnson, & Saldana, 1996; Sumby & Pollack, 1954; Summerfield, 1979).”
Conversational speech levels and signal-to-noise ratios in realistic acoustic conditions
Weisser, A., & Buchholz, J. M. (2019) · The Journal of the Acoustical Society of America · doi:10.1121/1.5087567 · Cited by 69
Auditory perception in the aging brain: the role of inhibition and facilitation in early processing
Stothart, G., & Kazanina, N. (2016) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2016.06.022 · Cited by 75
“Viewing a speaker while listening to continuous speech has been shown to be equivalent to a 15 dB increase in the auditory signal (Sumby and Pollack, 1954), whereas simply observing silent visual speech articulation activates the primary and association auditory cortices (Calvert et al.”
Bidelman, G. M. (2016) · Experimental Brain Research · doi:10.1007/s00221-016-4705-6 · Cited by 73
LIPS, TEETH, AND THE BENEFITS OF LIPREADING
Summerfield, Q., MacLeod, A., McGrath, M., & Brooke, M. (1989) · Handbook of Research on Face Processing · doi:10.1016/b978-0-444-87143-5.50019-6 · Cited by 64
Enhanced perception of pitch changes in speech and music in early blind adults
Arnaud, L., Gracco, V. L., & Ménard, L. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.06.009 · Cited by 71
“, 1992) and in faceto-face communication (Hubbard et al., 2009; McNeill, 1992; Sumby and Pollack, 1954) - compensatory mechanisms in the auditory”
LIPS2008: visual speech synthesis challenge
Theobald, B.-J., Fagel, S., Bailly, G., & Elisei, F. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-590 · Cited by 63
“Visual information is known to be especially helpful in environments with degraded audio [19, 20].”
BackgroundSpeech segmentation is facilitated by visual cues
Cunillera, T., Càmara, E., Laine, M., & Rodríguez‐Fornells, A. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210902888809 · Cited by 66
Everdell, I. T., Marsh, H., Yurick, M. D., Munhall, K. G., & Paré, M. (2007) · Perception · doi:10.1068/p5852 · Cited by 63
“Although speech research has been focused on auditory processing, visual information plays a vital role in communication and can affect perceived speech (eg Sumby and Pollack 1954; McGurk and MacDonald 1976).”
BackgroundAuditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus
Venezia, J. H., Vaden, K. I., Rong, F., Maddox, D., Saberi, K., & Hickok, G. S. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00174 · Cited by 74
“influenced by visual speech information (Sumby and Pollack, 1954; McGurk and MacDonald, 1976; Dodd, 1977; Reisberg et al., 1987; Callan et al., 2003), which is represented in part within biological motion circuits that specify the shape and position of vocal tract articulators.”
BackgroundIverson, P., Bernstein, L. E., & Auer, E. T. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00049-1 · Cited by 64
Social fidelity in virtual agents: Impacts on presence and learning
Sinatra, A. M., Pollard, K. A., Files, B. T., Oiknine, A. H., Ericson, M. A., & Khooshabeh, P. (2020) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106562 · Cited by 80
Lending a helping hand to hearing: another motor theory of speech perception
Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511541599.009 · Cited by 70
“Adding visible facial movements to speech enhances speech recognition as much as removing up to 20 dB of noise from the auditory signal (Sumby and Pollack, 1954).”
de Boisferon, A. H., Tift, A. H., Minar, N. J., & Lewkowicz, D. J. (2016) · Developmental Science · doi:10.1111/desc.12381 · Cited by 68
“…speech is more salient because it consists of overlapping and highly redundant auditory and visual information (Chandrasekaran, Trubanova, Stillittano, Caplier & Ghazanfar, 2009; Munhall & Vatikiotis-Bateson, 2004; Sumby & Pollack, 1954; Summerfield, 1979; Yehia, Rubin & Vatikiotis-Bateson, 1998).”
BackgroundSpeech Understanding in Complex Listening Environments by Listeners Fit With Cochlear Implants
Dorman, M. F., & Gifford, R. H. (2017) · Journal of Speech Language and Hearing Research · doi:10.1044/2017_jslhr-h-17-0035 · Cited by 62
“Visual information provides significant value for speech understanding in noise with improvements up to 15 dB in speech reception threshold even for listeners with normal hearing (Sumby & Pollack, 1954).”
BackgroundAnalysis, synthesis, and perception of visible articulatory movements
Brooke, N. M., & Summerfield, A. Q. (1983) · Journal of Phonetics · doi:10.1016/s0095-4470(19)30777-6 · Cited by 62
Breaking the Robustness Barrier: Recent Progress on the Design of Robust Multimodal Systems
Oviatt, S. L. (2002) · Advances in computers · doi:10.1016/s0065-2458(02)80009-2 · Cited by 69
“In contrast, research on multimodal speech and lip movements has been driven largely by cognitive science interest in intersensory audio-visual perception, and the coordination of speech output with lip and facial movements [5,7,27-36].”
BackgroundInfluentialMagnée, M. J. C. M., Oranje, B., van Engeland, H., Kahn, R. S., & Kemner, C. (2009) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2009.02.012 · Cited by 65
“It is for instance shown that seeing a speaker’s face increases the intelligibility of speech in a noisy environment (Sumby & Pollack, 1954), and hearing an emotional tone of voice improves judgment accuracy of the corresponding facial expression (de Gelder & Vroomen, 2000).”
BackgroundMagnée, M. J. C. M., de Gelder, B., van Engeland, H., & Kemner, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024196 · Cited by 59
“Multisensory integration (MSI) of these stimuli enables better understanding of the social intentions of others [1], which is of particular importance for perception of visual and auditory emotional stimuli [2].”
BackgroundAudiovisual sentence recognition not predicted by susceptibility to the McGurk effect
Engen, K. J. V., Xie, Z., & Chandrasekaran, B. (2016) · Attention Perception & Psychophysics · doi:10.3758/s13414-016-1238-9 · Cited by 64
Monkeys and Humans Share a Common Computation for Face/Voice Integration
Chandrasekaran, C., Lemus, L., Wieckowski, A. T., Gondan, M., & Ghazanfar, A. A. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002165 · Cited by 67
“can provide considerable intelligibility benefits to the perception of auditory speech [6,7], faster reaction times [8,9], and is hard to”
BackgroundThe Acoustic and Visual Phonetic Basis of Place of Articulation in Excrescent Nasals
Johnson, K. A., DiCanio, C., & MacKenzie, L. (2007) · UC Berkeley Phonology Lab Annual Reports · doi:10.5070/p70n17f74x · Cited by 60
Speech perception using combinations of auditory, visual, and tactile information.
Blamey, P. J., Cowan, R., Alcantara, J. I., Whitford, L. A., & Clark, G. M. (1989) · PubMed · Cited by 60
Audiovisual integration in high functioning adults with autism
Keane, B. P., Rosenthal, O., Chun, N. H., & Shams, L. B. (2009) · Research in Autism · doi:10.1016/j.rasd.2009.09.015 · Cited by 76
“…Mathiak, & Lutzenberger, 2005; McGurk & MacDonald, 1976) and because audiovisual congruence causes bisensory facilitation (Kaiser et al., 2005; Sumby & Pollack, 1954; see Campbell, 2008 for a review), performance was expected to be faster and more accurate on congruent than on incongruent…”
Righi, G., Tenenbaum, E. J., McCormick, C. E. B., Blossom, M., Amso, D., & Sheinkopf, S. J. (2018) · Autism Research · doi:10.1002/aur.1918 · Cited by 66
“Naturalistic face-to-face speech processing is a multisensory experience that relies heavily on the integration of auditory and visual information [Erber, 1975; Schwartz et al, 2004; Sumby & Pollack, 1954].”
BackgroundProsody off the top of the head: Prosodic contrasts can be discriminated by head motion
Cvejic, E., Kim, J., & Davis, C. (2010) · Speech Communication · doi:10.1016/j.specom.2010.02.006 · Cited by 59
“Yet it is clear that, when available, visual information about speech production (visual speech) also plays a role in speech perception (Benoı̂t and Le Goff, 1998; Davis and Kim, 2004; Sumby and Pollack, 1954; Summerfield, 1992).”
BackgroundSomething in the way she moves
Munhall, K. G., & Buchan, J. N. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2003.12.009 · Cited by 59
Freeman, E. D., Ipser, A., Palmbaha, A., Paunoiu, D., Brown, P., Lambert, C., Leff, A. P., & Driver, J. (2013) · Cortex · doi:10.1016/j.cortex.2013.03.006 · Cited by 67
The Impact of Spatial Incongruence on an Auditory-Visual Illusion
Innes-Brown, H., & Crewther, D. P. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006450 · Cited by 68
“For instance, the detection of weak visual stimuli is enhanced with a concurrent auditory stimulus [1], and speech perception is improved when video of the moving lips is available [2,3].”
Audiovisual asynchrony detection in human speech.
Maier, J. X., Luca, M. D., & Noppeney, U. (2010) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0019952 · Cited by 71
Cue Integration in Categorical Tasks: Insights from Audio-Visual Speech Perception
Bejjanki, V. R., Clayards, M., Knill, D. C., & Aslin, R. N. (2011) · PLoS ONE · doi:10.1371/journal.pone.0019812 · Cited by 59
A review of the perceptual effects of hearing loss for frequencies above 3 kHz
Moore, B. C. J. (2016) · International Journal of Audiology · doi:10.1080/14992027.2016.1204565 · Cited by 59
“When the face of the talker is visible, so lip-reading is possible, the high-frequency components in the acoustic signal become relatively less important (Kryter, 1962; Sumby & Pollack, 1954).”
BackgroundShic, F., Wang, Q., Macari, S. L., & Chawarska, K. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13118 · Cited by 61
“In TD children access to redundant oro-motor cues is known to play an important role in language acquisition and speech processing and diminished mouth looking in toddlers is associated with later language disadvantages (de Boisferon et al., 2018; Sumby & Pollack, 1954; Tenenbaum et al., 2014).”
BackgroundZhang, Y., Frassinelli, D., Tuomainen, J., Skipper, J. I., & Vigliocco, G. (2021) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2021.0500 · Cited by 92
Lebib, R., Papo, D., de Bode, S., & Baudonnière, P.-M. (2003) · Neuroscience Letters · doi:10.1016/s0304-3940(03)00131-9 · Cited by 59
“This could result in an enhancement of speech intelligibility, notably in noisy surroundings [17,19].”
BackgroundInvestigating the audio–visual speech detection advantage
Kim, J., & Davis, C. (2004) · Speech Communication · doi:10.1016/j.specom.2004.09.008 · Cited by 58
“It has long been known that speech intelligibility is increased when both the audio and visual components of speech are presented to human observers compared to when only the auditory ed. component is presented (Sumby and Pollack, 1954).”
BackgroundLewkowicz, D. J., Minar, N. J., Tift, A. H., & Brandon, M. (2014) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2014.10.006 · Cited by 71
“Because of its multisensory redundancy, adults usually perceive audiovisual speech as a coherent entity and not as two distinct streams of information (McGurk & MacDonald, 1976; Rosenblum, 2008; Sumby & Pollack, 1954; Summerfield, 1979; Yehia et al., 1998).”
Eye movements of monkey observers viewing vocalizing conspecifics
Ghazanfar, A. A., Nielsen, K. J., & Logothetis, N. K. (2006) · Cognition · doi:10.1016/j.cognition.2005.12.007 · Cited by 61
“The data therefore suggest that the sensorimotor strategies underlying bimodal speech perception may have a homologous counterpart in a closely related primate ancestor.”
BackgroundPET-imaging of brain plasticity after cochlear implantation
Strelnikov, K., Marx, M., Lagleyre, S., Fraysse, B., Déguine, O., & Barone, P. (2014) · Hearing Research · doi:10.1016/j.heares.2014.10.001 · Cited by 66
“Indeed, visual information can improve speech intelligibility, especially when presented simultaneously with a noisy or degraded acoustic signal (Benoit et al., 1994; Grant and Braida, 1991; MacLeod and Summerfield, 1987; Ross et al., 2007; Sumby and Pollack, 1954; Summerfield, 1979).”
BackgroundEffects of Distance on Visual and Audiovisual Speech Recognition
Jordan, T. R., & Sergeant, P. (2000) · Language and Speech · doi:10.1177/00238309000430010401 · Cited by 58
“Seeing the face of a talker can improve intelligibility in a noisy environment (Erber, 1969; MacLeod & Summerfield, 1987, 1990; Middleweerd & Plomp, 1987; Sumby & Pollack, 1954) and help recover a difficult message even when the auditory signal is clear (Reisberg, McLean, & Goldfield, 1987).”
BackgroundResultSmiljanić, R., Keerstock, S., Meemann, K., & Ransom, S. M. (2021) · The Journal of the Acoustical Society of America · doi:10.1121/10.0005191 · Cited by 63
“Masks also obscure visual cues that aid speech recognition by providing supplementary information about speech sounds missing in the auditory signal (Sumby and Pollack, 1954).”
BackgroundStrelnikov, K., Rouger, J., Barone, P., & Déguine, O. (2009) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.2009.00741.x · Cited by 70
Strand, J. F., Cooperman, A. W., Rowe, J., & Simenstad, A. (2014) · Journal of Speech Language and Hearing Research · doi:10.1044/2014_jslhr-h-14-0059 · Cited by 64
“Instead, visual information about a speaker, suchas mouth shape and facial movements, supplement the auditory signal (Sumby & Pollack, 1954).”
BackgroundDynamic faces speed up the onset of auditory cortical spiking responses during vocal detection
Chandrasekaran, C., Lemus, L., & Ghazanfar, A. A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1312518110 · Cited by 55
Influence of Visual Information on the Intelligibility of Dysarthric Speech
Keintz, C., Bunton, K., & Hoit, J. D. (2007) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2007/027 · Cited by 57
Leybaert, J., & LaSasso, C. J. (2010) · Trends in Amplification · doi:10.1177/1084713810375567 · Cited by 92
“Despite that input being highly unnatural and impoverished relative to the normally functioning cochlea, most adults who had hearing ability prior to their hearing loss can understand speech well enough to converse easily by telephone (Moore & Shannon, 2009).”
BackgroundThe influence of bilingualism on the preference for the mouth region of dynamic faces
Ayneto, A., & Sebastián‐Gallés, N. (2016) · Developmental Science · doi:10.1111/desc.12446 · Cited by 65
Age-related differences in inhibitory control predict audiovisual speech perception.
Dey, A., & Sommers, M. S. (2015) · Psychology and Aging · doi:10.1037/pag0000033 · Cited by 62
“This complementarity of available information from two modalities becomes especially important in the presence of noise, wherein perception of the A-only signal is more prone to degradation (Sumby & Pollack, 1954; Erber, 1969).”
BackgroundMethodBenoı̂t, C., & Goff, B. L. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00045-4 · Cited by 57
“He used the same protocol as that widely used for natural speech where speech stimuli are presented to normal hearing subjects, either auditorily or audio-visually, with various levels of background noise (Sumby and Pollack, 1954; Erber, 1969; BenoõÃt et al., 1994).”
BackgroundMethodInfluentialGreen, K. P., & Gerdman, A. (1995) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037//0096-1523.21.6.1409 · Cited by 57
Automatic audiovisual integration in speech perception
Gentilucci, M., & Cattaneo, L. (2005) · Experimental Brain Research · doi:10.1007/s00221-005-0008-z · Cited by 69
“Although humans are able to understand words without any visual input, audiovisual perception is shown to improve language comprehension (Sumby and Pollack 1954), even when the acoustic information is perfectly clear (Reisberg et al. 1987).”
BackgroundEvent Related Potentials Index Rapid Recalibration to Audiovisual Temporal Asynchrony
Simon, D. M., Noel, J., & Wallace, M. T. (2017) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2017.00008 · Cited by 59
EEG can predict speech intelligibility
Iotzov, I., & Parra, L. C. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab07fe · Cited by 65
“…previously found to capture a significant fraction of the EEG evoked response in video (Dmochowski et al 2017, Poulsen et al 2017), and because it captures lip and head movements, which can both aid speech recognition in noise (Sumby and Pollack 1954, Munhall et al 2004, Thomas and Jordan 2004).”
BackgroundAuditory, visual and audiovisual clear speech
Gagné, J., Rochette, A.-J., & Charest, M. (2002) · Speech Communication · doi:10.1016/s0167-6393(01)00012-7 · Cited by 56
Audiovisual gating and the time course of speech perception
Munhall, K. G., & Tohkura, Y. (1998) · The Journal of the Acoustical Society of America · doi:10.1121/1.423300 · Cited by 64
Embodied, Embedded Language Use
Fowler, C. A. (2010) · Ecological Psychology · doi:10.1080/10407413.2010.517115 · Cited by 63
“First, speech in noise is more intelligible if the listener can see the speaker (eg, Sumby & Pollack, 1954 ).”
BackgroundA Survey of Research on Lipreading Technology
Hao, M., Mamut, M., Yadikar, N., Aysa, A., & Ubul, K. (2020) · IEEE Access · doi:10.1109/access.2020.3036865 · Cited by 64
“The basic theory of lipreading in 1954 was proposed by Sumby and Pollack [9], and it was first proposed that the features of lip motion could be used to identify the speaker’s speech content.”
BackgroundSenkowski, D., Saint-Amour, D., Gruber, T., & Foxe, J. J. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.06.046 · Cited by 55
“degraded or masked by noise (Ross et al., 2007a, 2007b; Sumby and Pollack, 1954).”
BackgroundRapid temporal recalibration is unique to audiovisual stimuli
der Burg, E. V., Orchard-Mills, E., & Alais, D. (2014) · Experimental Brain Research · doi:10.1007/s00221-014-4085-8 · Cited by 63
“For instance, under noisy circumstances, speech comprehension is improved by observing the speaker’s lip movements (Sumby and Pollack 1954).”
BackgroundThe temporal distribution of information in audiovisual spoken-word identification
Jesse, A., & Massaro, D. W. (2009) · Attention Perception & Psychophysics · doi:10.3758/app.72.1.209 · Cited by 66
“This relative audiovisual benefit was calculated at each gate as the amount of improvement between the audiovisual and auditory conditions set in relation to the overall possible amount of improvement [(audiovisual–auditory) /(100%–auditory); Sumby & Pollack, 1954 ].”
BackgroundMethodInfluentialVisual Cortical Entrainment to Motion and Categorical Speech Features during Silent Lipreading
O’Sullivan, A. E., Crosse, M. J., Liberto, G. M. D., & Lalor, E. C. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00679 · Cited by 52
“…syllables (Bernstein et al., 2004), words (Sumby and Pollack, 1954) and sentences (Grant and Seitz, 2000), and that this gain is present in both noisy (Sumby and Pollack, 1954; Ross et al., 2007; Crosse et al., 2016b) and noise-free conditions (Reisberg et al., 1987; Crosse et al., 2015a).”
BackgroundThe redeployment of attention to the mouth of a talking face during the second year of life
de Boisferon, A. H., Tift, A. H., Minar, N. J., & Lewkowicz, D. J. (2018) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2018.03.009 · Cited by 63
Electro-haptic enhancement of speech-in-noise performance in cochlear implant users
Fletcher, M. D., Hadeedi, A., Goehring, T., & Mills, S. R. (2019) · Scientific Reports · doi:10.1038/s41598-019-47718-z · Cited by 56
Speech Processing in the Auditory System: An Overview
Greenberg, S., & Ainsworth, W. A. (2006) · Springer eBooks · doi:10.1007/0-387-21575-1_1 · Cited by 55
Speechreading in the akinetopsic patient, L.M
Campbell, R. (1997) · Brain · doi:10.1093/brain/120.10.1793 · Cited by 55
“It improves the understanding of speech in noise (Sumby and Pollack, 1954) and under clear hearing conditions (Reisberget al., 1987).”
BackgroundEvans, T. A., Howell, S., & Westergaard, G. C. (2005) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.31.4.399 · Cited by 55
Psychophysics of the McGurk and other audiovisual speech integration effects.
Jiang, J., & Bernstein, L. E. (2011) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0023100 · Cited by 56
The role of visual spatial attention in audiovisual speech perception
Andersen, T., Tiippana, K. M., Laarni, J., Kojo, I., & Sams, M. E. (2008) · Speech Communication · doi:10.1016/j.specom.2008.07.004 · Cited by 62
“First, watching congruent articulations enhances speech perception (Sumby and Pollack, 1954).”
BackgroundSato, M., Buccino, G., Gentilucci, M., & Cattaneo, L. (2009) · Speech Communication · doi:10.1016/j.specom.2009.12.004 · Cited by 65
“For instance, seeing speaker’s mouth movements enhances speech intelligibility in noisy conditions (Sumby and Pollack, 1954; MacLeod and Summerfield, 1987; Benoı̂t et al., 1994).”
BackgroundProsody in the Auditory and Visual Domains: A Developmental Perspective
Esteve‐Gibert, N., & Guellaï, B. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00338 · Cited by 70
“For example, adults’ comprehension of speech in noisy and quiet environments is enhanced when they have access to the visual cues conveyed by the speaker’s face (Sumby and Pollack, 1954).”
BackgroundPaul, B. T., Chen, J., Le, T., Lin, V., & Dimitrijevic, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254162 · Cited by 65
Conscious access to the unisensory components of a cross-modal illusion
Soto‐Faraco, S., & Alsius, A. (2007) · Neuroreport · doi:10.1097/wnr.0b013e32801776f9 · Cited by 58
“Indeed, seeing articulatory gestures of the speaker improves oral comprehension of spoken messages [10,11].”
BackgroundPsychometric analysis of the mutually compensatory property of multimedia QoS
Tasaka, S., & Ito, Y. (2004) · International Conference on Communications · doi:10.1109/icc.2003.1203925 · Cited by 54
“An early example is a paper by Sumby and Pollack entitled “visual contribution to speech intelligibility in noise,” which was published in the Journal of the Acoustical Society of America in March 1954 [2].”
BackgroundHearing Foreign Voices: Does Knowing What is Said Affect Visual-Masked-Speech Detection?
Kim, J., & Davis, C. (2003) · Perception · doi:10.1068/p3466 · Cited by 59
Brimijoin, W. O., McShefferty, D., & Akeroyd, M. A. (2011) · Hearing Research · doi:10.1016/j.heares.2011.10.009 · Cited by 56
“…SNR also conflicts with a second possible constraint on real world orienting behavior: being able to see lip movements or related head movements helps speech comprehension substantially (e.g., Sumby and Pollack, 1954; Erber, 1969; Grant, 2001; MacLeod and Summerfield, 1987; Schwartz et al., 2004).”
Gated audiovisual speech identification in silence vs. noise: effects on time and accuracy
Moradi, S., Lidestam, B., & Rönnberg, J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00359 · Cited by 62
Du, Y., Kong, L. Z., Wang, Q., Wu, X. H., & Li, L. (2011) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2011.05.008 · Cited by 54
“, 2007), prior knowledge about the topic f the target sentence (Helfer and Freyman, 2008), and viewing a peaker’s movements of the speech articulators (Grant and Seitz, 000; Helfer and Freyman, 2005; Rosenblum et al., 1996; Rudmann t al., 2003; Sumby and Pollack, 1954; Summerfield, 1979, 1992).”
BackgroundThe processing of information from multiple sources in simultaneous interpreting
Jesse, A., Vrignaud, N., Cohen, M. M., & Massaro, D. W. (2000) · Interpreting International Journal of Research and Practice in Interpreting · doi:10.1075/intp.5.2.04jes · Cited by 53
Audio-Visual Sound Separation Via Hidden Markov Models
Hershey, J. R., & Casey, M. A. (2001) · neural information processing systems · Cited by 70
Attina, V., Beautemps, D., Cathiard, M.-A., & Odisio, M. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.013 · Cited by 53
Plass, J., Guzman-Martinez, E., Ortega, L., Grabowecky, M., & Suzuki, S. (2014) · Psychological Science · doi:10.1177/0956797614542132 · Cited by 55
“In two experiments, we demonstrated visual encoding of invisible lip movements by way of cross-modal facilitation of spoken-word categorization (e.g., Sumby & Pollack, 1954; see the Supplemental Material available online for additional methodology).”
Multisensory speech perception without the left superior temporal sulcus
Baum, S. H., Martin, R. C., Hamilton, A. C., & Beauchamp, M. S. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.05.034 · Cited by 57
“Speech can be understood through the auditory modality alone, but combining audition with vision improves speech perception (Grant and Seitz, 2000; Stein and Meredith, 1993; Sumby and Pollack, 1954).”
Rapid, generalized adaptation to asynchronous audiovisual speech
der Burg, E. V., & Goodbourn, P. T. (2015) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2014.3083 · Cited by 57
Some normative data on lip-reading skills (L)
Altieri, N. A., Pisoni, D. B., & Townsend, J. T. (2011) · The Journal of the Acoustical Society of America · doi:10.1121/1.3593376 · Cited by 52
“…from studies in audiovisual speech perception has shown that visual speech cues, provided by optical information in the talker’s face, has facilitatory effects in terms of accuracy across a wide range of auditory signal-to-noise ratios (Grant and Seitz, 1998; Sumby and Pollack, 1954).”
BackgroundÖzker, M., Schepers, I. M., Magnotti, J. F., Yoshor, D., & Beauchamp, M. S. (2017) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01110 · Cited by 54
“Visual speech information is particularly important in noisy environments in which the auditory speech is difficult to comprehend (Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Bernstein, Auer, & Takayanagi, 2004; Sumby & Pollack, 1954).”
An autism-associated serotonin transporter variant disrupts multisensory processing
Siemann, J. K., Muller, C. L., Forsberg, C. G., Blakely, R. D., Veenstra-VanderWeele, J., & Wallace, M. T. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.17 · Cited by 52
Top-down attention regulates the neural expression of audiovisual integration
Fernández, L. M., Visser, M., Ventura‐Campos, N., Ávila, C., & Soto‐Faraco, S. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.06.052 · Cited by 65
“…beyond the pure statistical prevalence of AV speech congruency, previous studies report the benefit of using AV information when perceiving speech during noisy situations (Ross et al., 2007; Sumby and Pollack, 1954) or when acquiring language, at very early stages of life (Teinonen et al., 2008).”
An articulation model for audiovisual speech synthesis—Determination, adjustment, evaluation
Fagel, S., & Clemens, C. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.006 · Cited by 52
“Since half a century it is known that audiovisual speech surpasses audioalone presentation in intelligibility (Sumby and Pollack, 1954).”
BackgroundMethodBadin, P., Elisei, F., Bailly, G., & Tarabalka, Y. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-70517-8_14 · Cited by 51
“More generally, the importance of vision in speech perception has been demonstrated in a large number of studies: [ 5 ], and more recently [6], among others, have quantified the gain in speech intelligibility provided by the vision of lips and face in comparison with the sole acoustic signal (e.g.”
ResultBaşkent, D., & Bazo, D. (2011) · Ear and Hearing · doi:10.1097/aud.0b013e31820fca23 · Cited by 56
“Visual speech cues provide significant gain in speech understanding, especially in difficult listening conditions (Sumby & Pollack 1954; Erber 1969; Sanders & Goodrich 1971; Breeuwer & Plomp 1984; Helfer 1997; Schwartz et al. 2004; Helfer & Freyman 2005; Ross et al. 2007).”
BackgroundAudiovisual Speech Perception in Elderly Cochlear Implant Recipients
Hay-McCutcheon, M. J., Pisoni, D. B., & Kirk, K. I. (2005) · The Laryngoscope · doi:10.1097/01.mlg.0000173197.94769.ba · Cited by 51
Auditory-Visual Integration for Speech by Children With and Without Specific Language Impairment
Norrix, L. W., Plante, E., Vance, R., & Boliek, C. A. (2007) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2007/111 · Cited by 51
Matchin, W., Groulx, K., & Hickok, G. S. (2013) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00515 · Cited by 54
“Visible mouth movements provide information regarding the phonemic identity of auditory speech sounds and have been demonstrated to improve the perception of heard speech (Sumby & Pollack, 1954).”
Visual speech segmentation: using facial cues to locate word boundaries in continuous speech
Mitchel, A. D., & Weiss, D. J. (2013) · Language Cognition and Neuroscience · doi:10.1080/01690965.2013.791703 · Cited by 54
“In addition, visual speech displays have been shown to facilitate speech perception (Sumby & Pollack, 1954) and phoneme discrimination (Teinonen, Aslin, Alku, & Csibra, 2008), suggesting that observers are able to utilise visemic information to assist in acquiring auditory information.”
BackgroundDifferential Gaze Patterns on Eyes and Mouth During Audiovisual Speech Segmentation
Lusk, L. G., & Mitchel, A. D. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00052 · Cited by 58
“Visual speech has been shown to enhance a wide variety of speech and language processes, including speech comprehension (Sumby and Pollack, 1954), phoneme categorization (Patterson and Werker, 2003), language discrimination (Soto-Faraco et al., 2007; Weikum et al., 2007; Navarra et al., 2014), and…”
BackgroundAudiovisual speech integration in the superior temporal region is dysfunctional in dyslexia
Ye, Z., Rüsseler, J., Gerth, I., & Münte, T. F. (2017) · Neuroscience · doi:10.1016/j.neuroscience.2017.05.017 · Cited by 56
“…speech processing, i.e. whether watching a speaker’s lip movement can improve the comprehension of the speaker’s utterance in a noisy environment (Sumby and Pollack, 1954; Ma et al., 2009) or whether receiving mismatched phonemes and lip movements can lead to the perception of illusory…”
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.