Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 301–400 of 1,005
Informational masking of speech in children: Auditory-visual integration
Wightman, F. L., Kistler, D., & Brungart, D. S. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2195121 · Cited by 90
“…integration of the information from the auditory signal and the visual cues obtained from speechreading (lipreading) can provide a substantial improvement in speech understanding in a background of noise (Sumby and Pollack, 1954; Summerfield, 1979) or multitalker babble (Sommers et al., 2005).”
BackgroundContributions of Oral and Extraoral Facial Movement to Visual and Audiovisual Speech Perception.
Thomas, S. M., & Jordan, T. R. (2004) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.30.5.873 · Cited by 91
“In particular, seeing the face of a talker can improve intelligibility of speech sounds in a noisy environment (Erber, 1969; MacLeod & Summerfield, 1987, 1990; Middleweerd & Plomp, 1987; Sumby & Pollack, 1954 ).”
BackgroundMethodHötting, K., Rösler, F., & Röder, B. (2003) · Experimental Brain Research · doi:10.1007/s00221-002-1261-z · Cited by 88
“The additional visual cues significantly improve comprehension, especially in noisy conditions (Sumby and Pollack 1954).”
BackgroundHearing Lips: Gamma-band Activity During Audiovisual Speech Perception
Kaiser, J., Hertrich, I., Ackermann, H., Mathiak, K., & Lutzenberger, W. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhh166 · Cited by 87
Speech comprehension aided by multiple modalities: Behavioural and neural interactions
McGettigan, C., Faulkner, A., Altarelli, I., Obleser, J., Baverstock, H., & Scott, S. K. (2012) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2012.01.010 · Cited by 106
“…Grant & Seitz, 2000b; Hazan, Kim, & Chen, 2010; Helfer & Freyman, 2005; Kim & Davis, 2004; Kim, Davis, & Groot, 2009; Ma, Zhou, Ross, Foxe, & Parra, 2009; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Schwartz, Berthommier, & Savariaux, 2004; Sumby & Pollack, 1954; Thomas & Pilling, 2007).”
Audio–visual speech perception in schizophrenia: an fMRI study
Surguladze, S. A., Calvert, G. A., Brammer, M. J., Campbell, R., Bullmore, E. T., Giampietro, V., & David, A. S. (2001) · Psychiatry Research Neuroimaging · doi:10.1016/s0925-4927(00)00081-0 · Cited by 92
Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition
Zhang, Y., Yang, S., Xiao, J., Shan, S., & Chen, X. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00134 · Cited by 105
Velandia, M., Matthisen, A., Uvnäs‐Moberg, K., & Nissen, E. (2010) · Birth · doi:10.1111/j.1523-536x.2010.00406.x · Cited by 85
The Audiovisual Temporal Binding Window Narrows in Early Childhood
Lewkowicz, D. J., & Flom, R. A. (2013) · Child Development · doi:10.1111/cdev.12142 · Cited by 95
“In addition, studies have found that audiovisual redundancy enhances speech comprehension in adults (Sumby & Pollack, 1954; Summerfield, 1979) and that it affects discriminative responsiveness and learning in infants (Bahrick & Lickliter, 2000; Lewkowicz, 1985, 1986, 1992).”
Lipreading, Processing Speed, and Working Memory in Younger and Older Adults
Feld, J. E., & Sommers, M. S. (2009) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2009/08-0137 · Cited by 98
Top-down influences of written text on perceived clarity of degraded speech.
Sohoglu, E., Peelle, J. E., Carlyon, R. P., & Davis, M. H. (2013) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0033206 · Cited by 89
“…shown that lip movements, which typically precede arriving speech signals by 150 ms (Chandrasekaran, Trubanova, Stillittano, Caplier, & Ghazanfar, 2009), improve speech intelligibility in noise (Ma, Zhou, Ross, Foxe, & Parra, 2009; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Sumby, 1954).”
Listening comprehension and listening effort in the primary school classroom
Rudner, M., Lyberg-Åhlander, V., Brännström, J., Nirme, J., Pichora-Fuller, M. K., & Sahlén, B. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01193 · Cited by 61
Brown, V. A., Engen, K. V. V., & Peelle, J. (2021) · Cognitive Research · doi:10.1186/s41235-021-00314-0 · Cited by 86
“Acoustic challenges like background noise make this process more difficult, and in these cases intelligibility is facilitated by being able to see the talker (Erber, 1972; Sumby & Pollack, 1954).”
BackgroundBrain repair after stroke—a novel neurological model
Small, S. L., Buccino, G., & Solodkin, A. (2013) · Nature Reviews Neurology · doi:10.1038/nrneurol.2013.222 · Cited by 98
Audiovisual speech facilitates voice learning
Sheffert, S. M., & Olson, E. (2004) · Perception & Psychophysics · doi:10.3758/bf03194884 · Cited by 91
“…speech elements and improve word identification, particularly if the environment is noisy, if the listener has a hearing impairment, or if the auditory message is grammatically complex (Bernstein, Demorest, & Tucker, 2000; Erber, 1969; Reisberg, McLean, & Goldfield, 1987; Sumby & Pollack, 1954).”
BackgroundLinking neurons to behavior in multisensory perception: A computational review
Ma, W. J., & Pouget, A. (2008) · Brain Research · doi:10.1016/j.brainres.2008.04.082 · Cited by 98
“One of these is whether Bayesian models for cue integration can be applied to more complex stimuli, such as emotional state (De Gelder and Vroomen, 2000) or spoken syllables, either without (Ross et al., 2007; Sumby and Pollack, 1954) or with conflict (Massaro, 1987; 1998).”
Speech through ears and eyes: interfacing the senses with the supramodal brain
van Wassenhove, V. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00388 · Cited by 87
The benefit of head orientation to speech intelligibility in noise
Grange, J. A., & Culling, J. F. (2016) · The Journal of the Acoustical Society of America · doi:10.1121/1.4941655 · Cited by 89
Echoes of the spoken past: how auditory cortex hears context during speech perception
Skipper, J. I. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0297 · Cited by 82
“These are a visible form of context that listeners use to improve speech perception, equivalent to removing up to 20 dB of noise from the auditory signal [33].”
BackgroundMultisensory speech perception in autism spectrum disorder: From phoneme to whole‐word perception
Stevenson, R. A., Baum, S. H., Segers, M., Ferber, S., Barense, M. D., & Wallace, M. T. (2017) · Autism Research · doi:10.1002/aur.1776 · Cited by 82
“…Stevenson, Wallace, & Wenger, 2015; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007a; Ross et al., 2011; Stevenson & James, 2009; Stevenson et al., 2014e, 2015; Sumby & Pollack, 1954] and less effortful [Fraser, Gagne, Alepins, & Dubois, 2010] than when only auditory information is available.”
BackgroundInnes-Brown, H., Barutchu, A., Shivdasani, M. N., Crewther, D. P., Grayden, D. B., & Paolini, A. G. (2011) · Developmental Science · doi:10.1111/j.1467-7687.2011.01059.x · Cited by 95
“…weak visual stimuli can be enhanced by concurrently presented sounds (Frassinetti, Bolognini & Ladavas, 2002; Lovelace, Stein & Wallace, 2003) and the intelligibility of speech is improved when video of the appropriate lip movements is available (MacLeod & Summerfield, 1987; Sumby & Pollack, 1954).”
BackgroundHearing-impaired listeners show increased audiovisual benefit when listening to speech in noise
Puschmann, S., Daeglau, M., Stropahl, M., Mirkovic, B., Rosemann, S., Thiel, C., & Debener, S. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.04.017 · Cited by 75
Audio-Visual Speech Enhancement Using Conditional Variational Auto-Encoders
Sadeghi, M., Leglaive, S., Alameda-Pineda, X., Laurent Girin, B., & Horaud, R. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3000593 · Cited by 88
“[4]–[6], AVSE has been far less investigated than ASE.”
BackgroundWinneke, A. H., & Phillips, N. A. (2011) · Psychology and Aging · doi:10.1037/a0021683 · Cited by 96
“There is clear evidence that the availability of visual speech input in a noisy acoustic environment is perceptually equivalent to boosting the volume of the auditory speech by 10 to 15 dB (Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954).”
BackgroundResultThe effect of speechreading on masked detection thresholds for filtered speech
Grant, K. W. (2001) · The Journal of the Acoustical Society of America · doi:10.1121/1.1362687 · Cited by 81
“…due to visual speech cues may seem relatively small, a 1-3 dB improvement in masked thresholds may nevertheless lead to recognition of specific speech cues and understanding of a spoken message that might otherwise have been inaudible, and therefore, incomprehensible (Sumby and Pollack, 1954).”
BackgroundBadin, P., Tarabalka, Y., Elisei, F., & Bailly, G. (2010) · Speech Communication · doi:10.1016/j.specom.2010.03.002 · Cited by 81
“The talking head developed at the department was thus used in a audiovisual perception test based on the noise degradation paradigm used by [20] or [4].”
BackgroundMethodInfluentialInteraction of Audition and Vision for the Perception of Prosodic Contrastive Focus
Dohen, M., & Lœvenbruck, H. (2009) · Language and Speech · doi:10.1177/0023830909103166 · Cited by 88
Audiovisual integration of speech is disturbed in schizophrenia: An fMRI study
Szycik, G. R., Münte, T. F., Dillo, W., Mohammadi, B., Samii, A., Emrich, H. M., & Dietrich, D. E. (2009) · Schizophrenia Research · doi:10.1016/j.schres.2009.03.003 · Cited by 88
“ments facilitates the recognition of spoken words in noisy environments substantially (Erber, 1969; Grant and Seitz, 2000; Munhall et al., 2004; O'Neill, 1954; Sumby and Pollack, 1954).”
Saito, D. N., Yoshimura, K., Kochiyama, T., Okada, T., Honda, M., & Sadato, N. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi052 · Cited by 81
“As behavioral results have revealed cross-modal facilitation during lipreading (Sumby and Polack, 1954), the neural substrates of themechanisms underlying cross- modal binding can be evaluated based on the modulation of responses due to the congruency/incongruency of the stimuli.”
BackgroundAnimating speech: an automated approach using speech synthesised by rules
Hill, D. R., Pearce, A., & Wyvill, B. (1988) · The Visual Computer · doi:10.1007/bf01914863 · Cited by 81
The function of primate multimodal communication
Fröhlich, M., & van Schaik, C. (2018) · Animal Cognition · doi:10.1007/s10071-018-1197-8 · Cited by 60
Multi-sensory learning and learning to read
Blomert, L., & Froyen, D. (2010) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2010.06.025 · Cited by 96
Neural network models of sensory integration for improved vowel recognition
Yuhas, B. P., Goldstein, M. H., B, T. J. S. A., & Jenkins, R. E. (1990) · Proceedings of the IEEE · doi:10.1109/5.58349 · Cited by 87
The past, present, and future of speech processing
Juang, B.-H., & Chen, T. (1998) · IEEE Signal Processing Magazine · doi:10.1109/79.671130 · Cited by 81
Automatic Speechreading with Applications to Human-Computer Interfaces
Zhang, X., Broun, C. C., Mersereau, R. M., & Clements, M. A. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206137 · Cited by 82
Blind Audiovisual Source Separation Based on Sparse Redundant Representations
Casanovas, A. L., Monaci, G., Vandergheynst, P., & Gribonval, R. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2050650 · Cited by 79
“This is particularly evident if we think about speech: speaker lips movements are correlated with the produced sound and the listener can exploit this correspondence to better understand speech, especially in adverse environments [1], [2].”
BackgroundDeterminants of Responsiveness in Dyadic Interaction
Davis, D. (1982) · Springer eBooks · doi:10.1007/978-1-4613-9469-3_4 · Cited by 98
Enhancement of response times to bi- and tri-modal sensory stimuli during active movements
Hecht, D., Reiner, M., & Karni, A. (2008) · Experimental Brain Research · doi:10.1007/s00221-007-1191-x · Cited by 76
“signal-to-noise ratio by up to 15 decibels (Sumby and Pollack 1954).”
BackgroundSeymour, R., Stewart, D., & Ming, J. (2008) · EURASIP Journal on Image and Video Processing · doi:10.1155/2008/810362 · Cited by 78
“It has long been known that humans use available visual information when trying to understand speech, especially in noisy conditions [1].”
BackgroundAuditory and Visual Lexical Neighborhoods in Audiovisual Speech Perception
Tye‐Murray, N., Sommers, M. S., & Spehar, B. (2007) · Trends in Amplification · doi:10.1177/1084713807307409 · Cited by 78
“On the other hand, cues about place of articulation are conveyed by the visual signal but not very well by a degraded Most people, whether they have normal or impaired hearing, can recognize speech better when they can both see and hear the talker than when listening alone.(1,2) Often, the advantage of supplementing l…”
BackgroundSpeech perception and production
Casserly, E. D., & Pisoni, D. B. (2010) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.63 · Cited by 100
Möttönen, R., Schürmann, M., & Sams, M. E. (2004) · Neuroscience Letters · doi:10.1016/j.neulet.2004.03.076 · Cited by 91
“During audiovisual (AV) speech perception, seeing a talker’s articulatory gestures improves recognition of the speech sounds especially in noisy conditions [17].”
BackgroundUniversality and language-specific experience in the perception of lexical tone and pitch
Burnham, D. K., Kasisopa, B., REID, A., Luksaneeyanawin, S., Lacerda, F. P., ATTINA, V., Rattanasone, N. X., Schwarz, I., & Webster, D. (2014) · Applied Psycholinguistics · doi:10.1017/s0142716414000496 · Cited by 81
“In a classic study, Sumby and Pollack (1954) demonstrated a 40%–80% augmentation of AO speech perception when speech in a noisy environment is accompanied by the speaker’s face.”
BackgroundBisensory presentation of information.
Loveless, N. E., Brebner, J., & Hamilton, P. (1970) · Psychological Bulletin · doi:10.1037/h0028681 · Cited by 77
Hertrich, I., Dietrich, S., & Ackermann, H. (2010) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21421 · Cited by 81
“Speakers, both during formal and colloquial conversation, usuallytalktoeachotherfacetoface.Asaconsequence,visual information arising from articulatory movements, especially lip gestures, complements the acoustic speech signal and may have a significant impact upon spoken languagecomprehension.Forexample,“lipreading”ca…”
BackgroundAudiovisual Speech Source Separation: An overview of key methodologies
Rivet, B., Wang, W., Naqvi, S. M., & Chambers, J. A. (2014) · IEEE Signal Processing Magazine · doi:10.1109/msp.2013.2296173 · Cited by 77
The organization and reorganization of audiovisual speech perception in the first year of life
Danielson, D. K., Bruderer, A. G., Kandhadai, P., Vatikiotis‐Bateson, E., & Werker, J. F. (2017) · Cognitive Development · doi:10.1016/j.cogdev.2017.02.004 · Cited by 84
Reduced efficiency of audiovisual integration for nonnative speech
Yi, H., Phelps, J. E. B., Smiljanić, R., & Chandrasekaran, B. (2013) · The Journal of the Acoustical Society of America · doi:10.1121/1.4822320 · Cited by 75
Assessing the role of attention in the audiovisual integration of speech
Navarra, J., Alsius, A., Soto‐Faraco, S., & Spence, C. J. (2009) · Information Fusion · doi:10.1016/j.inffus.2009.04.001 · Cited by 86
“Seeing the vocal gestures of the speaker correlated with their auditory speech signal results in a significant improvement in speech comprehension under noisy conditions (e.g., see [50, 63 ]).”
BackgroundBinding and unbinding the auditory and visual streams in the McGurk effect
Nahorna, O., Berthommier, F., & Schwartz, J. (2012) · The Journal of the Acoustical Society of America · doi:10.1121/1.4728187 · Cited by 85
“Classical paradigms displaying audiovisual interaction mechanisms involve improvements in speech comprehension in noise due to lipreading (Sumby and Pollack, 1954), and the McGurk effect in which a conflicting visual input modifies the perception of an auditory input, e.g., visual /ga/ added on…”
BackgroundDeficient multisensory integration in schizophrenia: An event-related potential study
Stekelenburg, J. J., Maes, J. P., Gool, A. R. V., Sitskoorn, M. M., & Vroomen, J. (2013) · Schizophrenia Research · doi:10.1016/j.schres.2013.04.038 · Cited by 92
“conditions (Sumby and Pollack, 1954).”
Methods and Applications of the Audibility Index in Hearing Aid Selection and Fitting
Amlani, A. M., Punch, J. L., & Ching, T. Y. C. (2002) · Trends in Amplification · doi:10.1177/108471380200600302 · Cited by 75
Approaches to Understanding Multisensory Dysfunction in Autism Spectrum Disorder
Siemann, J. K., Veenstra‐VanderWeele, J., & Wallace, M. T. (2020) · Autism Research · doi:10.1002/aur.2375 · Cited by 74
“For example, human psychophysical studies have shown that the added visual information of seeing a speaker’s lip movements can facilitate speech intelligibility in a noisy environment [Sumby & Pollack, 1954].”
BackgroundSams, M. E., Manninen, P., Surakka, V., Helin, P., & Kättö, R. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00051-x · Cited by 89
Contributions of local speech encoding and functional connectivity to audio-visual speech perception
Giordano, B. L., Ince, R. A. A., Groß, J., Panzeri, S., Schyns, P. G., & Kayser, C. (2017) · eLife · doi:10.7554/elife.24763 · Cited by 76
“Movements of the lips, tongue or the eyes convey significant information that can boost speech intelligibility and facilitate the attentive tracking of individual speakers (Ross et al., 2007; Sumby and Pollack, 1954).”
BackgroundResultInfluentialChoi, D., Bruderer, A. G., & Werker, J. F. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01601-0 · Cited by 81
Unmasking the Difficulty of Listening to Talkers With Masks: lessons from the COVID-19 pandemic
Giovanelli, E., Valzolgher, C., Gessa, E., Todeschini, M., & Pavani, F. (2021) · i-Perception · doi:10.1177/2041669521998393 · Cited by 80
“…about 60% to 70% of a face, hiding the lower part which is particularly useful for identifying emotional expressions (Blais et al., 2017; Carbon, 2020) and for supporting communication through lip reading (Rosenblum et al., 1996), especially in presence of background noise (Sumby & Pollack, 1954).”
BackgroundSeeing Music? What musicians need to know about vision
Schutz, M. (2008) · Empirical Musicology Review · doi:10.18061/1811/34098 · Cited by 102
“Seeing a vocalist’s lip movements can also be helpful in comprehending song lyrics, much as they boost the intelligibility and comprehension of speech (Abry, Cathiard, Robert-Ribes, & Schwartz, 1994; Massaro & Stork, 1998; Middleweerd & Plomp, 1987; Sumby & Pollack, 1954).”
BackgroundHearing facial identities: Brain correlates of face–voice integration in person identification
Schweinberger, S. R., Kloth, N., & Robertson, D. M. C. (2010) · Cortex · doi:10.1016/j.cortex.2010.11.011 · Cited by 76
Poke and pop: Tactile–visual synchrony increases visual saliency
der Burg, E. V., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2008) · Neuroscience Letters · doi:10.1016/j.neulet.2008.11.002 · Cited by 82
“speaker better when we observe his or her lip movements [ 27 ].”
Grant, K. W., & Seitz, P. F. (2000) · The Journal of the Acoustical Society of America · doi:10.1121/1.428280 · Cited by 72
A taxonomy of Multimodal Interaction in the Human Information Processing System
Schomaker, L. (1995) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 109
The development of multisensory processes
Wallace, M. T. (2004) · Cognitive Processing · doi:10.1007/s10339-004-0017-z · Cited by 84
“Specifically, the ability to see the lips of an individual who is speaking provides an effective 15 dB gain in the intelligibility of the spoken signal (Sumby and Pollack 1954).”
BackgroundAuditory cross-modal reorganization in cochlear implant users indicates audio-visual integration
Stropahl, M., & Debener, S. (2017) · NeuroImage Clinical · doi:10.1016/j.nicl.2017.09.001 · Cited by 90
“It is well known that visual information during audio-visual (AV) speech perception can substantially improve speech understanding (Campbell, 2008; Grant and Seitz, 2000; Remez, 2012; Ross et al., 2007; Sumby and Pollack, 1954).”
Park, H., Ince, R. A. A., Schyns, P. G., Thut, G., & Groß, J. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2006558 · Cited by 76
“Previous research indicates that the availability of visual speech information improves speech intelligibility under difficult listening conditions [1, 6, 37].”
BackgroundBuchan, J. N., Paré, M., & Munhall, K. G. (2008) · Brain Research · doi:10.1016/j.brainres.2008.06.083 · Cited by 76
“When the acoustic speech signal is degraded by the addition of noise, the presence of dynamic visual speech information improves intelligibility (Sumby and Pollack, 1954).”
Recalibration of phonetic categories by lipread speech versus lexical information.
van Linden, S., & Vroomen, J. (2007) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.33.6.1483 · Cited by 71
“Numerous studies have shown that when listeners are asked to categorize the ambiguous sound, they do indeed use lipread and lexical information (Ganong, 1980; Sumby & Pollack, 1954).”
BackgroundAlegría, J., Charlier, B. L., & Mattys, S. L. (1999) · The European Journal of Cognitive Psychology · doi:10.1080/095414499382255 · Cited by 79
“Until the mid-1970s, the function of lip-reading was believed to be limited almost exclusively to assisting speech perception in poor auditory conditions (Binnie, Montgomery, & Jackson, 1974; Erber, 1969, 1974; Sumby & Pollack, 1954).”
BackgroundAn Asynchronous Hidden Markov Model for Audio-Visual Speech Recognition
Bengio, S. (2002) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 75
Cognitive aging and hearing acuity: modeling spoken language comprehension
Wingfield, A., Amichetti, N. M., & Lash, A. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00684 · Cited by 80
Facial expressions of singers influence perceived pitch relations
Thompson, W. F., Russo, F., & Livingstone, S. R. (2010) · Psychonomic Bulletin & Review · doi:10.3758/pbr.17.3.317 · Cited by 82
“When visual and auditory speech information is reinforcing (as in normal speech), availability of the visual channel improves intelligibility (Middleweerd & Plomp, 1987; Sumby & Pollack, 1954).”
BackgroundSpatial frequency requirements for audiovisual speech perception
Munhall, K. G., Kroos, C., Jozan, G., & Vatikiotis‐Bateson, E. (2004) · Perception & Psychophysics · doi:10.3758/bf03194902 · Cited by 83
“In one of the first experimental demonstrations of this effect, Sumby and Pollack (1954) showed that perception of spoken words was enhanced across a range of acoustic signal-to-noise ratios when subjects looked at a talker seated a few feet away.”
BackgroundAn fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory Cortex
Okada, K., Venezia, J. H., Matchin, W., Saberi, K., & Hickok, G. S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068959 · Cited by 75
“Visual cues provide powerful information and aid audition when speech occurs in noisy environments [1,2].”
BackgroundEncoding and memory for visible speech and gestures: A comparison between young and older adults.
Thompson, L. A. (1995) · Psychology and Aging · doi:10.1037//0882-7974.10.2.215 · Cited by 70
Discriminating languages by speech-reading
Soto‐Faraco, S., Navarra, J., Weikum, W., Vouloumanos, A., Sebastián‐Gallés, N., & Werker, J. F. (2007) · Perception & Psychophysics · doi:10.3758/bf03193744 · Cited by 74
“For example, many studies show that correlated visual speech information can greatly enhance the comprehension of spoken messages in noisy conditions (e.g., Sumby & Pollack, 1954 ), if the message is in a second language (Navarra & Soto-Faraco, 2007; Reisberg, McLean, & Goldfield, 1987), or if it is conceptually diffi…”
BackgroundCrossmodal Source Identification in Speech Perception
Lachs, L., & Pisoni, D. B. (2004) · Ecological Psychology · doi:10.1207/s15326969eco1603_1 · Cited by 76
“In their pioneering study, Sumby and Pollack (1954) found that the addition of visual information about articulation to an auditory signal can improve speech intelligibility performance in noise; these gains were equal to a +15-dB gain in signal-to-noise (S/N) ratio under auditory-alone conditions (MacLeod & Summerfie…”
BackgroundInfluentialTremblay, P., & Small, S. L. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.05.067 · Cited by 70
“Indeed, it has been shown that adding visual information enhances speech recognition (e.g., Reisberg et al., 1987; Sumby and Pollack, 1954), which could be associated with decreased reliance upon motor information during speech perception.”
Visual model structures and synchrony constraints for audio-visual speech recognition
Hazen, T. J. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tsa.2005.857572 · Cited by 77
“The improvements from adding the visual modality are often more pronounced in noisy conditions where the audio signal-to-noise ratio (SNR) is reduced [5], [6].”
BackgroundOkada, K., & Hickok, G. S. (2009) · Neuroscience Letters · doi:10.1016/j.neulet.2009.01.060 · Cited by 75
“Visual cues provide linguistic information beyond analysis of facial expressions and can facilitate speech perception when an auditory signal occurs in a noisy environment or is degraded by noise (Callan, Jones, Munhall, Callan, Kroos & Vatikiotis-Bateson, 2003; Dodd, 1977; Sumby & Pollack, 1954).”
BackgroundVisual Speech in the Head: The Effect of Cued-Speech on Rhyming, Remembering, and Spelling
Leybaert, J., & Charlier, B. (1996) · The Journal of Deaf Studies and Deaf Education · doi:10.1093/oxfordjournals.deafed.a014299 · Cited by 74
“…the mid—70s, lipreading was considered as improving speech perception in the case of a degraded acoustic signal, either because of an unfavorable signal-to-noise ratio or because of impairment of the sensory acoustic channel (Binnie, Montgomery, & Jackson, 1974; Erber, 1969; Sumby & Pollack, 1954).”
BackgroundThe relationship between multisensory integration and IQ in children.
Barutchu, A., Crewther, S. G., Fifer, J. M., Shivdasani, M. N., Innes-Brown, H., Toohey, S., Danaher, J., & Paolini, A. G. (2010) · Developmental Psychology · doi:10.1037/a0021903 · Cited by 71
Impact of face masks on voice radiation
Pörschmann, C., Lübeck, T., & Arend, J. M. (2020) · The Journal of the Acoustical Society of America · doi:10.1121/10.0002853 · Cited by 69
“Furthermore, other effects (e.g., lip reading) also strongly affect speech intelligibility (Matthews et al., 2002; Sumby and Pollack, 1954; Summerfield, 1992).”
BackgroundAgeing and multisensory integration: A review of the evidence, and a computational perspective
Jones, S. A., & Noppeney, U. (2021) · Cortex · doi:10.1016/j.cortex.2021.02.001 · Cited by 80
Multisensory integration of speech signals: the relationship between space and time
Jones, J. A., & Jarick, M. (2006) · Experimental Brain Research · doi:10.1007/s00221-006-0634-0 · Cited by 80
“speaker’s face (Sumby and Pollack 1954; Schwartz et al. 2004).”
Swerts, M. G. J., & Krahmer, E. (2010) · Journal of Phonetics · doi:10.1016/j.wocn.2009.10.002 · Cited by 90
“Such visual information is especially useful in noisy conditions (Sumby & Pollack, 1954), or when a speaker is addressing someone who is relatively distant in space (Jordan & Sergeant, 2000).”
BackgroundLip-Reading Driven Deep Learning Approach for Speech Enhancement
Adeel, A., Gogate, M., Hussain, A., & Whitmer, W. M. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2917039 · Cited by 85
“, lips, teeth, tongue) and speech reception has been previously shown in numerous behavioural studies [8]–[11].”
BackgroundPhonological coding by the prelinguistically deaf
Dodd, B. J., & Hermelin, B. (1977) · Perception & Psychophysics · doi:10.3758/bf03199495 · Cited by 69
Myers, B. R., Lense, M., & Gordon, R. L. (2019) · Brain Sciences · doi:10.3390/brainsci9030070 · Cited by 77
“also has marked benefits for neural tracking when presented in noisy conditions [98] (also [99,100]).”
BackgroundMégevand, P., Mercier, M. R., Groppe, D. M., Golumbic, E. Z., Mesgarani, N., Beauchamp, M. S., Schroeder, C. E., & Mehta, A. D. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.0555-20.2020 · Cited by 67
“Viewing one’s interlocutor significantly improves intelligibility under noisy conditions (Sumby and Pollack, 1954).”
BackgroundAudiovisual perception of openness and lip rounding in front vowels
Traunmüller, H., & Öhrström, N. (2006) · Journal of Phonetics · doi:10.1016/j.wocn.2006.03.002 · Cited by 81
“Sumby and Pollack (1954) investigated how the visibility of a speaker’s face influenced the intelligibility of English words produced by the speaker as a function of the auditory signal-to-noise (S/N) ratio.”
BackgroundResultAudiovisual temporal recalibration occurs independently at two different time scales
der Burg, E. V., Alais, D., & Cass, J. (2015) · Scientific Reports · doi:10.1038/srep14526 · Cited by 71
Enlarged temporal integration window in schizophrenia indicated by the double-flash illusion
Haß, K., Sinke, C., Reese, T., Roy, M., Wiswede, D., Dillo, W., Oranje, B., & Szycik, G. R. (2017) · Cognitive Neuropsychiatry · doi:10.1080/13546805.2017.1287693 · Cited by 80
“For example, in speech perception it is already known that presentation of speech signals in multisensory contexts (i.e. audio-visual) increases intelligibility especially within a noisy environment (Bishop & Miller, 2009; Grant & Seitz, 2000; Sumby & Pollack, 1954).”
BackgroundNeuroimaging of multisensory processing in vision, audition, touch, and olfaction
Thesen, T., Vibell, J. F., Calvert, G. A., & �sterbauer, R. A. (2004) · Cognitive Processing · doi:10.1007/s10339-004-0012-4 · Cited by 84
“…conditions, being able to see the talker's mouth movements can significantly enhance speech comprehension to the equivalent of changing the acoustic signal-to-noise ratio by 11 dB (i.e. increasing the physical intensity by a factor of four (Sumby and Pollack 1954; MacLeod and Summerfield 1987).”
BackgroundAuditory Perceptual Learning for Speech Perception Can be Enhanced by Audiovisual Training
Bernstein, L. E., Auer, E. T., Eberhardt, S. P., & Jiang, J. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00034 · Cited by 82
“Also, given that perception of AV speech stimuli is frequently faster and more reliable (Sumby and Pollack, 1954; Van Wassenhove et al., 2005; Ross et al., 2007), we surmise that in Experiment 3 the perceived effort to learn the paired associations was lower under AV versus AO conditions.”
BackgroundResultInfluentialStevenson, R. A., Nelms, C. E., Baum, S. H., Zurkovsky, L., Barense, M. D., NEWHOUSE, P. A., & Wallace, M. T. (2014) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2014.08.003 · Cited by 78
“Visual cues are known to significantly impact speech perception; when one can both hear a speaker’s utterance and concurrently see the articulation of that utterance (lip reading), speech comprehension is more accurate (Ross et al., 2007a, 2011; Sommers et al., 2005; Stevenson and James, 2009; Sumby and Pollack, 1954)”
BackgroundInfluentialAudio‐visual speech perception: a developmental ERP investigation
Knowland, V. C. P., Mercure, E., Karmiloff‐Smith, A., Dick, F., & Thomas, M. S. C. (2013) · Developmental Science · doi:10.1111/desc.12098 · Cited by 79
“…of particular benefit to adult listeners under conditions of auditory noise, when their availability can result in improvements in response accuracy equivalent to as much as a 15 dB increase in the auditory signal-to-noise ratio (Grant & Greenberg, 2001; Grant & Seitz, 2000; Sumby & Pollack, 1954).”
Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review
Opoku-Baah, C., Schoenhaut, A. M., Vassall, S. G., Tovar, D. A., Ramachandran, R., & Wallace, M. T. (2021) · Journal of the Association for Research in Otolaryngology · doi:10.1007/s10162-021-00789-0 · Cited by 94
“…studies on the effect of different levels of noise on the magnitude of visually facilitated speech comprehension and intelligibility reported a monotonic relationship where greater multisensory gains were achieved under very low signal-tonoise conditions (Erber 1969; Sumby and Pollack 1954).”
BackgroundInfluentialAttention to visual speech gestures enhances hemodynamic activity in the left planum temporale
Pekkola, J., Ojanen, V., Autti, T. H., Jääskeläinen, I. P., Möttönen, R., & Sams, M. E. (2005) · Human Brain Mapping · doi:10.1002/hbm.20190 · Cited by 69
“Key words: speech perception; attention; audiovisual; crossmodal; auditory cortex; planum temporale”
BackgroundCognitive Spare Capacity and Speech Communication: A Narrative Overview
Rudner, M., & Lunner, T. (2014) · BioMed Research International · doi:10.1155/2014/869726 · Cited by 65
“In everyday communication, seeing the face of your communication partner can enhance speech perception by several dB [92].”
BackgroundMultisensory Integration in Speech Processing: Neural Mechanisms of Cross-Modal Aftereffects
Kilian-Hütten, N., Formisano, E., & Vroomen, J. (2017) · Innovations in cognitive neuroscience · doi:10.1007/978-1-4939-7325-5_6 · Cited by 67
“Indeed, visual speech signals in the form of lip movements, head movements, and gestures exert a significant influence on the perception of the auditory signal (Ross et al. 2007; Sumby and Pollack 1954; von Kriegstein 2012).”
BackgroundInfluential
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.