Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 701–800 of 1,005
Bimodal audio–visual training enhances auditory adaptation process
Kawase, T., Sakamoto, S., Hori, Y., Maki, A., Suzuki, Y., & Kobayashi, T. (2009) · Neuroreport · doi:10.1097/wnr.0b013e32832fbef8 · Cited by 35
Maguinness, C., Setti, A., Burke, K. E., Kenny, R. A., & Newell, F. N. (2011) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2011.00019 · Cited by 36
Speechreading Skill and Visual Movement Sensitivity are Related in Deaf Speechreaders
Mohammed, T., Campbell, R., MacSweeney, M., Milne, E., Hansen, P., & Coleman, M. (2005) · Perception · doi:10.1068/p5211 · Cited by 40
“Having sight of the talker in a noisy environment can improve the intelligibility of the message by a value equivalent to an increase of 15 dB in signal-to-noise ratio (Sumby and Pollack 1954).”
BackgroundFor speech perception by humans or machines, three senses are better than one
Bernstein, L. E., & Benoı̂t, C. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607895 · Cited by 34
“Precedence belongs with researchers who showed that word identification accuracy under noisy acoustic conditions improves when the listener can also see the talker [7,8,9,10, also see 111.”
BackgroundEmbedding Conversational Agents into AR: Invisible or with a Realistic Human Body?
Reinhardt, J., Hillen, L., & Wolf, K. (2020) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/3374920.3374956 · Cited by 47
“Research in the field of audio-visual speech perception showed that the sight of the face of a talking person helps in understanding the spoken word [31], and seeing the speaker’s lips enables the listener to hear and thus to understand better [28].”
BackgroundVideo-Aided Model-Based Source Separation in Real Reverberant Rooms
Khan, M. S., Naqvi, S. M., Rehman, A. U., Wang, W., & Chambers, J. A. (2013) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2013.2261814 · Cited by 33
“Humans, however, perceive sound as a multi-modal process [3], [4] and therefore the authors in [5] proposed a multi-microphone audio-video source separation method showing improvement over audio-only and video-only schemes, but only in an over-determined setting.”
BackgroundKumar, G., Halder, T., Jaiswal, A. K., Mukherjee, A., Roy, D., & Banerjee, A. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01558 · Cited by 36
“During speech perception, visual cues from the speaker’s face enhances the intelligibility of auditory signal (Sumby, 1954; Helfer, 1997; Bulkin and Groh, 2006).”
BackgroundA General Audiovisual Temporal Processing Deficit in Adult Readers With Dyslexia
Francisco, A. A., Jesse, A., Groen, M. A., & McQueen, J. M. (2017) · Journal of Speech Language and Hearing Research · doi:10.1044/2016_jslhr-h-15-0375 · Cited by 36
Interface Oral Health Science 2014: Innovative Research on Biosis-Abiosis Intelligent Interface
Takahashi, N., Sasaki, K., & Takahashi, N. (2014) · Springer eBooks · doi:10.1007/978-4-431-55192-8 · Cited by 43
Thézé, R., Giraud, A., & Mégevand, P. (2020) · Science Advances · doi:10.1126/sciadv.abc6348 · Cited by 36
“Visual speech cues become particularly valuable for intelligibility when the auditory speech signal is less reliable, for instance, in noisy environments (1).”
BackgroundAller, M., Økland, H. S., MacGregor, L. J., Blank, H., & Davis, M. H. (2022) · Journal of Neuroscience · doi:10.1523/jneurosci.2476-21.2022 · Cited by 29
“Viewing the face of a conversation partner greatly improves speech comprehension especially under adverse listening conditions (Sumby and Pollack, 1954; Erber, 1975; Summerfield et al., 1992).”
BackgroundThe effect of seeing the interlocutor on auditory and visual speech production in noise
Fitzpatrick, M., Kim, J., & Davis, C. (2015) · Speech Communication · doi:10.1016/j.specom.2015.08.001 · Cited by 34
“[10]) it is likely that the availability of visual speech in FTF communication enables interlocutors to perceive each others‟ speech with relatively less need to employ effortful strategies such as increasing the amplitude of”
BackgroundAudiovisual Integration of Speech in a Bistable Illusion
Munhall, K. G., ten Hove, M., Brammer, M. J., & Paré, M. (2009) · Current Biology · doi:10.1016/j.cub.2009.03.019 · Cited by 35
“Visible speech enhances the intelligibility of auditory speech when listening conditions are poor [1], and can modify the perception of otherwise perfectly audible utterances [2].”
More than meets the eye: Context effects in word identification
Masson, M. E. J., & Borowsky, R. (1998) · Memory & Cognition · doi:10.3758/bf03201198 · Cited by 33
“... presented words are identified more quickly (see, e.g., Borowsky & Besner, 1991, 1993; Meyer, Schvaneveldt, & Ruddy, 1975; Neely, 1977) and accurately (see, e.g., Paul, Kellas, & Juola, 1992; Rhodes, Parkin, & Tremewan, 1993; Tulving & Gold, 1963) when presented in a relevant context, speech perception is improved…”
BackgroundThe information-divergence hypothesis of informational masking
Lutfi, R. A., Gilbertson, L. R., Heo, I., Chang, A.-C., & Stamas, J. (2013) · The Journal of the Acoustical Society of America · doi:10.1121/1.4817875 · Cited by 38
“In a cocktail party setting, for example, lip-reading may facilitate an individual’s ability to segregate one talker’s speech from another (Sumby and Pollack, 1954; Devergie et al., 2011), as well as visual cues signaling when and where to listen in a room (Kong and ShinnCunningham, 2009).”
BackgroundRules of Intersensory Integration in Spatial Scene Analysis and Speechreading
Colin, C., Radeau, M., Deltenre, P., & Morais, J. (2001) · Psychologica Belgica · doi:10.5334/pb.977 · Cited by 33
“Not only does lipreading improve intelligibility of speech embedded in noise (Sumby & Pollack, 1954) but it also affects perception in non-degraded listening conditions.”
BackgroundEnhancement of speech-in-noise comprehension through vibrotactile stimulation at the syllabic rate
Guilleminot, P. H., & Reichenbach, T. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2117000119 · Cited by 35
“Lip reading, for instance, refers to the human ability to integrate visual information from the moving lips with the auditory signal to aid speech comprehension (8, 9).”
BackgroundAudiovisual vowel monitoring and the word superiority effect in children
Fort, M., Spinelli, E., Savariaux, C., & Kandel, S. (2012) · International Journal of Behavioral Development · doi:10.1177/0165025412447752 · Cited by 38
“…word recognition, word superiority effect Several studies have shown that adults can rely on the information carried by the speaker’s oro-facial gestures to identify speech in noisy situations (Benoît, Mohamadi, & Kandel, 1994; Erber, 1969; Sumby & Pollack, 1954; see also Green, 1998 for a review).”
BackgroundProsody in the hands of the speaker
Guellaï, B., Langus, A., & Nespor, M. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00700 · Cited by 32
“ORIGINAL RESEARCH ARTICLE published: 07 July 2014 doi: 10.3389/fpsyg.2014.00700”
InfluentialA Novel Framework for Dynamic Utility-Based QoE Provisioning in Wireless Networks
Αριστομενόπουλος, Γ., Kastrinogiannis, T., Kaldanis, V., Karantonis, G., & Papavassiliou, S. (2010) · Globecom · doi:10.1109/glocom.2010.5683564 · Cited by 32
“For instance, in [5], it is revealed that if in addition to the oral speech visual factors are also utilized, humans can tolerate higher noise interference levels than if no visual factors were utilized.”
BackgroundThe temporal binding window for audiovisual speech: Children are like little adults
Hillock‐Dunn, A., Grantham, D. W., & Wallace, M. T. (2016) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2016.02.017 · Cited by 37
Shifts in Audiovisual Processing in Healthy Aging
Baum, S. H., & Stevenson, R. A. (2017) · Current Behavioral Neuroscience Reports · doi:10.1007/s40473-017-0124-7 · Cited by 37
“In fact, when comparing measures’ influence, working memory capacity (WMC) is one of the best predictors of performance on speech in noise tasks, second only to hearing loss itself [91]. Furthermore, there is an interaction of the impact of noise on speech perception and WMC such that individuals with greater WMC are…”
MethodInfluentialSchreitmüller, S., Frenken, M., Bentz, L., Ortmann, M., Walger, M., & Meister, H. (2017) · Ear and Hearing · doi:10.1097/aud.0000000000000502 · Cited by 32
About Face: Seeing the Talker Improves Spoken Word Recognition but Increases Listening Effort
Brown, V. A., & Strand, J. F. (2019) · Journal of Cognition · doi:10.5334/joc.89 · Cited by 37
Vision perceptually restores auditory spectral dynamics in speech
Plass, J., Brang, D., Suzuki, S., & Grabowecky, M. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2002887117 · Cited by 29
Investigating Communicative Feedback Phenomena across Languages and Modalities
Cerrato, L. (2007) · Cited by 34
Training of Procedural Tasks Through the Use of Virtual Reality and Direct Aids
Rodrguez, J., Gutirrez, T., Emilio, J., Casado, S., & Aguinag, I. (2012) · InTech eBooks · doi:10.5772/36650 · Cited by 34
“For example, under noisy conditions, observing the movements of the lips and gestures of the speaker can compensate for the lost speech perception which can be equivalent to increasing the auditory signal-to-noise ratio (Sumby & Pollack (1954)).”
BackgroundImpact of Visual Cues on Directional Benefit and Preference: Part I—Laboratory Tests
Wu, Y.-H., & Bentler, R. A. (2009) · Ear and Hearing · doi:10.1097/aud.0b013e3181bc767e · Cited by 32
“By using visual cues, the level of a listener’s speech recognition performance in noisy environments can increase substantially, ranging from 5 to 22 dB (Sumby & Pollack 1954; Erber 1969; MacLeod & Summerfield 1987; Hawkins et al. 1988).”
BackgroundMasking of speech in people with first-episode schizophrenia and people with chronic schizophrenia
Wu, C., Cao, S., Zhou, F., Wang, C., Wu, X. H., & Li, L. (2011) · Schizophrenia Research · doi:10.1016/j.schres.2011.09.019 · Cited by 33
“, 2011), and viewing a speaker's movements of the speech articulators (lipreading) (Sumby and Pollack, 1954; Summerfield, 1979; Rosenblum et al., 1996; Grant and Seitz, 2000; Rudmann et al., 2003; Helfer and Freyman, 2005).”
Kaganovich, N., Schumaker, J., Leonard, L. B., Gustafson, D., & Macias, D. (2014) · Journal of Speech Language and Hearing Research · doi:10.1044/2014_jslhr-l-13-0192 · Cited by 38
“It also significantly improves the understanding of speech-in-noise in both children and adults (Barutchu et al., 2010; Sumby & Pollack, 1954).”
BackgroundThe Effect of Visual Cues on Auditory Stream Segregation in Musicians and Non-Musicians
Marozeau, J., Innes-Brown, H., Grayden, D. B., Burkitt, A. N., & Blamey, P. J. (2010) · PLoS ONE · doi:10.1371/journal.pone.0011297 · Cited by 32
“When a speaker’s lip and facial movements are visible, an improvement in performance equivalent to increasing the signal-to-noise ratio by up to 15 dB has been observed [18].”
BackgroundLaird, K. L., Clements, P., Hunter, K. L., & Taylor, R. C. (2016) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-016-2160-9 · Cited by 32
“Instead, the visual cue has likely been integrated into the acoustic communication system as an additional mechanism to improve signal detection or localization, analogous to human lip reading in noisy environments (Sumby and Pollack 1954; Rowe 1999; Taylor and Ryan 2013).”
Hannah, B., Wang, Y., Jongman, A., Sereno, J. A., Cao, J., & Nie, Y. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.02051 · Cited by 43
“These modifications have been shown to enhance speech intelligibility (Sumby and Pollack, 1954; Gagné et al., 2002).”
BackgroundEffects of horizontal viewing angle on visual and audiovisual speech recognition.
Jordan, T. R., & Thomas, S. M. (2001) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.27.6.1386 · Cited by 32
“This aspect of our findings resonates with the findings of the classic study by Sumby and Pollack (1954) .”
BackgroundResultInfluentialShao, X., & Barker, J. (2007) · Speech Communication · doi:10.1016/j.specom.2007.11.002 · Cited by 36
“Sumby and Pollack [3] demonstrated that the visual speech signal could confer an increase in intelligibility equivalent to that produced by reducing the noise level by about 16 dB.”
BackgroundAn HMM-based speech-to-video synthesizer
Williams, J. J., & Katsaggelos, A. (2002) · IEEE Trans. Neural Networks · doi:10.1109/tnn.2002.1021891 · Cited by 33
Magnotti, J. F., Mallick, D. B., Feng, G. Y., Zhou, B., Zhou, W., & Beauchamp, M. S. (2016) · Experimental Brain Research · doi:10.1007/s00221-016-4634-4 · Cited by 32
Developmental factors and the non-native speaker effect in auditory-visual speech perception
Chen, Y., & Hazan, V. L. (2009) · The Journal of the Acoustical Society of America · doi:10.1121/1.3158823 · Cited by 31
Measurements of articulatory variation in expressive speech for a set of Swedish vowels
Nordstrand, M., Svanfeldt, G., Granström, B., & House, D. (2004) · Speech Communication · doi:10.1016/j.specom.2004.09.003 · Cited by 31
“The visual modality can complement the auditory information providing segmental cues on place of articulation (Sumby and Pollack, 1954; Erber, 1969), and prosodic information concerning prominence and phrasing (Granström et al., 1999; House et al., 2001).”
BackgroundCrawcour, S. C., Bowers, A. L., Harkrider, A. W., & Saltuklaroglu, T. (2009) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2009.05.001 · Cited by 36
A Danish open-set speech corpus for competing-speech studies
Nielsen, J. B., Dau, T., & Neher, T. (2014) · The Journal of the Acoustical Society of America · doi:10.1121/1.4835935 · Cited by 31
Audio-Visual and Meaningful Semantic Context Enhancements in Older and Younger Adults
Smayda, K. E., Engen, K. V. V., Maddox, W., & Chandrasekaran, B. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152773 · Cited by 29
Theta oscillations reflect conflict processing in the perception of the McGurk illusion
Fernández, L. M., Cuello, M. T., & Soto‐Faraco, S. (2017) · European Journal of Neuroscience · doi:10.1111/ejn.13804 · Cited by 35
McDaniel, J., Yoder, P. J., Woynaroski, T. G., & Watson, L. R. (2018) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-l-17-0101 · Cited by 42
“…2017; Tenenbaum et al., 2015; Young et al., 2009) and faster and more accurate speech processing and/or perception in the presence of visual cues for individuals without ASD (Davis & Kim, 2006; Gilley et al., 2010; Schwartz et al., 2004; Sumby & Pollack, 1954; van Wassenhove et al., 2005).”
Backgroundvan Laarhoven, T., Stekelenburg, J. J., & Vroomen, J. (2019) · Scientific Reports · doi:10.1038/s41598-019-46084-0 · Cited by 37
Development of the Mechanisms Underlying Audiovisual Speech Perception Benefit
Lalonde, K., & Werner, L. A. (2021) · Brain Sciences · doi:10.3390/brainsci11010049 · Cited by 35
“Numerous studies have demonstrated that adults rely on the multimodal nature of speech to compensate for noisy environments [24,25].”
BackgroundRoss, L. A., Molholm, S., Butler, J. S., Bene, V. A. D., & Foxe, J. J. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119598 · Cited by 38
Bernstein, L. E., Eberhardt, S. P., & Auer, E. T. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00934 · Cited by 36
Erdener, D., & Burnham, D. K. (2013) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2013.03.003 · Cited by 31
Smith, D. P., & Burnham, D. K. (2012) · The Journal of the Acoustical Society of America · doi:10.1121/1.3672703 · Cited by 31
“For normal-hearing listeners, visual speech improves speech perception in noise (Sumby and Pollack, 1954), or when listening to an unfamiliar dialect or foreignaccented speech (Arnold and Hill, 2001).”
BackgroundNew methods for describing and assessing direct speech communication under disturbing conditions
Lazarus, H. (1990) · Environment International · doi:10.1016/0160-4120(90)90006-r · Cited by 31
A comparison of the McGurk effect for spoken and sung syllables
Quinto, L. R., Thompson, W. F., Russo, F., & Trehub, S. E. (2010) · Attention Perception & Psychophysics · doi:10.3758/app.72.6.1450 · Cited by 38
“In speech, when auditory information is not clear, visual cues can provide additional information such as place of articulation (Neely, 1956) and lead to increased intelligibility of speech in noise (Sumby & Pollack, 1954).”
BackgroundMultisensory perceptual learning is dependent upon task difficulty
Niear, M. A. D., Koo, B., & Wallace, M. T. (2016) · Experimental Brain Research · doi:10.1007/s00221-016-4724-3 · Cited by 32
“Accordingly, the capacity to accurately utilize temporal cues to integrate sensory signals from different modalities is essential for developing perceptual coherence (Stein and Stanford 2008; Wallace and Stevenson 2014) and for driving the well-established behavioral benefits associated with multisensory integration (…”
Cross-modal integration of multimodal courtship signals in a wolf spider
Kozak, E. C., & Uetz, G. W. (2016) · Animal Cognition · doi:10.1007/s10071-016-1025-y · Cited by 39
“The innate nature of cross-modal binding of auditory and visual signals in human speech is often illustrated by lipreading in noisy environments (Sumby and Pollack 1954), the ‘‘McGurk effect’’ (McGurk and Macdonald 1976) created by combined visual and auditory input, and the ‘‘ventriloquism effect’’ (Hauser 1996), in…”
BackgroundAudio‐visual speech in noise perception in dyslexia
van Laarhoven, T., Keetels, M., Schakel, L., & Vroomen, J. (2016) · Developmental Science · doi:10.1111/desc.12504 · Cited by 41
“Observing a speaker’s articulatory movements can substantially enhance the ability to comprehend speech, especially in suboptimal listening conditions (MacLeod & Summerfield, 1987; Sumby & Pollack, 1954).”
Multisensory Integration: Frequency Tuning of Audio-Tactile Integration
Foxe, J. J. (2009) · Current Biology · doi:10.1016/j.cub.2009.03.029 · Cited by 31
Avatars in Assistive Homes for the Elderly
Morandell, M., Hochgatterer, A., Fagel, S., & Wassertheurer, S. (2008) · Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society · doi:10.1007/978-3-540-89350-9_27 · Cited by 47
“1. As also known from natural speech [ 14 ] synthetic visual speech increases the intelligibility when added to audio speech [15].”
BackgroundA novel behavioral paradigm to assess multisensory processing in mice
Siemann, J. K., Muller, C. L., Bamberger, G., Allison, J. D., Veenstra-VanderWeele, J., & Wallace, M. T. (2015) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00456 · Cited by 29
Timing in audiovisual speech perception: A mini review and new psychophysical data
Venezia, J. H., Thurman, S. M., Matchin, W., George, S. E., & Hickok, G. S. (2015) · Attention Perception & Psychophysics · doi:10.3758/s13414-015-1026-y · Cited by 34
“In fact, visual speech is quite informative as evidenced by large intelligibility gains in noise for audiovisual speech compared to auditory speech alone (Erber, 1969; MacLeod & Summerfield, 1987; Neely, 1956; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954).”
BackgroundHaider, C. L., Suess, N., Hauswald, A., Park, H., & Weisz, N. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119044 · Cited by 30
Visual information constrains early and late stages of spoken-word recognition in sentence context
Brunellière, A., Sánchez-García, C., Ikumi, N., & Soto‐Faraco, S. (2013) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2013.06.016 · Cited by 30
“…perception benefits from integrating cues across sensory modalities, especially when the processing of auditory information is difficult (such as in noisy contexts, Ma et al., 2009; Ross et al., 2007; Sumby & Pollack, 1954, or while perceiving non-native languages, Navarra & Soto-Faraco, 2007).”
BackgroundSlipped lips: onset asynchrony detection of auditory‐visual language in autism
Grossman, R. B., Schneps, M. H., & Tager‐Flusberg, H. (2009) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2008.02002.x · Cited by 30
“lipreading, with auditory speech input to enhance language comprehension ( Sumby & Pollack, 1954 ) and that lipreading improves recognition of speech in noisy environments (Calvert, Brammer, & Iversen, 1998; Massaro, 1984; Summerfield & McGrath, 1984).”
Visual face-movement sensitive cortex is relevant for auditory-only speech recognition
Riedel, P., Ragert, P., Schelinski, S., Kiebel, S. J., & von Kriegstein, K. (2014) · Cortex · doi:10.1016/j.cortex.2014.11.016 · Cited by 30
Johnels, J. Å., Gillberg, C., Falck‐Ytter, T., & Miniscalco, C. (2014) · Journal of Speech Language and Hearing Research · doi:10.1044/2014_jslhr-l-13-0268 · Cited by 35
Language identification from visual-only speech signals
Ronquest, R. E., Levi, S. V., & Pisoni, D. B. (2010) · Attention Perception & Psychophysics · doi:10.3758/app.72.6.1601 · Cited by 30
“In addition to the auditory properties of speech, the visual signal carries important information about the phonetic structure of the message that affects the perception and comprehension of the speech signal (see, e.g., Massaro, 1987; Sumby & Pollack, 1954; Summerfield, 1987).”
Using Visual Information to Train Foreign‐Language Vowel Production*
Flege, J. E. (1988) · Language Learning · doi:10.1111/j.1467-1770.1988.tb00417.x · Cited by 36
Comparison of human and machine-based lip-reading
Hilder, S., Harvey, R., & Theobald, B.-J. (2009) · Cited by 30
Talker variability in audio-visual speech perception
Heald, S. L. M., & Nusbaum, H. C. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00698 · Cited by 35
“For example, the intelligibility of speech in noise (Sumby and Pollack, 1954) as well as speech heard through cochlear implants (Goh et al., 2001; Lachs et al., 2001) is significantly improved by additionally seeing a speaker talk.”
BackgroundRhythmic Modulation of Entrained Auditory Oscillations by Visual Inputs
Simon, D. M., & Wallace, M. T. (2017) · Brain Topography · doi:10.1007/s10548-017-0560-4 · Cited by 34
“For speech signals in particular, the integration of visual mouth movements with the audible speech signal has been shown to confer significant improvements for intelligibility in challenging acoustic conditions (Sumby and Pollack 1954; Ross et al. 2007).”
BackgroundEye Movements During Visual Speech Perception in Deaf and Hearing Children
Worster, E., Pimperton, H., Ralph‐Lewis, A., Monroy, L., Hulme, C., & MacSweeney, M. (2017) · Language Learning · doi:10.1111/lang.12264 · Cited by 34
“It is well established that congruent visual speech can enhance speech perception (Lusk & Mitchel, 2016; Mitchel & Weiss, 2014; Sumby & Pollack, 1954), while incongruent visual information can disrupt it (McGurk & MacDonald, 1976).”
BackgroundThe impact of the Lombard effect on audio and visual speech recognition systems
Marxer, R., Barker, J., Alghamdi, N., & Maddock, S. (2018) · Speech Communication · doi:10.1016/j.specom.2018.04.006 · Cited by 37
“When communicating in noisy conditions, humans naturally exploit visual speech information concentrated in lip and jaw movements (Sumby and Pollack, 1954).”
BackgroundEffect of Video Frame Rate on Subjects’ Ability to Shadow One of Two Competing Verbal Passages
Vitkovitch, M., & BARBER, P. J. (1994) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3705.1204 · Cited by 30
A Comparative Analysis of the Audiovisual, Auditive and Visual Perception of Speech
Ewertsen, H. W., & Nielsen, H. B. (1971) · Acta Oto-Laryngologica · doi:10.3109/00016487109122473 · Cited by 30
Vigilance: A comparison in auditory, visual, and combined audio-visual tasks.
Baker, R. A., Ware, J. R., & Sipowicz, R. R. (1962) · Canadian Journal of Psychology/Revue Canadienne de Psychologie · doi:10.1037/h0083243 · Cited by 30
Multisensory processing of redundant information in go/no-go and choice responses
Blurton, S. P., Greenlee, M. W., & Gondan, M. (2014) · Attention Perception & Psychophysics · doi:10.3758/s13414-014-0644-0 · Cited by 36
“According to the same reasoning, Townsend and Altieri (2012) recently generalized the capacity index for more-complex tasks, such as the two-choice task used in Experiment 2.”
Cerebral Correlates and Statistical Criteria of Cross-Modal Face and Voice Integration
Pollick, F. E., Love, S., & Latinus, M. (2011) · Seeing and Perceiving · doi:10.1163/187847511x584452 · Cited by 35
Ehrhorn, A. M., Adlof, S. M., Fogerty, D., & Laing, S. (2020) · Scientific Studies of Reading · doi:10.1080/10888438.2020.1849223 · Cited by 38
“…shown that adults and children with TD demonstrate strong benefits of visual speech information during speech-related tasks (de Gelder & Vroomen, 1998; Leybaert et al., 2014; Megnin-Viggars & Goswami, 2013; Meronen et al., 2013; Norrix et al., 2007; Ramirez & Mann, 2005; Sumby & Pollack, 1954).”
BackgroundRethinking the McGurk effect as a perceptual illusion
Getz, L. M., & Toscano, J. C. (2021) · Attention Perception & Psychophysics · doi:10.3758/s13414-021-02265-6 · Cited by 34
“…aid spoken language comprehension by adults in both quiet (McGettigan et al., 2012; Sánchez-Garcı́a, Alsius, Enns, & Soto-Faraco, 2011)and for speech in background noise (Gilbert, Lansing, & Garnsey, 2012; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954; Summerfield, 1979).”
The Impact of Temporally Coherent Visual Cues on Speech Perception in Complex Auditory Environments
Yuan, Y., Lleo, Y., Daniel, R., White, A., & Oh, Y. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.678029 · Cited by 33
Left Motor δ Oscillations Reflect Asynchrony Detection in Multisensory Speech Perception
Biau, E., Schultz, B. G., Gunter, T. C., & Kotz, S. A. (2022) · Journal of Neuroscience · doi:10.1523/jneurosci.2965-20.2022 · Cited by 39
Turkish lip-reading using Bi-LSTM and deep learning models
Atila, Ü., & Sabaz, F. (2022) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2022.101206 · Cited by 41
Roger, É., de Almeida, L. R., Lœvenbruck, H., Perrone‐Bertolotti, M., Cousin, É., Schwartz, J., Perrier, P., Dohen, M., Vilain, A., Baraduc, P., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119672 · Cited by 34
Hearing loss and deafness in the pediatric population
Deltenre, P., & Maldergem, L. V. (2013) · Handbook of clinical neurology · doi:10.1016/b978-0-444-59565-2.00023-x · Cited by 49
“On the contrary: although clear speech conveyed in the auditory mode only can obviously be understood without difficulty, it is remarkable that in ecological conditions, especially in the presence of competitive noise, speech perception in normal face-to-face dialog is bimodal, obligatorily integrating auditory and vi…”
BackgroundIntegrating information from different senses in the auditory cortex
King, A. J., & Walker, K. M. M. (2012) · Biological Cybernetics · doi:10.1007/s00422-012-0502-x · Cited by 31
Audiovisual Integration in Children Listening to Spectrally Degraded Speech
Maidment, D. W., Kang, H., Stewart, H. J., & Amitay, S. (2014) · Journal of Speech Language and Hearing Research · doi:10.1044/2014_jslhr-s-14-0044 · Cited by 36
“When auditory and visual sourcesof verbal information are presented simultaneously, visual cues often influence the perception and comprehension of stimuli in the auditory modality (Sumby & Pollack, 1954).”
BackgroundSaunders, G. H., & Echt, K. V. (2012) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2010.08.0157 · Cited by 33
“For example, it has been long established that supplementing a degraded auditory signal with visual information results in considerable benefit for speech understanding [51–52], resulting in improvements in speech understanding of 50 percent over auditoryalone conditions [53].”
BackgroundNeural Mechanisms Underlying Cross-Modal Phonetic Encoding
Shahin, A. J., Backer, K. C., Rosenblum, L. D., & Kerlin, J. R. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.1566-17.2017 · Cited by 32
“Listeners often rely on visual cues (lip movements) to enhance speech comprehension in difficult listening environments (Sumby and Pollack, 1954).”
BackgroundAUTOMATIC SPEECHREADING OF IMPAIRED SPEECH
Potamianos, G., & Neti, C. (2001) · AVSP · Cited by 33
Multimodal Integration - A Biological View
Coen, M. H. (2001) · Cited by 29
Eye position and the control of auditory attention.
Reisberg, D., Scheiber, R., & Potemken, L. (1981) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.7.2.318 · Cited by 29
Petersen, E. B., MacDonald, E., & Sørensen, A. J. M. (2022) · Trends in Hearing · doi:10.1177/23312165221103340 · Cited by 42
“Considering that providing visual information improves the speech understanding significantly (Remez, 2012; Sumby & Pollack, 1954), it might have been expected that participants in the current study would speak at lower SNRs when seated face-to-face in order to reduce the talking effort.”
BackgroundKról, M. E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194491 · Cited by 36
Micheli, C., Schepers, I. M., Özker, M., Yoshor, D., Beauchamp, M. S., & Rieger, J. W. (2018) · European Journal of Neuroscience · doi:10.1111/ejn.13992 · Cited by 30
“…knowledge about the brain regions and neural responses that track concurrent features of the auditory and the visual speech signal will lead to a better understanding of how visual information improves speech perception (Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954).”
BackgroundAudiovisual speech perception and language acquisition in preterm infants: A longitudinal study
Imafuku, M., Kawai, M., Niwa, F., Shinya, Y., & Myowa, M. (2018) · Early Human Development · doi:10.1016/j.earlhumdev.2018.11.001 · Cited by 35
Schubotz, L., Holler, J., Drijvers, L., & Özyürek, A. (2020) · Psychological Research · doi:10.1007/s00426-020-01363-8 · Cited by 39
Tan, S., Kalashnikova, M., Liberto, G. M. D., Crosse, M. J., & Burnham, D. K. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119217 · Cited by 35
Perceptual Organization of Speech
Remez, R. E. (2008) · Blackwell Publishing Ltd eBooks · doi:10.1002/9780470757024.ch2 · Cited by 35
The neural basis of lip-reading capabilities is altered by early visual deprivation
Putzar, L., Goerendt, I. K., Heed, T., Richard, G. W., Büchel, C., & Röder, B. (2010) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2010.04.007 · Cited by 32
Naqvi, S. M., Wenwu Wang, B., Khan, M. S., Barnard, M., & Chambers, J. A. (2012) · IET Signal Processing · doi:10.1049/iet-spr.2011.0124 · Cited by 28
Talker familiarity and spoken word recognition in school-age children
Levi, S. V. (2014) · Journal of Child Language · doi:10.1017/s0305000914000506 · Cited by 35
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.