Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 801–900 of 1,005
Visual Influences on Interactive Speech Alignment
Dias, J. W., & Rosenblum, L. D. (2011) · Perception · doi:10.1068/p7071 · Cited by 33
“Visual information modulates audio speech perception (McGurk and MacDonald 1976) and provides an advantage of audiovisual information over audio alone (Sumby and Pollack 1954).”
BackgroundMultimodal interface research: a science without borders
Oviatt, S. L. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-464 · Cited by 28
“In the multimodal literature on speech and lip movements, similar robustness advantages have been documented when humans process dual input signals ([3], [12], [13], [28]), and more recently such robustness advantages also have been demonstrated for multimodal systems ([1], [29]).”
BackgroundAudiovisual speech recalibration in children*
van Linden, S., & Vroomen, J. (2008) · Journal of Child Language · doi:10.1017/s0305000908008817 · Cited by 28
“When the auditory signal is degraded or embedded in noise, seeing the speaker significantly enhances speech understanding (Sumby & Pollack, 1954).”
BackgroundContributions of Semantic and Facial Information to Perception of Nonsibilant Fricatives
Jongman, A., Wang, Y., & Kim, B. H. (2003) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2003/106 · Cited by 32
Popliteal artery injuries associated with fractures and dislocations about the knee.
Subaşı, M., Cakir, O., Kesemenlı, C. C., Arslan, H., Necmioğlu, S., & Eren, N. (2001) · PubMed · Cited by 34
Lip contour segmentation and tracking compliant with lip-reading application constraints
Stillittano, S., Girondel, V., & Caplier, A. (2012) · Machine Vision and Applications · doi:10.1007/s00138-012-0445-1 · Cited by 33
“It is well known that visual information can significantly increase speech comprehension under noisy conditions [1,2].”
BackgroundHearing Speech in Noise: Seeing a Loud Talker is Better
Kim, J., Sironic, A., & Davis, C. (2011) · Perception · doi:10.1068/p6941 · Cited by 27
A Network Model of Observation and Imitation of Speech
Mashal, N., Solodkin, A., Dick, A. S., Chen, E. E., & Small, S. L. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00084 · Cited by 30
“Observing these motor actions improves speech perception, particularly under noisy conditions (MacLeod and Summerfield, 1987) or when the auditory signal is degraded (Sumby and Pollack, 1954; Ross et al., 2007).”
BackgroundGrange, J. A., & Culling, J. F. (2016) · The Journal of the Acoustical Society of America · doi:10.1121/1.4968515 · Cited by 28
“None of the studies referred to above tested for audio-visual presentations, despite NH listeners’ reliance on lip-reading (Sumby and Pollack, 1954; Summerfield, 1987, 1992) and CI users’ higher reliance on lip-reading (Hay-McCutcheon et al., 2005; Rouger et al., 2007; Schorr et al., 2005;…”
BackgroundImpact of Bilingualism on Infants’ Ability to Learn From Talking and Nontalking Faces
Fort, M., Ayneto, A., Escrichs, A., & Sebastián‐Gallés, N. (2017) · Language Learning · doi:10.1111/lang.12273 · Cited by 32
Speech intelligibility in a realistic virtual sound environment
Mansour, N., Marschall, M., May, T., Westermann, A., & Dau, T. (2021) · The Journal of the Acoustical Society of America · doi:10.1121/10.0004779 · Cited by 30
“In addition, a non-adaptive procedure, which presented the sequence of 20 sentences at a constant SNR, was implemented to estimate speech reception scores in % correct, testing the second hypothesis that SRSs at real-world SNRs reflect difficulties with speech intelligibility.”
InfluentialModulation of Brain Activity by Selective Attention to Audiovisual Dialogues
Leminen, A., Verwoert, M., Moisala, M., Salmela, V. R., Wikman, P. A., & Alho, K. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00436 · Cited by 30
Monkeys and humans implement causal inference to simultaneously localize auditory and visual stimuli
Mohl, J. T., Pearson, J. M., & Groh, J. M. (2020) · Journal of Neurophysiology · doi:10.1152/jn.00046.2020 · Cited by 31
Cope, T., Sohoglu, E., Peterson, K. A., Jones, P. S., Rua, C., Passamonti, L., Sedley, W., Post, B., Coebergh, J. W. W., Butler, C., et al. (2023) · Cell Reports · doi:10.1016/j.celrep.2023.112422 · Cited by 33
“Nonetheless, it is clear that humans use cross-modal cues such as lip reading and subtitles to improve speech perception, especially in noisy listening environments.(6) Such cross-modal cuing typically improves comprehension but can lead to false perception when cues mismatch.”
BackgroundVisual speech discrimination and identification of natural and synthetic consonant stimuli
Files, B. T., Tjan, B. S., Jiang, J., & Bernstein, L. E. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00878 · Cited by 27
“Frontiers in Psychology | www.frontiersin.org 2 July 2015 | Volume 6 | Article 878 environments or with hearing loss (Sumby and Pollack, 1954; MacLeod and Summerfield, 1987; Ross et al., 2007; Grant et al., 2013), and subtle differences in individual speech tokens control the type of audiovisual…”
BackgroundAudiovisual cues benefit recognition of accented speech in noise but not perceptual adaptation
Banks, B., Gowen, E., Munro, K. J., & Adank, P. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00422 · Cited by 28
Sequential audiovisual interactions during speech perception: A whole-head MEG study
Hertrich, I., Mathiak, K., Lutzenberger, W., Menning, H., & Ackermann, H. (2006) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.09.019 · Cited by 27
Vatakis, A., Maragos, P. A., Rodomagoulakis, I., & Spence, C. J. (2012) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2012.00071 · Cited by 27
“…of consonants and vowels on audiovisual temporal perception Argiro Vatakis1*, Petros Maragos2, Isidoros Rodomagoulakis2 and Charles Spence3 1 Cognitive Systems Research Institute, Athens, Greece 2 Computer Vision, Speech Communication and Signal Processing Group, National Technical…”
BackgroundInfluentialThe effect of simulated visual impairment on speech‐reading ability
Dickinson, C., & Taylor, J. J. (2011) · Ophthalmic and Physiological Optics · doi:10.1111/j.1475-1313.2010.00810.x · Cited by 27
Cross-speaker generalisation in two phoneme-level perceptual adaptation processes
van der Zande, P., Jesse, A., & Cutler, A. (2014) · Journal of Phonetics · doi:10.1016/j.wocn.2014.01.003 · Cited by 27
Published estimates of group differences in multisensory integration are inflated
Magnotti, J. F., & Beauchamp, M. S. (2018) · PLoS ONE · doi:10.1371/journal.pone.0202908 · Cited by 32
“the talker’s face enhances speech recognition accuracy(1-3).”
Recalibration of Phonetic Categories by Lipread Speech: Measuring Aftereffects After a 24-hour Delay
Vroomen, J., & Baart, M. (2009) · Language and Speech · doi:10.1177/0023830909103178 · Cited by 27
“Numerous studies have shown that when listeners are asked to categorize the ambiguous sound, they do indeed use lipread and lexical information (Ganong, 1980; Sumby & Pollack, 1954).”
BackgroundCross-modal facilitation in speech prosody
Foxton, J. M., Rivière, L.-D., & Barone, P. (2009) · Cognition · doi:10.1016/j.cognition.2009.11.009 · Cited by 29
“In the case of speech, a specific multimodal phenomenon (Bernstein, Auer, & Moore, 2004), the interactions between visual and auditory cues can affect speech comprehension, especially in noisy environments (Ross et al., 2007; Schwartz, Berthommier, & Savariaux, 2004; Sumby & Pollack, 1954).”
The development of audiovisual speech perception
Soto‐Faraco, S., Calabresi, M., Navarra, J., Werker, J. F., & Lewkowicz, D. J. (2012) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199586059.003.0009 · Cited by 39
Discrimination of auditory-visual synchrony.
Grant, K. W., van Wassenhove, V., & Poeppel, D. (2003) · AVSP · Cited by 27
Liu, B., Lin, Y., Gao, X., & Dang, J. (2013) · Neuroscience · doi:10.1016/j.neuroscience.2013.05.007 · Cited by 26
“, 2012), particularly in noise environments (Sumby and Pollack, 1954; MacLeod and Summerfield, 1987).”
Audio-tactile enhancement of cortical speech-envelope tracking
Riecke, L., Snipes, S., van Bree, S., Kaas, A. L., & Hausfeld, L. G. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116134 · Cited by 28
Infants and Adults Use Visual Cues to Improve Detection and Discrimination of Speech in Noise
Lalonde, K., & Werner, L. A. (2019) · Journal of Speech Language and Hearing Research · doi:10.1044/2019_jslhr-h-19-0106 · Cited by 32
“Adults can detect, discriminate, and recognize speech with greater accuracy and at less favorable signal-to-noise ratios (SNRs) in audiovisual (AV) conditions than in auditory-only conditions (e.g., Grant & Seitz, 2000; Lalonde & Holt, 2016; Sumby & Pollack, 1954).”
BackgroundHighly proficient L2 speakers still need to attend to a talker’s mouth when processing L2 speech
Birulés, J., Bosch, L., Pons, F., & Lewkowicz, D. J. (2020) · Language Cognition and Neuroscience · doi:10.1080/23273798.2020.1762905 · Cited by 32
“…studies showing that speech comprehension is enhanced by concurrent visual speech cues when auditory speech is presented in noise (Cotton, 1935; Sumby & Pollack, 1954), when auditory speech is filtered (Sanders & Goodrich, 1971), or when auditory speech is presented in competition with other…”
BackgroundListening in complex acoustic scenes
King, A. J., & Walker, K. M. M. (2020) · Current Opinion in Physiology · doi:10.1016/j.cophys.2020.09.001 · Cited by 33
“These neural processes may account for why face reading helps human listeners to selectively attend to a single speaker in multi-talker listening situations [56].”
BackgroundHanulíková, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0259230 · Cited by 39
Similarity structure in visual speech perception and optical phonetic signals
Jiang, J., Auer, E. T., Alwan, A. A., Keating, P., & Bernstein, L. E. (2007) · Perception & Psychophysics · doi:10.3758/bf03193945 · Cited by 26
“Being able to see a talker produces substantial gains to comprehending acoustic speech in noise (MacLeod & Summerfield, 1987; Sumby & Pollack, 1954 ), improvements in comprehending difficult messages even under good listening conditions (Arnold & Hill, 2001; Reisberg, McLean, & Goldfield, 1987), speech detection under…”
BackgroundKawase, T., Yamaguchi, K., Ogawa, T., Suzuki, K.-I., Suzuki, M., Itoh, M., Kobayashi, T., & Fujii, T. (2005) · Neuroscience Letters · doi:10.1016/j.neulet.2005.03.050 · Cited by 28
Cross-Modal Prediction in Speech Perception
Sánchez-García, C., Alsius, A., Enns, J. T., & Soto‐Faraco, S. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025198 · Cited by 28
“trained lip-readers, at the levels of syntax and semantics [4], [36].”
The effects of delayed auditory and visual feedback on speech production
Chesters, J., Baghai-Ravary, L., & Möttönen, R. (2015) · The Journal of the Acoustical Society of America · doi:10.1121/1.4906266 · Cited by 36
“Visual cues can enhance auditory speech perception, especially in noise (Sumby and Pollack, 1954) or modify an auditory percept, as in the McGurk illusion (McGurk and MacDonald, 1976).”
Event-related potentials for better speech perception in noise by cochlear implant users
Soshi, T., Hisanaga, S., Kodama, N., Kanekama, Y., Samejima, Y., Yumoto, E., & Sekiyama, K. (2014) · Hearing Research · doi:10.1016/j.heares.2014.08.001 · Cited by 27
“…be due to a kind of floor effect: that is, AO speech perception in noise is generally difficult for CI users (Nelson et al., 2003; Fu and Nogaki, 2005), while slight differences in masking between the two noises in the AO condition might be enhanced in the AV condition (Sumby and Pollack, 1954).”
BackgroundResultInfluentialThe role of visual speech information in supporting perceptual learning of degraded speech.
Wayne, R. V., & Johnsrude, I. S. (2012) · Journal of Experimental Psychology Applied · doi:10.1037/a0031042 · Cited by 31
Interactions between space and effectiveness in human multisensory performance
Nidiffer, A. R., Stevenson, R. A., Fister, J. K., Barnett, Z. P., & Wallace, M. T. (2016) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2016.01.031 · Cited by 31
“, 2004), effectiveness (Stevenson et al., 2012b; Sumby and Pollack, 1954) but see (Chandrasekaran et al.”
BackgroundLaurent, R., Barnaud, M.-L., Schwartz, J., Bessìère, P., & Diard, J. (2017) · Psychological Review · doi:10.1037/rev0000069 · Cited by 33
Toward a Model of Auditory-Visual Speech Intelligibility
Grant, K. W., & Bernstein, J. G. W. (2019) · Springer handbook of auditory research · doi:10.1007/978-3-030-10461-0_3 · Cited by 26
A low-latency lip-synchronized videoconferencing system
Chen, M. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642692 · Cited by 26
“A Low-Latency Lip-Synchronized Videoconferencing System Milton Chen Interactivity Laboratory and Computer Graphics Laboratory Stanford University miltchen@graphics.stanford.edu”
MethodSchmid, G., & Ziegler, W. (2005) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2005.07.002 · Cited by 26
“E-mail address: Gabi.Schmid@extern.lrz-muenchen.de (G. Schmid). environments (Sumby & Pollack, 1954).”
The role of multisensory interplay in enabling temporal expectations
Ball, F., Michels, L. E., Thiele, C., & Noesselt, T. (2017) · Cognition · doi:10.1016/j.cognition.2017.09.015 · Cited by 26
The development of multisensory processes for perceiving the environment and the self
Lewkowicz, D. J., & Bremner, A. J. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-812492-5.00004-8 · Cited by 32
Baart, M., & Samuel, A. G. (2015) · Journal of Memory and Language · doi:10.1016/j.jml.2015.06.008 · Cited by 26
“Two substantial literatures exist in this domain: One body of research has shown that perceivers rely on the visible articulatory gestures of the speaker, here referred to as lip-reading (e.g., Sumby & Pollack, 1954).”
BackgroundÖzker, M., Yoshor, D., & Beauchamp, M. S. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00141 · Cited by 28
“Visual speech information can compensate for noisy auditory speech (Bernstein et al 2004, Ross et al 2007, Sumby & Pollack 1954), but this requires recruitment of multisensory brain regions capable of combining the auditory and visual speech information to restore intelligibility.”
BackgroundSekiyama, K., Hisanaga, S., & Mugitani, R. (2021) · Cortex · doi:10.1016/j.cortex.2021.03.023 · Cited by 28
Examining visible articulatory features in clear and plain speech
Tang, L. Y. W., Hannah, B., Jongman, A., Sereno, J. A., Wang, Y., & Hamarneh, G. (2015) · Speech Communication · doi:10.1016/j.specom.2015.09.008 · Cited by 25
“This is presumably due to the additional stream of linguistic information available to the perceiver in the visible articulatory movements of the speaker’s lips, teeth, and tongue as useful sources for segmental perception (Kim and Davis, 2014b; Tasko and Greilick, 2010; Traunmüller and…”
BackgroundMethodInfluentialTreille, A., Cordeboeuf, C., Vilain, C., & Sato, M. (2014) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2014.02.004 · Cited by 30
“Visual speech information is known to effectively improve speech intelligibility in noise (Benoît, Mohamadi, & Kandel, 1994; Sumby & Pollack, 1954), the understanding of a semantically complex acoustic statement (Reisberg, McLean, & Goldfield, 1987) or a foreign language (Navarra & SotoFaraco,…”
BackgroundCortical integration of audio–visual speech and non-speech stimuli
Wyk, B. C. V., Ramsay, G., Hudac, C. M., JONES, W. R., Lin, D. J., Klin, A., Lee, S. M., & Pelphrey, K. A. (2010) · Brain and Cognition · doi:10.1016/j.bandc.2010.07.002 · Cited by 33
Treille, A., Vilain, C., & Sato, M. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00420 · Cited by 29
“…alone, adding visual speech information from the speaker’s face is known to improve speech intelligibility in case of a degraded acoustic signal (Sumby and Pollack, 1954; Benoît et al., 1994; Schwartz et al., 2004), to facilitate the understanding of a semantically complex statement (Reisberg…”
BackgroundPutative mechanisms mediating tolerance for audiovisual stimulus onset asynchrony
Bhat, J., Miller, L. M., Pitt, M. A., & Shahin, A. J. (2014) · Journal of Neurophysiology · doi:10.1152/jn.00200.2014 · Cited by 26
“One such means is speech reading—attending to the talker’s mouth (Grant et al. 1998; Grant and Seitz 2000; Sumby and Pollack 1954).”
BackgroundZmigrod, L., & Zmigrod, S. (2016) · Multisensory Research · doi:10.1163/22134808-00002532 · Cited by 26
“…critical mechanism for shaping how organisms perceive and interact with a dynamic, noisy, stimulirich environment (Bishop and Miller, 2009; Girin et al., 2001; Grant et al., 1998; Stevenson and James, 2009; Sumby and Pollack, 1954; see reviews by: Calvert and Thesen, 2004; King and Calvert, 2001).”
BackgroundMoradi, S., Lidestam, B., & Rönnberg, J. (2016) · Trends in Hearing · doi:10.1177/2331216516653355 · Cited by 33
“In daily face-to-face conversation, listeners benefit from combined auditory and visual speech signals that facilitate the identification of speech stimuli in comparison with auditory-only or visual-only presentation (Erber, 1969; Sumby & Pollack, 1954).”
BackgroundThe Impact of Early Bilingualism on Face Recognition Processes
Kandel, S., Burfin, S., Méary, D., Ruiz-Tada, E., Costa, A., & Pascalis, O. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01080 · Cited by 30
“In addition, oro-facial information enhances consonant and vowel intelligibility in noisy environments (Sumby and Pollack, 1954; Benoît et al., 1994).”
BackgroundAudiovisual speech recognition: A review and forecast
Xia, L., Chen, G., Xu, X., Cui, J., & Gao, Y. (2020) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881420976082 · Cited by 35
“As early as 1954, Pollack and Sumby were the advocate of AVSR, who explicitly stated that visual information contributes to the understandings of speech in noisy environments.(17) Representative McGurk effect (described in article titled “Hearing Lips and Seeing Voices” by Harry McGurk and John MacDonald) further pres…”
BackgroundOn audiovisual spatial synergy: The fragility of the phenomenon
Soto‐Faraco, S., Morein‐Zamir, S., & Kingstone, A. F. (2005) · Perception & Psychophysics · doi:10.3758/bf03193323 · Cited by 32
“This is perhaps not surprising, since in naturalistic environments, auditory information may be accompanied by redundant visual cues (see, e.g., Grant & Seitz, 1998, 2000; Reisberg, 1978; Sumby & Pollack, 1954).”
BackgroundTraining Baldi to be multilingual: A case study for an Arabic Badr
Ouni, S., Cohen, M., & Massaro, D. (2005) · Speech Communication · doi:10.1016/j.specom.2004.11.008 · Cited by 29
“…able to compare the improvement made in the first version with the improvement made in the second version, we used a normalized measure presented by (Sumby and Pollack, 1954) that allows the comparison of the improvement even though auditory performance in each experiment was not exactly the same.”
BackgroundMethodInfluentialSoroker, N. (1995) · Neuropsychologia · doi:10.1016/0028-3932(94)00130-h · Cited by 25
Adeel, A. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00015 · Cited by 30
“…cocktail party), human brain naturally utilizes other modalities (such as lips, body language, facial expressions) to perceive speech or the conveyed message [i.e., speech-in-noise (SIN) perception] (Sumby and Pollack, 1954; McGurk and MacDonald, 1976; Summerfield, 1979; Patterson and Werker, 2003).”
BackgroundThe Development of the Mealings, Demuth, Dillon, and Buchholz Classroom Speech Perception Test
Mealings, K. T., Demuth, K., Buchholz, J. M., & Dillon, H. (2015) · Journal of Speech Language and Hearing Research · doi:10.1044/2015_jslhr-h-14-0332 · Cited by 24
“It has been well established that seeing the talker’s face facilitates speech perception, particularly in noisy listening conditions in which talkers exaggerate spoken articulation (Kim, Sironic, & Davis, 2011; Sumby & Pollack, 1954).”
Backgroundvan Hoesel, R. J. M. (2015) · Journal of the Association for Research in Otolaryngology · doi:10.1007/s10162-014-0503-7 · Cited by 24
“It is well established that the addition of visual information can improve speech understanding in noise, both for listeners with normal hearing and for users of hearing aids and cochlear implants (e.g., Sumby and Pollack 1954; Erber 1972; Summerfield 1979; Desai et al. 2008).”
BackgroundShared and modality-specific brain regions that mediate auditory and visual word comprehension
Keitel, A., Groß, J., & Kayser, C. (2020) · eLife · doi:10.7554/elife.56972 · Cited by 24
“, 2019; Summerfield, 1992), and can help to enhance speech intelligibility in noisy environments and in a second language (Navarra and Soto-Faraco, 2007; Sumby and Pollack, 1954; Yi et al., 2013).”
BackgroundPrince, P., Paul, B. T., Chen, J., Le, T., Lin, V. W., & Dimitrijevic, A. (2021) · European Journal of Neuroscience · doi:10.1111/ejn.15365 · Cited by 27
“In the context of social settings, successful speech perception relies not only on the characteristics of auditory signals and the level of background noise but also on the visual speech cues that influence neural representations of auditory information (Hirst et al., 2018; Sumby & Pollack, 1954).”
BackgroundCan audio-visual integration strengthen robustness under multimodal attacks?
Tian, Y., & Xu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00555 · Cited by 47
A Novel Machine Lip Reading Model
Huang, H., Song, C., Ting, J., Tian, T., Chen, H., Zhang, D., & Gao, D. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.01.181 · Cited by 24
Speech experience shapes the speechreading network and subsequent deafness facilitates it
Suh, M., Lee, H., Kim, J. S., Chung, C. K., & Oh, S. H. (2009) · Brain · doi:10.1093/brain/awp159 · Cited by 29
“For example, observing/viewing articulatory gestures improves speech recognition, especially in the presence of background noise (Sumby and Pollack, 1954).”
The Influence of Aging on Audiovisual Temporal Order Judgments
Fiacconi, C. M., Harvey, E. C., Sekuler, A. B., & Bennett, P. (2013) · Experimental Aging Research · doi:10.1080/0361073x.2013.761896 · Cited by 24
“Virsu et al. (2003) found that audiovisual temporal sensitivity declined significantly with age, at least when the two stimuli were presented from different locations. In Experiment 2, we attempted a more direct replication of Virsu et al. (2003) by presenting the auditory and visual stimuli from different locations.…”
BackgroundResultInfluentialA lip reading application on MS Kinect camera
Yargıç, A., & Doğan, M. (2013) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista.2013.6577656 · Cited by 24
“However, the importance of the visual features, such as lip movements and shape of the mouth, cannot be underestimated [6, 7].”
BackgroundDavis, C., Kislyuk, D. S., Kim, J., & Sams, M. E. (2008) · Brain Research · doi:10.1016/j.brainres.2008.04.077 · Cited by 29
How is the McGurk effect modulated by Cued Speech in deaf and hearing adults?
Bayard, C., Colin, C., & Leybaert, J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00416 · Cited by 27
“In face-to-face communication, speech perception is a multimodal process involving mainly auditory and visual (lip-reading) modalities (Sumby and Pollack, 1954; Grant and Seitz, 2000).”
BackgroundAuditory-tactile speech perception in congenitally blind and sighted adults
Sato, M., Cavé, C., Ménard, L., & Brasseur, A. (2010) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2010.08.017 · Cited by 24
“For instance, seeing the face of a speaker nhances auditory speech intelligibility (Benoît, Mohamadi, & andel, 1994; Sumby & Pollack, 1954) and, even when audition s lacking, some seen speech gestures can be captured by lipreadng (Bernstein, Demorest & Tucker, 1998).”
BackgroundPhonetic recalibration does not depend on working memory
Baart, M., & Vroomen, J. (2010) · Experimental Brain Research · doi:10.1007/s00221-010-2264-9 · Cited by 26
“For example, viewing a speaker’s articulatory movements (i.e. lipreading) is known to improve auditory speech intelligibility (e.g. Erber 1974), especially when the auditory input is ambiguous (Sumby and Pollack 1954).”
BackgroundSound Location Can Influence Audiovisual Speech Perception When Spatial Attention Is Manipulated
Möttönen, R., Tiippana, K. M., Sams, M. E., & Puharinen, H. (2011) · Seeing and Perceiving · doi:10.1163/187847511x557308 · Cited by 30
Taitelbaum‐Swead, R., & Fostick, L. (2016) · International Journal of Pediatric Otorhinolaryngology · doi:10.1016/j.ijporl.2016.11.022 · Cited by 24
Cuppini, C., Ursino, M., Magosso, E., Ross, L. A., Foxe, J. J., & Molholm, S. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00518 · Cited by 25
“…that congruent visual articulatory information enhances an observer’s speech perception abilities, improving the number of correct identifications (Sumby and Pollack, 1954; Calvert et al., 1998), and, conversely, that observation of incongruent speech gestures can rather dramatically alter what…”
BackgroundMultisensory Processing in Cochlear Implant Listeners
Barone, P., & Déguine, O. (2011) · Springer handbook of auditory research · doi:10.1007/978-1-4419-9434-9_15 · Cited by 29
A Multisensory Perspective on Human Auditory Communication
von Kriegstein, K. (2011) · Frontiers in neuroscience · doi:10.1201/b11092-43 · Cited by 32
Multisensory integration enhances phonemic restoration
Shahin, A. J., & Miller, L. M. (2009) · The Journal of the Acoustical Society of America · doi:10.1121/1.3075576 · Cited by 27
“…hypothesis is motivated © 2009 Acoustical Society of America25 3 /1744/7/$25.00 by numerous demonstrations that vision improves comprehension in noise Sumby and Pollack, 1954; Grant et al., 1998 , influences speech perception at a basic phonetic level McGurk and MacDonald, 1976; Repp et al., 1992;…”
BackgroundInfluentialJordan, T. R., & Thomas, S. M. (2011) · Attention Perception & Psychophysics · doi:10.3758/s13414-011-0152-4 · Cited by 24
“…powerful component of speech perception (e.g., Erber, 1969; Jordan, McCotter, & Thomas, 2000; MacLeod & Summerfield, 1987, 1990; McGurk & MacDonald, 1976; Middleweerd & Plomp, 1987; Reisberg, McLean, & Goldfield, 1987; Rosenblum, Yakel, & Green, 2000; Sumby & Pollack, 1954; Thomas & Jordan, 2002).”
BackgroundSeeing a singer helps comprehension of the song’s lyrics
Jesse, A., & Massaro, D. W. (2010) · Psychonomic Bulletin & Review · doi:10.3758/pbr.17.3.323 · Cited by 24
Lipread-induced phonetic recalibration in dyslexia
Baart, M., de Boer-Schellekens, L., & Vroomen, J. (2012) · Acta Psychologica · doi:10.1016/j.actpsy.2012.03.003 · Cited by 24
“It has been known for a long time that in daily life, lipread information helps to improve the eligibility of auditory speech (e.g. Sumby & Pollack, 1954).”
BackgroundAlegría, J. (2004) · Springer eBooks · doi:10.1007/978-94-017-1731-1_24 · Cited by 38
Grauwinkel, K., Dewitt, B., & Fagel, S. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-295 · Cited by 24
“Since five decades many investigations have documented that pertinent visual information enhances speech intelligibility when added to audible speech (e.g. [1], [2]).”
BackgroundIssues in the development of auditory-visual speech perception: adults, infants, and children
Sekiyama, K., & Burnham, D. K. (2004) · Interspeech (USB) · doi:10.21437/interspeech.2004-412 · Cited by 24
“Optical information from facial movements of a talker contributes to speech perception not only when acoustic information is degraded [1] or when the listener is hearingimpaired, but also when the acoustic information is clearly audible.”
BackgroundMatthews, I. A., Bangham, J., Harvey, R., & Cox, S. J. (1998) · Lecture notes in computer science · doi:10.1007/bfb0054762 · Cited by 24
“Even for untrained human listeners, being able to see the face of a speaker is known to significantly improve intelligibility particularly under noisy conditions [17, 39].”
BackgroundEffects of Spectro-Temporal Asynchrony in Auditory and Auditory-Visual Speech Processing
Grant, K. W., Greenberg, S., Poeppel, D., & van Wassenhove, V. (2004) · Seminars in Hearing · doi:10.1055/s-2004-832858 · Cited by 23
Evaluation of movement generation systems using the point-light technique
Bailly, G., Gibert, G., & Odisio, M. (2002) · Proceedings of 2002 IEEE Workshop on Speech Synthesis, 2002. · doi:10.1109/wss.2002.1224365 · Cited by 30
“The expected results should reproduce strong properties of natural speech: at all noise levels and even with clear speech, audiovisual performance is always superior to monomodal (audio or video-only) perception [15, 31].”
BackgroundAge Differences in the Ability to Use Visual Cues in Auditory Communication
Farrimond, T. (1959) · Language and Speech · doi:10.1177/002383095900200402 · Cited by 24
Training Hidden Markov Models by Hybrid Simulated Annealing for Visual Speech Recognition
Lee, J., & Park, C. H. (2006) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2006.384382 · Cited by 24
“There are several observations that the visual information is useful for speech recognition by both humans and machines in noisy circumstances [2], [3].”
BackgroundAudiovisual Speech Perception and the McGurk Effect
Rosenblum, L. D. (2019) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199384655.013.420 · Cited by 27
Towards a somatosensory theory of speech perception
Franken, M. K., Liu, B. C., & Ostry, D. J. (2022) · Journal of Neurophysiology · doi:10.1152/jn.00381.2022 · Cited by 32
“Using functional magnetic resonance imaging (fMRI) Skipper and colleagues (11) showed that several somatosensory areas, including left supramarginal gyrus and Brodmann areas 2 and 3, showed more activity in response to auditory compared with audiovisual speech stimuli.”
Biau, E., & Soto‐Faraco, S. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00527 · Cited by 25
“Corresponding lip movements have been long shown to facilitate comprehension in noisy conditions (Sumby and Pollack, 1954), or in contrast, affect speech processing when incongruent with utterance, e.g., the famous McGurk illusion (McGurk and MacDonald, 1976).”
BackgroundSilent articulation modulates auditory and audiovisual speech perception
Sato, M., Troille, É., Ménard, L., Cathiard, M.-A., & Gracco, V. L. (2013) · Experimental Brain Research · doi:10.1007/s00221-013-3510-8 · Cited by 31
“, multisensory enhancement is greatest when unimodal stimuli are least effective), concordant visual information improved speech intelligibility in noisy auditory conditions (Sumby and Pollack 1954; MacLeod and Summerfield 1987; Benoît et al. 1994).”
BackgroundAltieri, N. A., Pisoni, D. B., & Townsend, J. T. (2011) · Seeing and Perceiving · doi:10.1163/187847611x595864 · Cited by 28
“The ability to obtain visual information from the movements of a talker’s articulators through lip-reading is important for both normal-hearing and hearing-impaired listeners when perceiving speech (Erber, 1969; Sumby and Pollack, 1954).”
InfluentialGifford, R. H., Loiselle, L. H., Natale, S. C., Sheffield, S. W., Sunderhaus, L. W., Dietrich, M. S., & Dorman, M. F. (2018) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-h-16-0444 · Cited by 23
“In the majority of real-world communication settings, even listeners with normal hearing make use of visual cues to aid speech understanding in noise (e.g., Sumby and Pollack, 1954).”
BackgroundVisual analog of the acoustic amplitude envelope benefits speech perception in noise
Yuan, Y., Wayland, R. P., & Oh, Y. (2020) · The Journal of the Acoustical Society of America · doi:10.1121/10.0000737 · Cited by 23
Rosemann, S., Smith, D., Dewenter, M., & Thiel, C. M. (2020) · Cortex · doi:10.1016/j.cortex.2020.04.022 · Cited by 27
Peelle, J. E., Spehar, B., Jones, M. S., McConkey, S. M., Myerson, J., Hale, S., Sommers, M. S., & Tye‐Murray, N. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.0114-21.2021 · Cited by 25
MacGregor, L. J., Gilbert, R. A., Balewski, Z. Z., Mitchell, D. J., Erzinçlioğlu, S., Rodd, J. M., Duncan, J., Fedorenko, E. G., & Davis, M. H. (2022) · Neurobiology of Language · doi:10.1162/nol_a_00081 · Cited by 27
“…informative semantic contextual cues (Davis, Ford, Kherif, & Johnsrude, 2011; Miller, Heise, & Lichten, 1951; Obleser, Wise, Dresner, & Scott, 2007; Rysop, Schmitt, Obleser, & Hartwigsen, 2021) and other forms of prior knowledge (Miller & Isard, 1963; Sohoglu et al., 2014; Sumby & Pollack, 1954).”
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.