Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 501–600 of 1,005
The Principle of Inverse Effectiveness in Audiovisual Speech Perception
van de Rijt, L. P. H., Roye, A., Mylanus, E. A. M., Opstal, A. J. V., & van Wanrooij, M. M. (2019) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2019.00335 · Cited by 56
O’Sullivan, A. E., Crosse, M. J., Liberto, G. M. D., Alain De Cheveigné, A., & Lalor, E. C. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.0906-20.2021 · Cited by 60
Utility curves: mean opinion scores considered biased
Knoche, H., De Meer, H. G., & Kirsh, D. (2003) · 1999 Seventh International Workshop on Quality of Service. IWQoS'99. (Cat. No.98EX354) · doi:10.1109/iwqos.1999.766473 · Cited by 51
“Sumby and Pollack [8] reported that the relative contribution of visual information is independent of the signalto-noise ratio (S/N), but the absolute contribution could be more pro tably exploited at low S/N.”
BackgroundLong-term training, transfer, and retention in learning to lipread
Massaro, D. W., Cohen, M. M., & Gesi, A. T. (1993) · Perception & Psychophysics · doi:10.3758/bf03205203 · Cited by 51
Audio-visual Speech Perception, Lipreading and Artificial Stimulation
Summerfield, A. Q. (1983) · Elsevier eBooks · doi:10.1016/b978-0-12-460440-7.50010-7 · Cited by 52
Audiovisual Speech Perception and Eye Gaze Behavior of Adults with Asperger Syndrome
Saalasti, S., Kätsyri, J., Tiippana, K. M., Laine‐Hernandez, M., von Wendt, L., & Sams, M. E. (2011) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-011-1400-0 · Cited by 57
It's not what you say but the way you say it: Matching faces and voices.
Lander, K., Hill, H. C. H., Kamachi, M., & Vatikiotis‐Bateson, E. (2007) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.33.4.905 · Cited by 50
Using Technology for Pronunciation Teaching, Learning, and Assessment
Pennington, M. C., & Rogerson‐Revell, P. (2018) · Research and practice in applied linguistics · doi:10.1057/978-1-137-47677-7_5 · Cited by 76
Audio–Visual Interactions with Intact Clearly Audible Speech
Davis, C., & Kim, J. (2004) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/02724980343000701 · Cited by 59
“The current result of finding no AV benefit for stimuli that are not perceived as speech even though they retain some of its temporal properties is similar to that of Summerfield (1979). In his experiment the presentation of a moving Lissajous curve, produced by an oscilloscope to have the temporal pattern as speech,…”
MethodResultPerception of Synthetic Speech
Pisoni, D. B. (1997) · Springer eBooks · doi:10.1007/978-1-4612-1894-4_43 · Cited by 56
Multimodal Chemosensory Interactions and Perception of Flavor
Prescott, J. (2011) · Frontiers in neuroscience · doi:10.1201/b11092-45 · Cited by 57
Is desynchrony tolerance adaptable in the perceptual organization of speech?
Remez, R. E., Ferro, D. F., Dubowski, K. R., Meer, J., Broder, R. S., & Davids, M. L. (2010) · Attention Perception & Psychophysics · doi:10.3758/app.72.8.2054 · Cited by 55
Holt, R. F., Kirk, K. I., & Hay-McCutcheon, M. J. (2010) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2010/09-0148 · Cited by 49
Susceptibility to a multisensory speech illusion in older persons is driven by perceptual processes
Setti, A., Burke, K. E., Kenny, R. A., & Newell, F. N. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00575 · Cited by 60
Visual Speech Perception in Children With Language Learning Impairments
Knowland, V. C. P., Evans, S., Snell, C., & Rosen, S. (2016) · Journal of Speech Language and Hearing Research · doi:10.1044/2015_jslhr-s-14-0269 · Cited by 60
The supramodal brain: implications for auditory perception
Rosenblum, L. D., Dias, J. W., & Dorsi, J. (2016) · Journal of Cognitive Psychology · doi:10.1080/20445911.2016.1181691 · Cited by 66
“In fact, similar to how visual speech information can improve comprehension of natural speech heard in difficult listening conditions (e.g. Sumby & Pollack, 1954), sine-wave comprehension is improved when paired with the visible articulating face associated with the transformed utterance (Remez,…”
BackgroundA sex difference in visual influence on heard speech
IRWIN, J. R., Whalen, D. H., & Fowler, C. A. (2006) · Perception & Psychophysics · doi:10.3758/bf03208760 · Cited by 53
Effects of separating auditory and visual sources on audiovisual integration of speech
Jones, J. A., & Munhall, K. G. (1997) · Canadian acoustics · Cited by 69
Alignment to visual speech information
Miller, R. M., Sanchez, K., & Rosenblum, L. D. (2010) · Attention Perception & Psychophysics · doi:10.3758/app.72.6.1614 · Cited by 58
“When the auditory signal is degraded either by hearing loss or by a noisy environment, individuals are aided by seeing the articulating face of a talker (e.g., Grant & Seitz, 2000; Sumby & Pollack, 1954).”
BackgroundLinguistic experience and audio-visual perception of non-native fricatives
Wang, Y., Behne, D. M., & Jiang, H. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2956483 · Cited by 47
“For native L1 perceivers speech perception is enhanced with visual information e.g., Erber, 1969; Sumby and Pollack, 1954 , especially when auditory distinctiveness decreases, such as in a noisy environment Bernstein et al., 2004; Erber, 1969; Sumby and Pollack, 1954; Summerfield, 1979 .”
BackgroundInfluentialWinn, M. B., Rhone, A. E., Chatterjee, M., & Idsardi, W. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00824 · Cited by 54
Neural signatures of face–voice synchrony in 5‐month‐old human infants
Hyde, D. C., Jones, B. L., Flom, R., & Porter, C. L. (2011) · Developmental Psychobiology · doi:10.1002/dev.20525 · Cited by 48
Song, J., Lee, H., Kang, H., Lee, D. S., Chang, S. O., & Oh, S. H. (2014) · Brain Structure and Function · doi:10.1007/s00429-013-0704-6 · Cited by 57
“However, while congruent orofacial articulatory movements significantly contribute to speech comprehension (Sumby and Pollack 1954; van Wassenhove et al. 2005), incongruent audiovisual (AV) stimuli may lead to novel percepts that match neither the auditory nor the visual information (McGurk and…”
BackgroundSchelinski, S., & von Kriegstein, K. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-019-04244-1 · Cited by 50
The biological basis of speech: What to infer from talking to the animals.
Trout, J. D. (2001) · Psychological Review · doi:10.1037/0033-295x.108.3.523 · Cited by 46
The effects of blurred vision on auditory-visual speech perception in younger and older adults
Legault, I., Gagné, J., Rhoualem, W., & Gosselin, P. A. (2010) · International Journal of Audiology · doi:10.3109/14992027.2010.509112 · Cited by 50
“…investigators have reported that speech understanding is signifi cantly improved when visual-speech cues are added to the auditory-speech signal (Sumby & Pollack, 1954; Erber, 1979; Hack & Erber, 1982; MacLeod & Summerfi eld, 1987, 1990; Grant & Braida, 1991; Grant et al, 1998; Jordan et al,…”
BackgroundLexical and indexical cues in masking by competing speech
Helfer, K. S., & Freyman, R. L. (2009) · The Journal of the Acoustical Society of America · doi:10.1121/1.3035837 · Cited by 48
Automatic optically-based recognition of speech
Finn, K. E., & Montgomery, A. A. (1988) · Pattern Recognition Letters · doi:10.1016/0167-8655(88)90094-3 · Cited by 48
Making myself understood: perceived factors affecting the intelligibility of sung text
Fine, P. A., & Ginsborg, J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00809 · Cited by 58
An acquired deficit of audiovisual speech processing
Hamilton, R. H., Shenton, J. T., & Coslett, H. B. (2006) · Brain and Language · doi:10.1016/j.bandl.2006.02.001 · Cited by 47
Visual voice activity detection as a help for speech source separation from convolutive mixtures
Rivet, B., Laurent Girin, B., & Christian Jutten, B. (2007) · Speech Communication · doi:10.1016/j.specom.2007.04.008 · Cited by 47
“For instance, pioneer works by Girin et al. [3] and then by Sodoyer et al. [4] have proposed to use a statistical model between the coherence of audio and visual speech features to estimate the separating matrix for additive mixtures.”
Visual Feedback of Tongue Movement for Novel Speech Sound Learning
Katz, W. F., & Mehta, S. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00612 · Cited by 60
“Research on healthy hearing populations has also shown that audiovisual integration enhances comprehension of spoken stimuli, particularly in noisy environments or situations where the speaker has a strong foreign accent (O’Neill, 1954; Sumby and Pollack, 1954; Erber, 1975; Reisberg et al., 1987).”
BackgroundPhysiology, Psychoacoustics and Cognition in Normal and Impaired Hearing
van Dijk, P., Başkent, D., Gaudrain, É., de Kleine, E., Wagner, A. E., & Lanting, C. (2016) · Advances in experimental medicine and biology · doi:10.1007/978-3-319-25474-6 · Cited by 55
Garnier, M., Ménard, L., & Alexandre, B. (2018) · The Journal of the Acoustical Society of America · doi:10.1121/1.5051321 · Cited by 51
“…(Auer and Bernstein, 2007; Bernstein et al., 2000; Conrad, 1977), for foreign listeners (Davis and Kim, 2001; Reisberg et al., 1987), or when communicating in a noisy background (Bernstein et al., 2004; Erber, 1975; MacLeod and Summerfield, 1987; Robert-Ribes et al., 1998; Sumby and Pollack, 1954).”
BackgroundLooking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation
Lee, J., Chung, S.-W., Kim, S., Kang, H.-G., & Sohn, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00139 · Cited by 71
La langue et les lèvres: Cross-language influences on bimodal speech perception.
Werker, J. F., Frost, P. E., & McGuirk, H. (1992) · Canadian Journal of Psychology/Revue Canadienne de Psychologie · doi:10.1037/h0084331 · Cited by 50
Specification of cross-modal source information in isolated kinematic displays of speech
Lachs, L., & Pisoni, D. B. (2004) · The Journal of the Acoustical Society of America · doi:10.1121/1.1757454 · Cited by 51
“…al. (1996) also demonstrated that providing point-light information about articulation in conjunction with auditory speech embedded in noise can result in increased speech intelligibility scores, just as fully illuminated visual displays can improve speech intelligibility (Sumby and Pollack, 1954).”
BackgroundMethodResultInfluentialAudio—Visual Interaction in the Context of Multi-Media Applications
Kohlrausch, A., & van de Par, S. (2005) · Springer eBooks · doi:10.1007/3-540-27437-5_5 · Cited by 58
ERP evidence that auditory–visual speech facilitates working memory in younger and older adults.
Frtusova, J. B., Winneke, A. H., & Phillips, N. A. (2013) · Psychology and Aging · doi:10.1037/a0031243 · Cited by 51
Murray, M. M., & Wallace, M. T. (2011) · Frontiers in neuroscience · doi:10.1201/b11092-14 · Cited by 46
Examining the McGurk illusion using high-field 7 Tesla functional MRI
Szycik, G. R., Stadler, J., Tempelmann, C., & Münte, T. F. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00095 · Cited by 49
“The presence of additional visual (V) information leads to a considerable improvement of intelligibility of auditory (A) input under noisy conditions (Sumby and Pollack, 1954; Schwartz et al., 2004; Ross et al., 2007).”
BackgroundPons, F., Andreu, L., Sanz‐Torrent, M., Buil‐Legaz, L., & Lewkowicz, D. J. (2012) · Journal of Child Language · doi:10.1017/s0305000912000189 · Cited by 53
“Given the potential benefit of audiovisual, as opposed to auditory only, articulatory information (Sumby & Pollack, 1954; Summerfield, 1979), it would be beneficial for children with SLI to respond to the combination of [1] The McGurk effect (McGurk & MacDonald, 1976) is a powerful illustration…”
Neural networks supporting audiovisual integration for speech: A large-scale lesion study
Hickok, G. S., Rogalsky, C., Matchin, W., Basilakos, A., Cai, J., Pillay, S. B., Ferrill, M., Mickelsen, S., Anderson, S. W., Love, T. E., et al. (2018) · Cortex · doi:10.1016/j.cortex.2018.03.030 · Cited by 51
Karas, P. J., Magnotti, J. F., Metzger, B. A., Zhu, L. L., Smith, K. B., Yoshor, D., & Beauchamp, M. S. (2019) · eLife · doi:10.7554/elife.48116 · Cited by 50
“It has long been known that viewing the talker’s face enhances the intelligibility of noisy auditory speech (Sumby and Pollack, 1954).”
BackgroundPerception of Visible Speech: Influence of Spatial Quantization
Campbell, C. S., & Massaro, D. W. (1997) · Perception · doi:10.1068/p260627 · Cited by 49
Limited visual control of the intelligibility of speech in face-to-face dialogue
Anderson, A., Bard, E. G., Sotillo, C., Newlands, A., & Doherty‐Sneddon, G. (1997) · Perception & Psychophysics · doi:10.3758/bf03211866 · Cited by 55
Some determinants of communication accuracy.
Mehrabian, A., & Reed, H. (1968) · Psychological Bulletin · doi:10.1037/h0026483 · Cited by 59
Paul, B. T., Bajin, M. D., Uzelac, M., Chen, J., Le, T., Lin, V. W., & Dimitrijevic, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-22117-z · Cited by 45
Audiovisual speech perception: Moving beyond McGurk
Engen, K. J. V., Dey, A., Sommers, M. S., & Peelle, J. E. (2022) · The Journal of the Acoustical Society of America · doi:10.1121/10.0015262 · Cited by 58
Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals
Liu, X., Kirilyuk, V., Yuan, X., Olwal, A., Chi, P., Chen, X., & Du, R. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581566 · Cited by 80
“Our work is inspired by prior research in automatic text-to-visual enhancement, proactive AI agents, and augmented communication.”
MethodThe neural processing of foreign-accented speech and its relationship to listener bias
Yi, H., Smiljanić, R., & Chandrasekaran, B. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00768 · Cited by 55
“…a conclusive implication of the behavioral results, but rather cautiously suggesting that the listener bias likely led to altered incorporation of visual cues which are beneficial for enhancing speech intelligibility in adverse listening situations (Sumby and Pollack, 1954; Grant and Seitz, 2000).”
ResultSeeing to hear? Patterns of gaze to speaking faces in children with autism spectrum disorders
IRWIN, J. R., & Brancazio, L. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00397 · Cited by 55
“Patterns of gaze to speaking faces in children with autism spectrum disorders”
BackgroundAudiovisual speech perception development at varying levels of perceptual processing
Lalonde, K., & Holt, R. F. (2016) · The Journal of the Acoustical Society of America · doi:10.1121/1.4945590 · Cited by 45
“The changes in the shape of the vocal tract that give rise to the structured acoustic signal also produce visible speech cues, creating a robust and reliable—and thus highly advantageous—multimodal signal (Rosenblum, 2005; Rowe, 1999; Sumby and Pollack, 1954).”
BackgroundSeeing and Being Seen: The Effects on Gesture Production
Mol, L., Krahmer, E., Maes, A., & Swerts, M. G. J. (2011) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2011.01558.x · Cited by 58
“Such an intuition would be consistent with actual findings (e.g. Sumby & Pollack, 1954).”
BackgroundResultCallan, D. E., Jones, J. A., & Callan, A. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00389 · Cited by 45
“Visual observation of gestural information available from a speaker’s face improves speech perception, especially under noisy conditions (Sumby and Pollack, 1954; Grant and Braida, 1991; Callan et al., 2001, 2003).”
BackgroundVisual Processing Affects the Neural Basis of Auditory Discrimination
Kislyuk, D. S., Möttönen, R., & Sams, M. E. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20152 · Cited by 55
“This has been shown for speech embedded in noise (Dodd, 1977; Sumby & Pollack, 1954), and for foreign and complex native speech (Reisberg, McLean, & Goldfield, 1987).”
BackgroundComparing Fusion Models for DNN-Based Audiovisual Continuous Speech Recognition
Abdelaziz, A. H. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2783545 · Cited by 51
“Especially in noisy environments, humans use lipreading as an additional source of information to compensate the distorted acoustic signals and to enhance their speech intelligibility [6].”
BackgroundDrijvers, L., Vaitonyté, J., & Özyürek, A. (2019) · Cognitive Science · doi:10.1111/cogs.12789 · Cited by 57
Deep-learning-based audio-visual speech enhancement in presence of Lombard effect
Michelsanti, D., Tan, Z., Sigurdsson, S. O., & Jensen, J. (2019) · Speech Communication · doi:10.1016/j.specom.2019.10.006 · Cited by 57
“Previous studies show that observing the speaker’s facial and lip movements contributes to speech perception (Sumby and Pollack, 1954; Erber, 1975; McGurk and MacDonald, 1976).”
BackgroundFoucart, A., Garcia, X., Ayguasanosa, M., Thierry, G., Martin, C. D., & Costa, A. (2015) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2015.06.027 · Cited by 52
Influence of native language phonetic system on audio-visual speech perception
Wang, Y., Behne, D. M., & Jiang, H. (2009) · Journal of Phonetics · doi:10.1016/j.wocn.2009.04.002 · Cited by 50
“Given previous findings that perceivers rely more on visual speech information when intelligibility is poor (Erber, 1969; Sumby & Pollack, 1954; Summerfield, 1979), these nonnative perceivers conceivably made use of the visual information as an additional channel of input in perceiving the…”
BackgroundAudio-visual speech perception off the top of the head
Davis, C., & Kim, J. (2005) · Cognition · doi:10.1016/j.cognition.2005.09.002 · Cited by 45
Sekiyama, K. (1994) · Journal of the Acoustical Society of Japan (E) · doi:10.1250/ast.15.143 · Cited by 45
What's in a face? Visual contributions to speech segmentation
Mitchel, A. D., & Weiss, D. J. (2009) · Language and Cognitive Processes · doi:10.1080/01690960903209888 · Cited by 53
“Though faces did not facilitate segmentation in this condition, it is possible that such visual displays could enhance performance under noisy conditions, as has been reported for other tasks (e.g., Sumby & Pollack, 1954).”
BackgroundResultInfluentialSpeech Perception and Spoken Word Recognition: Research and Theory
Cleary, M., & Pisoni, D. B. (2008) · Wiley eBooks · doi:10.1002/9780470753477.ch16 · Cited by 45
“Access to a speaker’s face has been shown to significantly improve intelligibility under less than ideal conditions, such as low signal-to-noise ratios (Sumby & Pollack, 1954).”
BackgroundGrossman, R. B., Steinhart, E., Mitchell, T., & McIlvane, W. (2015) · Autism Research · doi:10.1002/aur.1447 · Cited by 44
“TD participants derive significant comprehension benefit from mouth movements [Sumby & Pollack, 1954] and show reduced AV integration when relying on peripheral face regions [Par e, Richler, Hove, & Munhall, 2003].”
BackgroundHay-McCutcheon, M. J., Pisoni, D. B., & Hunt, K. K. (2009) · International Journal of Audiology · doi:10.1080/14992020802644871 · Cited by 48
“For example, Sumby and Pollack (1954) demonstrated the importance of visual information for speech understanding in the presence of background noise.”
Hearing Faces and Seeing Voices: The Integration and Interaction of Face and Voice Processing
Stevenage, S. V., & Neil, G. J. (2014) · Psychologica Belgica · doi:10.5334/pb.ar · Cited by 44
Banks, M. I., Uhlrich, D. J., Smith, P. H., Krause, B. M., & Manning, K. A. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhr048 · Cited by 48
“For example, visual cues from a speaker effectively ‘‘amplify’’ speech sounds in noisy backgrounds (Sumby and Pollack 1954), silent lipreading activates auditory cortical areas in humans (Calvert et al. 1997), and conflicting visual information can alter speech perception (McGurk and MacDonald…”
BackgroundRecognition of Visual Speech Elements Using Adaptively Boosted Hidden Markov Models
Foo, S. W., Lian, Y., & Dong, L. (2004) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2004.826773 · Cited by 49
Venezia, J. H., Fillmore, P., Matchin, W., Isenberg, A. L., Hickok, G. S., & Fridriksson, J. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.038 · Cited by 49
“Adding visual speech to an auditory speech signal improves intelligibility for both clear (Arnold and Hill, 2001) and distorted (Erber, 1969; MacLeod and Summerfield, 1987; McCormick, 1979; Neely, 1956; Ross et al., 2007; Sumby and Pollack, 1954) speech, improves acquisition of non-native speech sound categories (Hard…”
BackgroundEffects of perceptual training on second language vowel perception and production
Inceoglu, S. (2015) · Applied Psycholinguistics · doi:10.1017/s0142716415000533 · Cited by 59
“…with good hearing ability in environments degraded by background noise (e.g., MacLeod & Summerfield, 1990; Rosenblum, Johnson, & Saldaña, 1996; Sumby & Pollack, 1954), when messages are conceptually difficult to understand (e.g., Arnold & Hill, 2001; Reisberg, McLean, & Goldfield, 1987), and…”
BackgroundWhat accounts for individual differences in susceptibility to the McGurk effect?
Brown, V. A., Hedayati, M., Zanger, A., Mayn, S., Ray, L., Dillman-Hasso, N. H., & Strand, J. F. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207160 · Cited by 47
Multistage audiovisual integration of speech: dissociating identification and detection
Eskelund, K., Tuomainen, J., & Andersen, T. (2010) · Experimental Brain Research · doi:10.1007/s00221-010-2495-9 · Cited by 48
“The integration of auditory and visual speech signals affects phonetic identification (McGurk and MacDonald 1976), detection (Grant and Seitz 2000), intelligibility ( Sumby and Pollack 1954 ), localization (Bertelson 1999), time of occurrence (Soto-Faraco and Alsius 2009), and decoding of emotional tone (de Gelder and…”
BackgroundSzycik, G. R., Jansma, H., & Münte, T. F. (2009) · Human Brain Mapping · doi:10.1002/hbm.20640 · Cited by 44
“Thus, speech comprehension in noisy environments is considerably improved when the comprehender can look at the speaker’s articulatory movements [Ross et al., 2007; Sumby and Pollack, 1954].”
BackgroundSzycik, G. R., Tausche, P., & Münte, T. F. (2007) · Brain Research · doi:10.1016/j.brainres.2007.08.027 · Cited by 52
Developing an audio-visual speech source separation algorithm
Sodoyer, D., Laurent Girin, B., Christian Jutten, B., & Schwartz, J. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.002 · Cited by 44
“…one: to paraphrase the formula used by Lynne Bernstein and Christian Benoît (1996) to introduce the AVSP special session in ICSLP’96, we know, since Sumby and Pollack (1954) at least, that lipreading improves speech identification in noise, and since Petajan (1984), that Audio-Visual Speech…”
Trainable Articulatory Control Models for Visual Speech Synthesis
Beskow, J. (2004) · International Journal of Speech Technology · doi:10.1023/b:ijst.0000037076.86366.8d · Cited by 48
“This has been shown for natural speech (Sumby and Pollack, 1954) as well as for synthetic visual speech in several languages (LeGoff et al., 1994; Massaro, 1998; Siciliano et al., 2003).”
BackgroundSpeech Perception in Perceivers With Hearing Loss: Synergy of Multiple Modalities
Massaro, D. W., & Cohen, M. M. (1999) · Journal of Speech Language and Hearing Research · doi:10.1044/jslhr.4201.21 · Cited by 43
Articulatory features for robust visual speech recognition
Saenko, K., Darrell, T., & Glass, J. R. (2004) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1027933.1027960 · Cited by 47
“The idea is to use a set of parallel SVM classifiers to extract different articulatory attributes from the input images, and then combine their decisions to obtain higher-level units, such as visemes or words.”
MethodA New 3D Lip Model for Analysis and Synthesis of Lip Motion in Speech Production
Revéret, L., & Benoît, C. (1998) · HAL · Cited by 47
An ALE meta‐analysis on the audiovisual integration of speech signals
Erickson, L. C., Heeg, E., Rauschecker, J. P., & Turkeltaub, P. E. (2014) · Human Brain Mapping · doi:10.1002/hbm.22572 · Cited by 48
“Common everyday examples include trying to have a conversation with someone in a noisy setting [Nath and Beauchamp, 2011; Sumby and Pollack, 1954], or viewing a dubbed foreign language film or poorly downloaded/synchronized video.”
BackgroundPreschoolers Benefit From Visually Salient Speech Cues
Lalonde, K., & Holt, R. F. (2014) · Journal of Speech Language and Hearing Research · doi:10.1044/2014_jslhr-h-13-0343 · Cited by 46
“In adulthood, visual information enhances speech perception beyond what is possible with auditory information alone, particularly when the auditory signal is degraded by noise (e.g., Sumby & Pollack, 1954) or hearing loss (e.g., Erber, 1969).”
BackgroundPrieto, P., Puglesi, C., Borràs-Comes, J., Arroyo, E., & Blat, J. (2014) · Journal of Phonetics · doi:10.1016/j.wocn.2014.10.005 · Cited by 43
Reading faces for information about words and emotions in adolescents with autism
Grossman, R. B., & Tager-Flusberg, H. (2008) · Research in Autism Spectrum Disorders · doi:10.1016/j.rasd.2008.02.004 · Cited by 43
Gordon, M. S., & Allen, S. (2009) · Experimental Aging Research · doi:10.1080/03610730902720398 · Cited by 51
“…Group, LLC ISSN: 0361-073X print=1096-4657 online DOI: 10.1080/03610730902720398 audio and visual signals has been found to improve detection and comprehension of speech across a wide variety of contexts for younger adults (e.g., Campbell & Dodd, 1980; Rosenblum, 2005; Sumby & Pollack, 1954).”
BackgroundInfluentialThe word superiority effect in audiovisual speech perception
Fort, M., Spinelli, E., Savariaux, C., & Kandel, S. (2010) · Speech Communication · doi:10.1016/j.specom.2010.02.005 · Cited by 43
“Several studies have shown that the information on the speaker's orofacial gestures enhances phoneme identification, especially in noisy situations (Benoıˆt et al., 1994; Erber, 1969; Sumby and Pollack, 1954; see Green, 1998 for a review).”
BackgroundSeeing the initial articulatory gestures of a word triggers lexical access
Fort, M., Kandel, S., Chipot, J., Savariaux, C., Granjon, L., & Spinelli, E. (2012) · Language and Cognitive Processes · doi:10.1080/01690965.2012.701758 · Cited by 43
“For example, seeing the articulatory gestures of a speaker’s face enhances speech intelligibility in noisy environments (Sumby & Pollack, 1954).”
BackgroundReal-time analysis-synthesis and intelligibility of talking faces.
Goff, B. L., Guiard‐Marigny, T., Cohen, M., & Benoı̂t, C. (1994) · SSW · Cited by 43
Huber, L. S., Geirhos, R., & Wichmann, F. A. (2023) · Journal of Vision · doi:10.1167/jov.23.7.4 · Cited by 42
Resolution of impaired multisensory processing in autism and the cost of switching sensory modality
Crosse, M. J., Foxe, J. J., Tarrit, K., Freedman, E. G., & Molholm, S. (2022) · Communications Biology · doi:10.1038/s42003-022-03519-1 · Cited by 55
Hendrikse, M. M. E., Llorach, G., Grimm, G., & Hohmann, V. (2018) · Speech Communication · doi:10.1016/j.specom.2018.05.008 · Cited by 42
Processing communicative facial and vocal cues in the superior temporal sulcus
Deen, B., Saxe, R. R., & Kanwisher, N. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117191 · Cited by 51
Kushnerenko, E. V., Tomalski, P., Ballieux, H., Potton, A., Birtles, D., Frostick, C., & Moore, D. G. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00432 · Cited by 56
From single cells to social perception
Barraclough, N. E., & Perrett, D. I. (2011) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2010.0352 · Cited by 55
Audiovisual perception in adverse conditions: Language, speaker and listener effects
Hazan, V. L., Kim, J., & Chen, Y. (2010) · Speech Communication · doi:10.1016/j.specom.2010.05.003 · Cited by 42
“This is surprising, as visual cues are an important source of information for native speakers in adverse listening conditions (Sumby and Pollack, 1954; Erber, 1969; Binnie et al., 1974; Grant and Seitz, 2000; Ross et al., 2007).”
BackgroundMethodStatistical multimodal integration for audio-visual speech processing
Nakamura, S. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1021886 · Cited by 42
Multimodal speech processing using asynchronous Hidden Markov Models
Bengio, S. (2003) · Information Fusion · doi:10.1016/j.inffus.2003.04.001 · Cited by 49
“It is well known that seeing the speaker’s face in addition to hearing his (her) voice can often improve speech intelligibility, particularly in noisy environments [ 3 ], mainly thanks to the complementarity of the visual and acoustic signals.”
BackgroundA 3-d virtual head as a tool for speech therapy for children
Fagel, S., & Madany, K. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-655 · Cited by 41
“Several properties of audible speech – especially intelligibility – can be enhanced by the visibility of natural speech movements [1] or synthetic speech movements [2].”
BackgroundVisual Contribution to Speech Perception: Measuring the Intelligibility of Animated Talking Heads
Ouni, S., Cohen, M. M., Ishak, H., & Massaro, D. W. (2006) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1155/2007/47891 · Cited by 43
“One of our goals is to extend the metric proposed by Sumby and Pollack [1] to describe the benefit provided by a synthetic animated face relative to the benefit provided by a natural face.”
BackgroundMethodInfluential
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.