Visual Contribution to Speech Intelligibility in Noise
The DOI matches its Crossref record.
Abstract
Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech-to-noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech-to-noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech-to-noise ratios, its absolute contribution can be exploited most profitably under these conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies show that non-verbal facial gestures often account for a substantial portion of the perceived feeling and attitude of a speaker18,19 and that face-to-face communication enhances social connectivity20 and intelligibility21.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [21]
In the citing paper
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude130,131, that provides increased clarity132.”
From The speech neuroprosthesis (2024) · cites this paper as [132]
In the citing paper
“Estimates suggest that the information gain in this situation is the equivalent of boosting the audible signal by up to 15 dB (Sumby and Pollack, 1954)”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as Sumby and Pollack, 1954
In the citing paper
“Seeing the face ofa speaker can significantly embellish a degraded or noisy auditory speech signal so that it functionally raises the signal-tonoise ratio by as much as 22 dB”
From The McGurk effect in infants (1997) · cites this paper as Sumby & Pollack, 1954
Citations
Cited by
Items 201–300 of 1,005
Lansing, C. R., & McConkie, G. W. (2003) · Perception & Psychophysics · doi:10.3758/bf03194581 · Cited by 133
“Nevertheless, visual cues can significantly enhance perceptual accuracy of ambiguous or degraded auditory signals (Miller & Nicely, 1955; Sumby & Pollack, 1954).”
BackgroundGestures Orchestrate Brain Networks for Language Understanding
Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2009) · Current Biology · doi:10.1016/j.cub.2009.02.051 · Cited by 125
Older adults expend more listening effort than young adults recognizing audiovisual speech in noise
Gosselin, P. A., & Gagné, J. (2011) · International Journal of Audiology · doi:10.3109/14992027.2011.599870 · Cited by 153
“These results suggest that while visual cues can improve audiovisual speech recognition (Grant & Braida, 1991; Macleod & Summerfi eld, 1987, 1990; Sumby & Pollack, 1954), under conditions where performance is equated the additional visual cues place an extra demand on processing resources with…”
BackgroundTalker and Lexical Effects on Audiovisual Word Recognition by Adults With Cochlear Implants
Kaiser, A. R., Kirk, K. I., Lachs, L., & Pisoni, D. B. (2003) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2003/032 · Cited by 142
Speechreading using Probabilistic Models
Luettin, J., & Thacker, N. A. (1997) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0570 · Cited by 136
Computer lipreading for improved accuracy in automatic speech recognition
Silsbee, P. L., & Bovik, A. C. (1996) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.536928 · Cited by 121
Perceptual learning of degraded speech by minimizing prediction error
Sohoglu, E., & Davis, M. H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1523266113 · Cited by 122
“Previous behavioral studies have shown that the provision of relevant prior knowledge enhances both immediate perceptual clarity (1, 29, 35, 45, 46) and perceptual learning of degraded speech (6, 14, 16–18).”
BackgroundSamuel, A. G. (2010) · Annual Review of Psychology · doi:10.1146/annurev.psych.121208.131643 · Cited by 145
“At a general level, it had been known very early on that seeing the speaker's mouth can improve speech recognition scores (Sumby & Pollack 1954).”
Background(2010) · Handbook of Behavioral Neuroscience · doi:10.1016/b978-0-12-374593-4.00025-5 · Cited by 120
Remote Interpreting: Issues of Multi-Sensory Integration in a Multilingual Task
Moser‐Mercer, B. (2005) · Meta Journal des traducteurs · doi:10.7202/011014ar · Cited by 215
The evolution of speech: vision, rhythm, cooperation
Ghazanfar, A. A., & Takahashi, D. Y. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.06.004 · Cited by 119
ROGERS, W. T. (1978) · Human Communication Research · doi:10.1111/j.1468-2958.1978.tb00622.x · Cited by 120
Cue Integration: A Common Framework for Social Cognition and Physical Perception
Zaki, J. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613475454 · Cited by 118
“…because convergence between cues increases sensory acuity: The sound of someone speaking at a noisy party may be difficult to parse on its own, but it becomes much easier to understand when paired with the sight of that person’s lips moving (Stein & Meredith, 1993; Sumby & Pollack, 1954).”
BackgroundKeeping time in the brain: Autism spectrum disorder and audiovisual temporal processing
Stevenson, R. A., Segers, M., Ferber, S., Barense, M. D., Camarata, S., & Wallace, M. T. (2016) · Autism Research · doi:10.1002/aur.1566 · Cited by 118
Ripe for solution: Delayed development of multisensory processing in autism and its remediation
Beker, S., Foxe, J. J., & Molholm, S. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.11.008 · Cited by 114
“…the breath of a speaker on your skin (Gick & Derrick, 2009) can all provide additional information to an observer trying to understand what a speaker is saying to them (Ma, Zhou, Ross, Foxe, & Parra, 2009; Ross et al., 2011; Ross, SaintAmour, Leavitt, Javitt, & Foxe, 2007; Sumby & Pollack, 1954).”
Neural processing of asynchronous audiovisual speech perception
Stevenson, R. A., Altieri, N. A., Kim, S., Pisoni, D. B., & James, T. W. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.12.001 · Cited by 117
“…et al., 2004; Vroomen et al., 2001) or stimulus quality may be modulated to investigatemultisensory enhancement at low signal-to-noise ratios (James et al., 2009; Kim and James, in press; Stevenson and James, 2009; Sumby and Pollack, 1954; Summerfield, 1987; Werner and Noppeney, in press).”
InfluentialNeural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures
Kircher, T. T. J., Straube, B., Leube, D., Weis, S., Sachs, O., Willmes, K., Konrad, K., & Green, A. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2008.08.009 · Cited by 120
Bats perceptually weight prey cues across sensory systems when hunting in noise
Gomes, D. G. E., Page, R. A., Geipel, I., Taylor, R. C., Ryan, M. J., & Halfwerk, W. H. (2016) · Science · doi:10.1126/science.aaf7934 · Cited by 116
Visual Cues and Listening Effort: Individual Variability
Picou, E. M., Ricketts, T. A., & Hornsby, B. W. Y. (2011) · Journal of Speech, Language, and Hearing Research · doi:10.1044/1092-4388(2011/10-0154 · Cited by 116
Callan, D. E., Jones, J. A., Munhall, K., Kroos, C., Callan, A. M., & Vatikiotis-Bateson, E. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/089892904970771 · Cited by 115
“It is well known that the addition of visual speech information improves speech intelligibility over that of auditory speech information alone (Grant & Braida 1991; Sumby & Pollack, 1954).”
BackgroundReading Fluent Speech from Talking Faces: Typical Brain Networks and Individual Differences
Hall, D. A., Fussell, C., & Summerfield, A. Q. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929054021175 · Cited by 113
“Thus, linguistic facial movements are particularly informative for speech perception when the auditory signal is degraded by noise (Sumby & Pollack, 1954), reverberation, or the distortions introduced by a sensorineural hearing impairment (Jeffers & Barley, 1971).”
BackgroundIndividual differences in the perception of regional, nonnative, and disordered speech varieties
Bent, T., Baese‐Berk, M. M., Borrie, S. A., & McKee, M. (2016) · The Journal of the Acoustical Society of America · doi:10.1121/1.4966677 · Cited by 114
“…al., 2009), as is the speech from individuals with pathological speech disorders (Borrie et al., 2012a).1 Similarly, challenging environmental listening conditions, such as high levels of noise or reverberation, can substantially hinder a listener’s speech understanding (Sumby and Pollack, 1954).”
BackgroundThe Importance of Modality Specificity in Diagnosing Central Auditory Processing Disorder
Cacace, A. T., & McFarland, D. J. (2005) · American Journal of Audiology · doi:10.1044/1059-0889(2005/012 · Cited by 140
Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect
Walker, S., Bruce, V., & O’Malley, C. L. (1995) · Perception & Psychophysics · doi:10.3758/bf03208369 · Cited by 140
The race that precedes coactivation: development of multisensory facilitation in children
Barutchu, A., Crewther, D. P., & Crewther, S. G. (2009) · Developmental Science · doi:10.1111/j.1467-7687.2008.00782.x · Cited by 113
“Multisensory integration is recognized to enhance (e.g. Binnie, Montgomery & Jackson, 1974; Sumby & Pollack, 1954) and alter (e.g. Alais & Burr, 2004; Mateeff, Hohnsbein & Noack, 1985; McGurk & McDonald, 1976; Spence & Driver, 2000) many behavioural and cognitive processes.”
BackgroundSeeing and hearing others and oneself talk
Sams, M. E., Möttönen, R., & Sihvonen, T. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.11.006 · Cited by 134
“This is especially evident when speech is presented with poor signal-to-noise ratio [27,36], but seeing speech also improves identification of a difficult acoustic message when speech is presented without any noise [2].”
BackgroundAtypical rapid audio‐visual temporal recalibration in autism spectrum disorders
Noel, J., De Niear, M. A., Stevenson, R., Alais, D., & Wallace, M. T. (2017) · Autism Research · doi:10.1002/aur.1633 · Cited by 113
D’Souza, D., D’Souza, H., & Karmiloff‐Smith, A. (2017) · Journal of Child Language · doi:10.1017/s030500091700006x · Cited by 139
Noise Adaptive Stream Weighting in Audio-Visual Speech Recognition
Heckmann, M., Berthommier, F., & Kroschel, K. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206150 · Cited by 125
Lip Movement Exaggerations During Infant-Directed Speech
Green, J. R., Nip, I. S. B., Wilson, E. M., Mefferd, A. S., & Yunusova, Y. (2010) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2010/09-0005 · Cited by 127
Soto‐Faraco, S. (2004) · Cognition · doi:10.1016/j.cognition.2003.10.005 · Cited by 114
“The integration of audiovisual cues to speech not only helps language comprehension when the auditory signal is degraded (e.g. Sumby & Pollack, 1954), but it has an important role under good listening conditions, as long as lip movement information is available 0022-2860/$ - see front matter q 2004…”
BackgroundInfluences of Multisensory Experience on Subsequent Unisensory Processing
Shams, L., Wozny, D. R., Kim, R., & Seitz, A. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00264 · Cited by 111
“PERSPECTIVE ARTICLE published: 18 October 2011 doi: 10.3389/fpsyg.2011.00264”
BackgroundInfluentialSpence, C., Ranson, J., & Driver, J. (2000) · Perception & Psychophysics · doi:10.3758/bf03205560 · Cited by 128
Deconstructing the McGurk–MacDonald illusion.
Soto-Faraco, S., & Alsius, A. (2009) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/a0013483 · Cited by 110
Stevenson, R. A., Segers, M., Ferber, S., Barense, M. D., & Wallace, M. T. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00379 · Cited by 110
“Speech signals in visual and auditory modalities provide complementary information to the listener (Kavanagh and Mattingly, 1974), and when both are perceived in unison, behavioral gains in in speech perception are observed (Sumby and Pollack, 1954).”
BackgroundThe Principle of Inverse Effectiveness in Multisensory Integration: Some Statistical Considerations
Holmes, N. P. (2009) · Brain Topography · doi:10.1007/s10548-009-0097-2 · Cited by 110
“Other researchers point out that the PoIE in a particular dataset may depend upon the analytic methods used, and the assumptions made (Holmes 2007; Ross et al. 2007; Sumby and Pollack 1954).”
Advances in robust multimodal interface design
Oviatt, S. L. (2003) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2003.1231179 · Cited by 129
“1,16 Researchers developed actual systems capable of processing combined speech and lip movements during the 1980s and 1990s, including the classic work by Petajan”
BackgroundEndogenous Delta/Theta Sound-Brain Phase Entrainment Accelerates the Buildup of Auditory Streaming
Riecke, L., Sack, A. T., & Schroeder, C. E. (2015) · Current Biology · doi:10.1016/j.cub.2015.10.045 · Cited by 108
The Role of Audiovisual Speech and Orthographic Information in Nonnative Speech Production
Erdener, D., & Burnham, D. K. (2005) · Language Learning · doi:10.1111/j.0023-8333.2005.00303.x · Cited by 121
“Sumby and Pollack (1954) showed that in noisy conditions, visual input increases the perceived clarity of the auditory signal by a magnitude equivalent to 20 dB.”
Neural network lipreading system for improved speech recognition
Stork, D. G., Wolff, G. J., & Levine, E. (2003) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.226994 · Cited by 112
Visual speech perception without primary auditory cortex activation
Bernstein, L. E., Auer, E. T., Moore, J. K., Ponton, C. W., Don, M., & Singh, M. (2002) · Neuroreport · doi:10.1097/00001756-200203040-00013 · Cited by 108
Forty Years After Hearing Lips and Seeing Voices: the McGurk Effect Revisited
Alsius, A., Paré, M., & Munhall, K. G. (2017) · Multisensory Research · doi:10.1163/22134808-00002565 · Cited by 114
“…incidence of the McGurk illusion in different studies and thus, results from studies using open-set tasks (e.g., Ma et al. , 2009; McGurk and MacDonald, 1976; Ross et al. , 2007) and studies using closed-set tasks (e.g., Sekiyama et al. , 2014; Sumby and Pollack, 1954) cannot be directly compared.”
BackgroundResultHuman auditory cortex is sensitive to the perceived clarity of speech
Wild, C. J., Davis, M. H., & Johnsrude, I. S. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.035 · Cited by 116
“…and semantic content (Kalikow et al., 1977; Miller and Isard, 1963; Miller et al., 1951), familiarity with a speaker (Nygaard and Pisoni, 1998), and visual cues (Sumby and Pollack, 1954; Zekveld et al., 2008) can all greatly enhance the intelligibility of speech presented in masking noise.”
Musacchia, G., & Schroeder, C. E. (2009) · Hearing Research · doi:10.1016/j.heares.2009.06.018 · Cited by 117
“Related to perception, enhancement by vision is well known to be larger for weaker vocalizations (Sumby and Pollack, 1954; Ross et al., 2007).”
BackgroundMotor activation from visible speech: Evidence from stimulus response compatibility.
Kerzel, D., & Bekkering, H. (2000) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.26.2.634 · Cited by 114
“Listeners in a noisy environment have been shown to integrate visual and acoustic speech, resulting in improved intelligibility of the impoverished auditory signal (e.g., Sumby & Pollack, 1954).”
BackgroundRobust Self-Supervised Audio-Visual Speech Recognition
Shi, B., Hsu, W.-N., & Mohamed, A. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-99 · Cited by 185
Callan, D. E., Callan, A. M., Kroos, C., & Vatikiotis‐Bateson, E. (2001) · Cognitive Brain Research · doi:10.1016/s0926-6410(00)00054-9 · Cited by 107
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103
“Non-verbal facial expressions that accompany virtual or face-to-face communication provide numerous benefits compared with audio-only communication, including improved conveyance of emotion and attitude 130,131 , that provides increased clarity 132 .”
Computational principles and models of multisensory integration
Chandrasekaran, C. (2017) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2016.11.002 · Cited by 103
Multisensory Integration in Cochlear Implant Recipients
Stevenson, R. A., Sheffield, S. W., Butera, I. M., Gifford, R. H., & Wallace, M. T. (2017) · Ear and Hearing · doi:10.1097/aud.0000000000000435 · Cited by 119
The use of visual cues in the perception of non-native consonant contrasts
Hazan, V. L., Sennema, A., Faulkner, A., Ortega-Llebaría, M., Iba, M., & Chung, H. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2166611 · Cited by 119
A Multisensory Cortical Network for Understanding Speech in Noise
Bishop, C. W., & Miller, L. M. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21118 · Cited by 119
“Comprehension has long been known to improve when watching a speaker’s mouth movements, especially in noisy environments (Benoit, Mohamadi, & Kandel, 1994; MacLeod & Summerfield, 1987; Sumby & Pollack, 1954).”
Visually-guided Attention Enhances Target Identification in a Complex Auditory Scene
Best, V., Ozmeral, E. J., & Shinn‐Cunningham, B. G. (2007) · Journal of the Association for Research in Otolaryngology · doi:10.1007/s10162-007-0073-z · Cited by 104
“The role of visual cues in speech perception It is well documented that visual speech information obtained via lip reading can supplement speech intelligibility in noise (Sumby and Pollack 1954), and can even override acoustic cues in some cases of audiovisual conflict (McGurk and MacDonald 1976).”
BackgroundSadato, N., Okada, T., Honda, M., Matsuki, K.-I., Yoshida, M., Kashikura, K., Takei, W., Sato, T., Kochiyama, T., & Yonekura, Y. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh210 · Cited by 124
“Combining audible speech with visible articulation movements can improve comprehension to the same extent as altering the acoustic signal-to-noise ratio by 15--20 dB (Sumby and Polack, 1954).”
Seeing speech affects acoustic information processing in the human brainstem
Musacchia, G., Sams, M. E., Nicol, T. G., & Kraus, N. (2005) · Experimental Brain Research · doi:10.1007/s00221-005-0071-5 · Cited by 117
“2000), especially in noisy environments (Sumby and Pollack 1954; MacLeod and Summerfield 1987).”
Partan, S. R., Fulmer, A. G., Gounard, M. A. M., & Redmond, J. E. (2010) · Current Zoology · doi:10.1093/czoolo/56.3.313 · Cited by 103
“Multimodal shifts due to noise may be commonplace; they are predicted by the so-called “cocktail party phenomenon” in which people in acoustically noisy situations rely more on visual cues from mouth articulation than they would otherwise (Cherry, 1953; Sumby and Pollack, 1954; see discussions in Partan, 2004a, and Pa…”
BackgroundPerceiving Biological Motion: Dissociating Visible Speech from Walking
Santi, A., Servos, P., Vatikiotis-Bateson, E., Kuratate, T., & Munhall, K. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903322370726 · Cited by 103
“…a speaker’s face (especially the jaws, lips, and tongue) assists speech comprehension in good hearing conditions, when semantic content is difficult or when the speaker has an unfamiliar accent (Reisberg, McLean, & Goldfield, 1987), as well as under noisy hearing conditions (Sumby & Pollack, 1954).”
Spatiotemporal dynamics of audiovisual speech processing
Bernstein, L. E., Auer, E. T., Wagner, M., & Ponton, C. W. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.08.035 · Cited by 110
“For example, being able to see as well as hear a talker results in substantial gains to comprehending speech in noise (MacLeod and Summerfield, 1987; Sumby and Pollack, 1954), improvements in comprehending difficult messages under good listening conditions (Arnold and Hill, 2001; Reisberg et al.”
Sekiyama, K., Soshi, T., & Sakamoto, S. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00323 · Cited by 122
“Combined with the fact that visual contribution is generally larger in harder hearing circumstances (Sumby and Pollack, 1954; Grant and Seitz, 2000; Schwartz et al., 2004; Ross et al., 2007), it is thought that older adults tend to utilize visual information more to compensate for their declined…”
BackgroundDetection of Temporal Delays in Visual-Haptic Interfaces
Vogels, I. M. L. C. (2004) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.118.30394 · Cited by 112
“For instance, the intelligibility of speech in noise improves when the facial movements of the speaker are visible (Sumby & Pollack, 1954).”
BackgroundAging, Audiovisual Integration, and the Principle of Inverse Effectiveness
Tye‐Murray, N., Sommers, M. S., Spehar, B., Myerson, J., & Hale, S. (2010) · Ear and Hearing · doi:10.1097/aud.0b013e3181ddf7ff · Cited by 110
“Most adults, younger and older, recognize more speech when they can both see and hear a talker compared with either listening or watching alone (e.g. Grant et al. 1998; Sumby and Pollack 1954), but young adults on average recognize more speech in an audiovisual condition than do older adults (e.”
BackgroundInfluentialRosemann, S., & Thiel, C. M. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.04.023 · Cited by 110
“From other studies, we already know that the visual input in audio-visual speech can facilitate speech understanding (Campbell, 2008; Grant and Seitz et al., 2000; Driver and Noesselt, 2008; Irwin and DiBlasi, 2017; Rosenblum, 2008; Ross et al., 2007; Sumby and Pollack, 1954).”
Lip movement synthesis from speech based on Hidden Markov Models
Yamamoto, E., Nakamura, S., & Shikano, K. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00054-5 · Cited by 102
Adeel, A., Gogate, M., & Hussain, A. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.08.008 · Cited by 96
“The correlation between the visible properties of the articulatory organs and speech reception has been previously shown in numerous behavioural studies [1][2][3][4].”
BackgroundDetection of auditory (cross-spectral) and auditory–visual (cross-modal) synchrony
Grant, K. W., van Wassenhove, V., & Poeppel, D. (2004) · Speech Communication · doi:10.1016/j.specom.2004.06.004 · Cited by 101
“…listeners often make use of visual speech cues (i.e., via speechreading) to provide additional support to audition, and in most cases, are able to restore intelligibility back to what it would have been had the speech been presented in the quiet (Sumby and Pollack, 1954; Grant and Braida, 1991).”
BackgroundPredictive Coding in Sensory Cortex
Kok, P., & de Lange, F. P. (2015) · An Introduction to Model-Based Cognitive Neuroscience · doi:10.1007/978-1-4939-2236-9_11 · Cited by 98
Tye‐Murray, N., Spehar, B., Myerson, J., Hale, S., & Sommers, M. S. (2016) · Psychology and Aging · doi:10.1037/pag0000094 · Cited by 107
Auditory-visual processing represented in the human superior temporal gyrus
Reale, R. A., Calvert, G. A., Thesen, T., Jenison, R. L., Kawasaki, H., Oya, H., Howard, M. A., & Brugge, J. F. (2007) · Neuroscience · doi:10.1016/j.neuroscience.2006.11.036 · Cited by 103
Individual differences in the processing of speech and nonspeech sounds by normal-hearing listeners
Surprenant, A. M., & Watson, C. S. (2001) · The Journal of the Acoustical Society of America · doi:10.1121/1.1404973 · Cited by 97
Beat gestures modulate auditory integration in speech perception
Biau, E., & Soto‐Faraco, S. (2013) · Brain and Language · doi:10.1016/j.bandl.2012.10.008 · Cited by 106
“Then, as beats can be considered as rapid and simple visual cues of the paralinguistic context, one can assume that they act early in speech processing, the same way lips modulate speech perception (Sumby & Pollack, 1954; McGurk & MacDonald, 1976).”
BackgroundEnhancing Speech Intelligibility: Interactions Among Context, Modality, Speech Style, and Masker
Van Engen, K. J., Phelps, J. E. B., Smiljanic, R., & Chandrasekaran, B. (2014) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jslhr-h-13-0076 · Cited by 95
Multisensory interactions in primate auditory cortex: fMRI and electrophysiology
Kayser, C., Petkov, C. I., & Logothetis, N. K. (2009) · Hearing Research · doi:10.1016/j.heares.2009.02.011 · Cited by 112
“In such situations, the visual information can boost hearing capabilities by an equivalent of about 10–20 dB sound intensity (Ross et al., 2007; Sumby and Polack, 1954), although the exact gain depends on the signal to noise ratio and the kind of target signal itself.”
BackgroundCausal inference of asynchronous audiovisual speech
Magnotti, J. F., Ma, W. J., & Beauchamp, M. S. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00798 · Cited by 96
“When there is a single talker, integrating the auditory and visual speech information increases perceptual accuracy (Sumby and Pollack, 1954; Rosenblum et al., 1996; Schwartz et al., 2004; Ma et al., 2009); most computational work on audiovisual integration of speech has focused on this condition…”
BackgroundMultisensory integration: perceptual grouping by eye and ear
King, A. J., & Calvert, G. A. (2001) · Current Biology · doi:10.1016/s0960-9822(01)00175-0 · Cited by 99
“[1].”
Auditory-visual speech perception in normal-hearing and cochlear-implant listeners
Desai, S. B., Stickney, G. S., & Zeng, F. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2816573 · Cited by 103
Effects of Open-Set and Closed-Set Task Demands on Spoken Word Recognition
Clopper, C. G., Pisoni, D. B., & Tierney, A. T. (2006) · Journal of the American Academy of Audiology · doi:10.3766/jaaa.17.5.4 · Cited by 94
“In their seminal paper on multimodal speech perception, Sumby and Pollack (1954) examined the effects of response set size on spoken word recognition performance.”
BackgroundComparing models for audiovisual fusion in a noisy-vowel recognition task
Teissier, P., Robert-Ribes, J., Schwartz, J., & Guérin-Dugué, A. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.799688 · Cited by 95
Sensory-cognitive interactions in older adults
Humes, L. E., & Young, L. A. (2016) · Ear & Hearing · doi:10.1097/aud.0000000000000303 · Cited by 94
“…is known to vary with the acoustical signal to noise ratio such that the perceiver makes little use of visual information in quiet listening conditions, but relies heavily on this same information in acoustically challenging conditions (Sumby & Pollack 1954; Grant et al. 2007; Altieri et al. 2011).”
BackgroundAuditory and auditory-tactile processing in congenitally blind humans
Hötting, K., & Röder, B. (2009) · Hearing Research · doi:10.1016/j.heares.2009.07.012 · Cited by 120
“For example, reaction times are faster to multisensory stimuli than to unisensory stimuli (Miller, 1991), seeing lip movements enhances auditory speech perception (Sumby and Pollack, 1954) and both multisensory object recognition (Amedi et al.”
Speech, movement, and gaze behaviours during dyadic conversation in noise
Hadley, L. V., Brimijoin, W. O., & Whitmer, W. M. (2019) · Scientific Reports · doi:10.1038/s41598-019-46416-0 · Cited by 103
Lexical Influences in Audiovisual Speech Perception.
Brancazio, L. (2004) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.30.3.445 · Cited by 101
Lip-Reading Enables the Brain to Synthesize Auditory Features of Unknown Silent Speech
Bourguignon, M., Baart, M., Kapnoula, E. C., & Molinaro, N. (2019) · Journal of Neuroscience · doi:10.1523/jneurosci.1101-19.2019 · Cited by 103
“In everyday situations, seeing a speaker’s articulatory mouth gestures, here referred to as lip-reading or visual speech, can help us decode the auditory speech signal (Sumby and Pollack, 1954).”
BackgroundBrief Report: Arrested Development of Audiovisual Speech Perception in Autism Spectrum Disorders
Stevenson, R. A., Siemann, J. K., Woynaroski, T. G., Schneider, B. C., Eberly, H. E., Camarata, S. M., & Wallace, M. T. (2014) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-013-1992-7 · Cited by 84
“Visual cues especially enhance perception when the auditory signal is degraded (Sumby and Pollack 1954; Middelweerd and Plomp 1987) and when the message is particularly difficult to comprehend (Arnold and Hill 2001).”
BackgroundWorthy of swift trust? How brief interpersonal contact affects trust accuracy.
Schilke, O. S., & Huang, L. (2018) · Journal of Applied Psychology · doi:10.1037/apl0000321 · Cited by 114
“…in addition to providing cues on voice quality, timbre, and inflection, also offers a means for synchronizing and categorizing information (Sumby & Pollack, 1954)—each of which should also enable individuals to make attributions and inferences about the other person’s point of view, again…”
BackgroundChemosensory learning and flavour: Perception, preference and intake
Prescott, J. (2012) · Physiology & Behavior · doi:10.1016/j.physbeh.2012.04.008 · Cited by 107
Predicting foreign-accent adaptation in older adults
Janse, E., & Adank, P. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.658822 · Cited by 108
Neural development of networks for audiovisual speech comprehension
Dick, A. S., Solodkin, A., & Small, S. L. (2010) · Brain and Language · doi:10.1016/j.bandl.2009.08.005 · Cited by 90
Inside bilingualism: Language background modulates selective attention to a talker's mouth
Birulés, J., Bosch, L., Brieke, R., Pons, F., & Lewkowicz, D. J. (2018) · Developmental Science · doi:10.1111/desc.12755 · Cited by 100
Beat gestures influence which speech sounds you hear
Bosker, H. R., & Peeters, D. (2021) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2020.2419 · Cited by 119
Electrophysiological evidence for speech-specific audiovisual integration
Baart, M., Stekelenburg, J. J., & Vroomen, J. (2013) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2013.11.011 · Cited by 100
“…it is well-established that the brain integrates the auditory and lip-read signals into one coherent speech percept (e.g., McGurk & MacDonald, 1976; Sumby & Pollack, 1954) there is an ongoing debate whether audiovisual (AV) speech integration is qualitatively different from non-speech AV…”
BackgroundRivet, B., Laurent Girin, B., & Christian Jutten, B. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.872619 · Cited by 90
“Index Terms—Audiovisual coherence, blind source separation, convolutive mixture, speech enhancement, statistical modeling.”
BackgroundNoel, J., Stevenson, R. A., & Wallace, M. T. (2018) · European Journal of Neuroscience · doi:10.1111/ejn.13911 · Cited by 102
Primary and multisensory cortical activity is correlated with audiovisual percepts
Benoit, M. M., Raij, T., Lin, F., Jääskeläinen, I. P., & Stufflebeam, S. (2010) · Human Brain Mapping · doi:10.1002/hbm.20884 · Cited by 89
“Integration effects are most noticeable when one sensory system is impaired or when a component of the audiovisual signal is degraded [Callan et al., 2003; Stein and Meredith, 1993; Sumby and Pollack, 1954].”
BackgroundMagnotti, J. F., & Beauchamp, M. S. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005229 · Cited by 96
No rapid audiovisual recalibration in adults on the autism spectrum
Turi, M., Karaminis, T., Pellicano, E., & Burr, D. (2016) · Scientific Reports · doi:10.1038/srep21756 · Cited by 84
Audiovisual integration in noise by children and adults
Barutchu, A., Danaher, J., Crewther, S. G., Innes-Brown, H., Shivdasani, M. N., & Paolini, A. G. (2010) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2009.08.005 · Cited by 88
van der Smagt, M. J., van Engeland, H., & Kemner, C. (2007) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-006-0346-0 · Cited by 88
“For instance lip-reading can improve speech understanding, mainly under conditions of poor auditory intelligibility, as in noisy environments ( Sumby & Pollack, 1954 ).”
Receiver psychology turns 20: is it time for a broader approach?
Miller, C. T., & Bee, M. A. (2012) · Animal Behaviour · doi:10.1016/j.anbehav.2011.11.025 · Cited by 87
“…known examples of such multimodal interactions in humans include our reliance on lip reading to understand speech under noisy listening conditions (Sumby & Pollack 1954), the McGurk effect demonstrating the combined influence of acoustic and visual information on phoneme determination (McGurk &…”
BackgroundYi, H., Pingsterhaus, A., & Song, W. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.682677 · Cited by 94
Paré, M., Richler, R. C., ten Hove, M., & Munhall, K. G. (2003) · Perception & Psychophysics · doi:10.3758/bf03194582 · Cited by 91
“It is well known that the sight of a talker’s face can influence the perception of speech sounds: The intelligibility of speech in noise is enhanced when the listener can see the talker articulating (Sumby & Pollack, 1954).”
Background
Metadata
- Authors
- Sumby, W. H., & Pollack, I.
- Affiliations
- Human Factors Operations Research Laboratories, Washington 25, D. C.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1954-03-01
- Volume
- 26
- Issue
- 2
- Pages
- 212-215
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2014621385
- Cited by
- 2,583
- References
- 0
- Citations by year
- 2026: 58 · 2025: 80 · 2024: 82 · 2023: 101 · 2022: 105 · 2021: 136 · 2020: 100 · 2019: 123 · 2018: 102 · 2017: 129 · 2016: 105 · 2015: 107 · 2014: 110 · 2013: 122 · 2012: 96
- FWCI
- 1.81
- Type
- journal article
- Language
- en
- Update notices
- correction (1954-07-01)
- Open access
- closed
- Locations
- The Journal of the Acoustical Society of America (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Health Professions › Speech and Hearing › Hearing Loss and Rehabilitation
- Keywords
- speech intelligibility in noise, visual speech perception, lip reading, speech-to-noise ratio, vocabulary size, oral speech, speech intelligibility test
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.