Perception of Synthesized Audible and Visible Speech
Abstract
The research reported in this paper uses novel stimuli to study how speech perception is influenced by information presented to ear and eye. Auditory and visual sources of information (syllables) were synthesized and presented in isolation or in factorial combination. A five-step continuum between the syllables ibal and idal was synthesized along both auditory and visual dimensions, by varying properties of the syllable at its onset. The onsets of the second and third formants were manipulated in the audible speech. For the visible speech, the shape of the lips and the jaw position at the onset of the syllable were manipulated. Subjects’ identification judgments of the test syllables presented on videotape were influenced by both auditory and visual information. The results were used to test between a fuzzy logical model of speech perception (FLMP) and a categorical model of perception (CMP). These tests indicate that evaluation and integration of the two sources of information makes available continuous as opposed to just categorical information. In addition, the integration of the two sources appears to be nonadditive in that the least ambiguous source has the largest impact on the judgment. The two sources of information appear to be evaluated, integrated, and identified as described by the FLMP-an optimal algorithm for combining information from multiple sources. The research provides a theoretical framework for understanding the improvement in speech perception by hearing-impaired listeners when auditory speech is supplemented with other sources of information.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The McGurk effect occurs with both elaborate and schematic synthetic visual stimuli.
“The effect occurs with the use of both elaborate and schematic synthetic visual stimuli”
From The McGurk effect in infants (1997) · cites this paper as Massaro & Cohen, 1990
Cites
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · see citing works
Platt, J. R. (1964) · Science · doi:10.1126/science.146.3642.347 · Cited by 2,907 · see citing works
Software for a cascade/parallel formant synthesizer
Klatt, D. H. (1980) · The Journal of the Acoustical Society of America · doi:10.1121/1.383940 · Cited by 1,585 · see citing works
Continuous Shading of Curved Surfaces
Gouraud, H. (1971) · IEEE Transactions on Computers · doi:10.1109/t-c.1971.223313 · Cited by 700 · see citing works
Speech perception by ear and eye a paradigm for psychological inquiry
Massaro, D. W. (1987) · Cited by 585 · see citing works
Self-citationParameterized Models for Facial Animation
Parke (1982) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.1982.1674492 · Cited by 487 · see citing works
Visual influences on speech perception processes
Macdonald, J. B., & McGurk, H. (1978) · Perception & Psychophysics · doi:10.3758/bf03206096 · Cited by 458 · see citing works
Physical Characteristics of Comets
Vsekhsvyatskii, S. K., Raugh, A. C., & A'Hearn, M. (1998) · Soviet Astronomy · doi:10.26007/9dtm-ym89 · Cited by 458 · see citing works
Platt, S. M., & Badler, N. I. (1981) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/800224.806812 · Cited by 404 · see citing works
Speech understanding and aging
Working Group on Speech Understanding and Aging (1988) · The Journal of the Acoustical Society of America · doi:10.1121/1.395965 · Cited by 393 · see citing works
Use of Visual Information for Phonetic Perception
Summerfield, Q. A. (1979) · Phonetica · doi:10.1159/000259969 · Cited by 320 · see citing works
Speech Understanding and Aging
National Research Council (1977) · National Academies Press eBooks · doi:10.17226/20619 · Cited by 286 · see citing works
Platt, S. M., & Badler, N. I. (1981) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965161.806812 · Cited by 278 · see citing works
A parametric model for human faces.
Parke, F. I. (1974) · Defense Technical Information Center (DTIC) · Cited by 272 · see citing works
Effects of Training on the Visual Recognition of Consonants
Walden, B. E., Prosek, R. A., Montgomery, A. A., Scherr, C. K., & Jones, C. J. (1977) · Journal of Speech and Hearing Research · doi:10.1044/jshr.2001.130 · Cited by 218 · see citing works
The Role of Vision in the Perception of Speech
Dodd, B. J. (1977) · Perception · doi:10.1068/p060031 · Cited by 154 · see citing works
Physical characteristics of the lips underlying vowel lipreading performance
Montgomery, A. A., & Jackson, P. L. (1983) · The Journal of the Acoustical Society of America · doi:10.1121/1.389537 · Cited by 131 · see citing works
Generating vocal tract shapes from formant frequencies
Ladefoged, P., Harshman, R. A., Goldstein, L. M., & Rice, L. (1978) · The Journal of the Acoustical Society of America · doi:10.1121/1.382086 · Cited by 89 · see citing works
Speech and expression: A computer solution to face animation
Pearce, A., Wyvill, B., Wyvill, G., & Hill, D. J. (1986) · Proceedings · doi:10.20380/gi1986.24 · Cited by 87 · see citing works
A model for human faces that allows speech synchronized animation
Parke, F. I. (1975) · Computers & Graphics · doi:10.1016/0097-8493(75)90024-2 · Cited by 71 · see citing works
Analysis, synthesis, and perception of visible articulatory movements
Brooke, N. M., & Summerfield, A. Q. (1983) · Journal of Phonetics · doi:10.1016/s0095-4470(19)30777-6 · Cited by 62 · see citing works
Speechreading supplemented with frequency-selective sound-pressure information
Breeuwer, M. M., & Plomp, R. (1984) · The Journal of the Acoustical Society of America · doi:10.1121/1.391255 · Cited by 58 · see citing works
Audio-visual Speech Perception, Lipreading and Artificial Stimulation
Summerfield, A. Q. (1983) · Elsevier eBooks · doi:10.1016/b978-0-12-460440-7.50010-7 · Cited by 51 · see citing works
Predicting Audiovisual Consonant Recognition Performance of Hearing-Impaired Adults
Walden, B. E., Prosek, R. A., & Worthington, D. W. (1974) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1702.270 · Cited by 47 · see citing works
Auditory and Audiovisual Feature Transmission in Hearing-Impaired Adults
Walden, B. E., Prosek, R. A., & Worthington, D. W. (1975) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1802.272 · Cited by 35 · see citing works
Voice/mouth synthesis and tactual/visual perception of /pa, ba, ma/
Erber, N. P., & Filippo, C. L. D. (1978) · The Journal of the Acoustical Society of America · doi:10.1121/1.382084 · Cited by 25 · see citing works
Development of a model for generating synthetic animated lip shapes
Montgomery, A. A. (1980) · The Journal of the Acoustical Society of America · doi:10.1121/1.2004815 · Cited by 11 · see citing works
Schiffman, R. L. (1975) · Computers & Graphics · doi:10.1016/0097-8493(75)90023-0 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Speech listening specifically modulates the excitability of tongue muscles: a TMS study
Fadiga, L., Craighero, L., Buccino, G., & Rizzolatti, G. (2002) · European Journal of Neuroscience · doi:10.1046/j.0953-816x.2001.01874.x · Cited by 855 · see citing works
“The ®rst posits that speech is formed by sounds that rely for their production and perception on separate processes, neither of which is distinctly linguistic (Kuhl & Miller, 1975; Klatt, 1979; Massaro & Cohen, 1983, 1990; Sussman, 1989).”
BackgroundConfigural information in facial expression perception.
Calder, A. J., Young, A. W., Keane, J., & Dean, M. P. (2000) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.26.2.527 · Cited by 651 · see citing works
Comprehension of illustrated text: Pictures help to build mental models
Glenberg, A. M., & Langston, W. E. (1992) · Journal of Memory and Language · doi:10.1016/0749-596x(92)90008-l · Cited by 497 · see citing works
Oviatt, S. L. (2007) · Human factors and ergonomics · doi:10.1201/9781410615862.ch21 · Cited by 485 · see citing works
Modeling Coarticulation in Synthetic Visual Speech
Cohen, M. M., & Massaro, D. W. (1993) · Springer eBooks · doi:10.1007/978-4-431-66911-1_13 · Cited by 447 · see citing works
“Our current software is a direct descendant of Parke (1974) incorporating code developed by Pearce, Wyvill, Wyvill, and Hill (1986) and ourselves (Cohen & Massaro, 1990; Massaro & Cohen, 1990).”
BackgroundMethodSelf-citationRosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. (1997) · Perception & Psychophysics · doi:10.3758/bf03211902 · Cited by 367 · see citing works
Perceiving affect from the voice and the face
Massaro, D. W., & Egan, P. B. (1996) · Psychonomic Bulletin & Review · doi:10.3758/bf03212421 · Cited by 286 · see citing works
Self-citationAn audiovisual test of kinematic primitives for visual speech perception.
Rosenblum, L. D., & Saldaña, H. M. (1996) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.22.2.318 · Cited by 204 · see citing works
Matching phonetic information in lips and voice is robust in 4.5-month-old infants
Patterson, M. L., & Werker, J. F. (1999) · Infant Behavior and Development · doi:10.1016/s0163-6383(99)00003-x · Cited by 177 · see citing works
Featural evaluation, integration, and judgment of facial affect.
Ellison, J. W., & Massaro, D. W. (1997) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.23.1.213 · Cited by 119 · see citing works
“The fuzzy logical model of perception (FLMP) (Massaro & Cohen, 1990, 1993; Massaro & Ferguson, 1993) has been shown to provide superior predictive capability in similar experiments in the realm of speech perception and other domains and also could lend itself to analysis of facial affect…”
MethodSelf-citationSpeech and Non-Speech Audio-Visual Illusions: A Developmental Study
Tremblay, C., Champoux, F., Voss, P., Bacon, B. A., Leporé, F., & Théoret, H. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000742 · Cited by 110 · see citing works
“Subsequent studies have confirmed that the McGurk effect is a very robust illusion [5,6].”
BackgroundBimodal speech perception: an examination across languages
Massaro, D. W., Cohen, M. M., Gesi, A., Heredia, R., & Tsuzaki, M. (1993) · Journal of Phonetics · doi:10.1016/s0095-4470(19)30230-x · Cited by 108 · see citing works
Self-citationPerformative Facial Expressions in Animated Faces
Poggi, I., & Pélachaud, C. (2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.003.0008 · Cited by 94 · see citing works
“At the same time, looking at a face while talking improves human perception (Benoit 1990; Massaro and Cohen 1990; Summerfield 1992).”
BackgroundThe paradigm and the fuzzy logical model of perception are alive and well.
Massaro, D. W., & Cohen, M. M. (1993) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.122.1.115 · Cited by 92 · see citing works
Self-citationVisual, orthographic, phonological, and lexical influences in reading.
Massaro, D. W., & Cohen, M. M. (1994) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.20.6.1107 · Cited by 80 · see citing works
“We remind the reader of our history of falsifying various linear (additive, averaging, single channel, and categorical) models (Massaro, 1987; Massaro & Cohen, 1990).”
BackgroundMethodSelf-citationRoy, A. C., Craighero, L., Fabbri‐Destro, M., & Fadiga, L. (2008) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2008.03.006 · Cited by 77 · see citing works
“The most accepted view is that speech perception depends on auditory-cognitive mechanisms specifically devoted to the analysis of sounds conveying words (Kuhl and Miller, 1975; Klatt, 1979; Sussman, 1989; Massaro and Cohen, 1990).”
BackgroundThe Distribution of Attention Across a Talker's Face
Thompson, L. A., Malmberg, J., Goodell, N. K., & Boring, R. L. (2004) · Discourse Processes · doi:10.1207/s15326950dp3801_6 · Cited by 73 · see citing works
“…male and female pictures and voices are swapped (Green, Kuhl, Meltzoff, & Stevens, 1991), when synthetic visual and auditory stimuli are presented (Massaro & Cohen, 1990), and when images are used that consist of a few illuminated dots pasted onto a darkened moving facial display (Rosenblum &…”
BackgroundA taxonomy of Multimodal Interaction in the Human Information Processing System
Schomaker, L. (1995) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 72 · see citing works
Encoding and memory for visible speech and gestures: A comparison between young and older adults.
Thompson, L. A. (1995) · Psychology and Aging · doi:10.1037//0882-7974.10.2.215 · Cited by 70 · see citing works
Audiovisual speech synthesis: An overview of the state-of-the-art
Mattheyses, W., & Verhelst, W. (2014) · Speech Communication · doi:10.1016/j.specom.2014.11.001 · Cited by 66 · see citing works
“Although the auditory speech mode can be considered as the primary communication channel, receiving also the visual speech information helps to better understand the message [1][2][3][4][5][6], especially in conditions where the auditory speech is degraded by noise”
BackgroundMethodCohen, M. M., & Massaro, D. W. (1990) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203157 · Cited by 66 · see citing works
“Experiments have shown that this animated visible speech influences speech perception (Massaro & Cohen, 1990).”
BackgroundSelf-citationPerceiving asynchronous bimodal speech in consonant-vowel and vowel syllables
Massaro, D. W., & Cohen, M. M. (1993) · Speech Communication · doi:10.1016/0167-6393(93)90064-r · Cited by 61 · see citing works
“Recent research has clarified the processes involved in bimodal speech perception (Massaro, 1987, 1989; Massaro and Cohen, 1990).”
BackgroundSelf-citationEffects of Distance on Visual and Audiovisual Speech Recognition
Jordan, T. R., & Sergeant, P. (2000) · Language and Speech · doi:10.1177/00238309000430010401 · Cited by 58 · see citing works
“…and shape of mouth opening, teeth visibility, and position of the tongue (all of which could be critical features for visual speech perception; e.g., Massaro & Cohen, 1990; Montgomery & Jackson, 1983; Petajan, 1984; Summerfield, 1987, 1992; Summerfield & McGrath, 1984) may be accurately encoded.”
BackgroundAuditory, visual and audiovisual clear speech
Gagné, J., Rochette, A.-J., & Charest, M. (2002) · Speech Communication · doi:10.1016/s0167-6393(01)00012-7 · Cited by 56 · see citing works
Long-term training, transfer, and retention in learning to lipread
Massaro, D. W., Cohen, M. M., & Gesi, A. T. (1993) · Perception & Psychophysics · doi:10.3758/bf03205203 · Cited by 51 · see citing works
Self-citationPerceiving speech from inverted faces
Massaro, D. W., & Cohen, M. M. (1996) · Perception & Psychophysics · doi:10.3758/bf03206832 · Cited by 50 · see citing works
Self-citationPerception of Visible Speech: Influence of Spatial Quantization
Campbell, C. S., & Massaro, D. W. (1997) · Perception · doi:10.1068/p260627 · Cited by 46 · see citing works
Self-citationThe Paradigm Shift to Multimodality in Contemporary Computer Interfaces
Oviatt, S. L., & Cohen, P. R. (2015) · Synthesis lectures on human-centered informatics · doi:10.2200/s00636ed1v01y201503hci030 · Cited by 58 · see citing works
Integration of orthographic and semantic information in memory retrieval.
Massaro, D. W., Weldon, M. S., & Kitzis, S. N. (1991) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.17.2.277 · Cited by 39 · see citing works
Self-citationDiscrimination tests of visually influenced syllables
Rosenblum, L. D., & Saldaña, H. M. (1992) · Perception & Psychophysics · doi:10.3758/bf03206706 · Cited by 37 · see citing works
Visual and audiovisual speech perception with color and gray-scale facial images
Jordan, T. R., Mccotter, M. V., & Thomas, S. M. (2000) · Perception & Psychophysics · doi:10.3758/bf03212141 · Cited by 35 · see citing works
Cross-linguistic comparisons in the integration of visual and auditory speech
Massaro, D. W., Cohen, M. M., & Smeele, P. M. T. (1995) · Memory & Cognition · doi:10.3758/bf03210561 · Cited by 35 · see citing works
Self-citationMore than meets the eye: Context effects in word identification
Masson, M. E. J., & Borowsky, R. (1998) · Memory & Cognition · doi:10.3758/bf03201198 · Cited by 33 · see citing works
“For example, hearing the syllable /ba / while watching a speaker enunciate the syllable /ga / leads an observer to experience yet a different syllable, such as /da/ (see, e.g., Massaro & Cohen, 1990; McGurk & MacDonald, 1976).”
BackgroundMethodVisual Word Recognition in Two Facial Motion Conditions: Full-Face Versus Lips-Plus-Mandible
Marassa, L. K., & Lansing, C. R. (1995) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3806.1387 · Cited by 32 · see citing works
Effects of horizontal viewing angle on visual and audiovisual speech recognition.
Jordan, T. R., & Thomas, S. M. (2001) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.27.6.1386 · Cited by 32 · see citing works
“However, one source may be the size and shape of lip separation (e.g., Jordan & Sergeant, 1998, 2000; Massaro & Cohen, 1990, 1996; Montgomery & Jackson, 1983; Munhall & Vatikiotis-Bat eson, 1998; Petajan, 1984; Summerfield, 1987, 1992; Summerfield & McGrath, 1984).”
Limitations on infants' response to rate-based auditory-visual relations.
Lewkowicz, D. J. (1994) · Developmental Psychology · doi:10.1037/0012-1649.30.6.880 · Cited by 32 · see citing works
“Without a doubt, temporal attributes such as synchrony, duration, rate, and rhythmic pattern are the most frequent and important sources of auditory-visual relational information ( Massaro & Cohen, 1990; Summerfield, 1979).”
BackgroundCrawcour, S. C., Bowers, A. L., Harkrider, A. W., & Saltuklaroglu, T. (2009) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2009.05.001 · Cited by 31 · see citing works
Lewkowicz, D. J. (1998) · Developmental Psychobiology · doi:10.1002/(sici)1098-2302(199805)32:4 · Cited by 31 · see citing works
Independence of lexical context and phonological information in speech perception.
Massaro, D. W., & Oden, G. C. (1995) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.21.4.1053 · Cited by 31 · see citing works
Self-citationResolving ambiguous behavioral intentions by means of involuntary prioritization of gaze processing.
Hudson, M., & Jellema, T. (2011) · Emotion · doi:10.1037/a0023264 · Cited by 28 · see citing works
“The second model is the Fuzzy Logic Model of Perception (FLMP; Massaro & Cohen, 1990, 1993).”
A pattern recognition account of decision making
Massaro, D. W. (1994) · Memory & Cognition · doi:10.3758/bf03198400 · Cited by 27 · see citing works
“For example, deciding between Linda versus Joan is exactly analogous to choosing between /ba/ and /da/ in a speech recognition task given auditory and visual sources of information ( Massaro & Cohen, 1990 ).”
MethodSelf-citationContrast Sensitivity in a Dynamic Environment: Effects of Target Conditions and Visual Impairment
Long, G. M., & Zavod, M. J. (2002) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720024494784 · Cited by 25 · see citing works
Development and experimentation with synthetic visible speech
Cohen, M. M., & Massaro, D. W. (1994) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204633 · Cited by 25 · see citing works
Self-citationSpeech intelligibility of virtual humans
Devesse, A., Dudek, A., Wieringen, A. V., & Wouters, J. M. (2018) · International Journal of Audiology · doi:10.1080/14992027.2018.1511922 · Cited by 22 · see citing works
“…humans in which the visual speech gestures are simulated by directly controlling the facial parameters that are linked to visual articulators (Massaro and Cohen 1990; Salvi et al. 2009), whereas in anatomy-based systems the deformations are the result of an indirect modelling of the facial…”
BackgroundLexical Mediation between Sight and Sound in Speechreading
Repp, B. H., Frost, R., & Zsiga, E. C. (1992) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640749208401313 · Cited by 22 · see citing works
“This interaction even takes place prior to the categorization of the phonemes involved (Massaro & Cohen, 1990; Summerfield, 1987).”
BackgroundDoes audiovisual speech perception use information about facial configuration?
Hietanen, J. K., Manninen, P., Sams, M. E., & Surakka, V. (2001) · The European Journal of Cognitive Psychology · doi:10.1080/09541440126006 · Cited by 22 · see citing works
“These results add to the experimental evidence of the robustness of the McGurk eVect (Green, Kuhl, MeltzoV, & Stevens, 1991; Jordan & Bevan, 1997; Massaro & Cohen, 1990; Rosenblum & Saldaña, 1996; Rosenblum et al., in press; Sams et al., 1991; Walker et al., 1995) and shows that a normal face con…”
ResultThézé, R., Gadiri, M. A., Albert, L., Provost, A., Giraud, A., & Mégevand, P. (2020) · Scientific Reports · doi:10.1038/s41598-020-72375-y · Cited by 21 · see citing works
Frame rate and viseme analysis for multimedia applications
Williams, J. J., Rutledge, J. C., Garstecki, D. C., & Katsaggelos, A. K. (2002) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.1997.602606 · Cited by 20 · see citing works
“Massaro and Cohen [5] state that watching the speaker's face and lips provides important information for speech perception and language understanding.”
BackgroundWu, D. (2012) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee.2012.6250779 · Cited by 19 · see citing works
“Ellison and Massaro [14] had performed experiments to verify that the fuzzy logical model of perception (FLMP) [36] fit the human judgments significantly better than an additive model, and they questioned previous claims of categorical and holistic perception of affect.”
BackgroundCan you McGurk yourself? Self-face and self-voice in audiovisual speech
Aruffo, C., & Shore, D. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0176-8 · Cited by 19 · see citing works
Multimodal Speech Perception: A Paradigm for Speech Science
Massaro, D. W. (2002) · Text, speech and language technology · doi:10.1007/978-94-017-2367-1_4 · Cited by 19 · see citing works
Self-citationThe role of pitch and temporal diversity in the perception and production of musical sequences
Prince, J. B., & Pfordresher, P. Q. (2012) · Acta Psychologica · doi:10.1016/j.actpsy.2012.07.013 · Cited by 18 · see citing works
Crossmodal Interaction in Humans
Watanabe, K. (2010) · CaltechTHESIS (California Institute of Technology) · doi:10.7907/66cn-aq26 · Cited by 15 · see citing works
Multimodal interaction, interfaces, and analytics
Oviatt, S. (2022) · Handbook of Human Computer Interaction · doi:10.1007/978-3-319-27648-9_22-1 · Cited by 12 · see citing works
Speech-Driven Facial Animations Improve Speech-in-Noise Comprehension of Humans
Varano, E., Vougioukas, K., Ma, P., Petridis, S., Pantić, M., & Reichenbach, T. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.781196 · Cited by 14 · see citing works
“These early models yielded animations capable of augmenting speech comprehension in background noise (Massaro and Cohen, 1990; Le Goff et al., 1997; Munhall et al., 2004) but required the previous or simultaneous recording of a human speaker wearing facial markers or myographic electrodes (Bailly et al.”
BackgroundMethodA Survey of Computer Facial Animation Techniques
Li, L., Liu, Y., & Zhang, H. (2012) · International Conference on Computer Science and Electronics Engineering (ICCSEE) · doi:10.1109/iccsee.2012.129 · Cited by 14 · see citing works
“They later extended the model to include a tongue and to model co-articulation effects [18,19], in which they give the examples of the articulation of the final consonant in “boot” and “beet” - backward coarticulation - and the way the lips round at the beginning of “stew” in anticipation of the “t”.”
BackgroundFrame Rate and Viseme Analysis for Multimedia Applications to Assist Speechreading
Williams, J. J., Rutledge, J. C., Katsaggelos, A. K., & Garstecki, D. C. (1998) · The Journal of VLSI Signal Processing Systems for Signal Image and Video Technology · doi:10.1023/a:1008062122135 · Cited by 14 · see citing works
Perceptual recognition of facial affect: Cross-cultural comparisons
Massaro, D. W., & Ellison, J. W. (1996) · Memory & Cognition · doi:10.3758/bf03201104 · Cited by 14 · see citing works
Self-citationFadiga, L., Craighero, L., Buccino, G., Rizzolatti, G., & Biomediche, S. (2002) · Cited by 13 · see citing works
Manning, S. K., & Shofner, E. (1991) · The American Journal of Psychology · doi:10.2307/1422942 · Cited by 11 · see citing works
Exploiting Visual Information in Automatic Speech Processing
Aleksic, P. S., Potamianos, G., & Katsaggelos, A. K. (2005) · Elsevier eBooks · doi:10.1016/b978-012119792-6/50134-0 · Cited by 10 · see citing works
“For example, in [ 60 ], Cohen and Massaro used Parke’s facial model as the basis for their text-driven speech synthesizer.”
MethodSchwartz, J. (2003) · AVSP · Cited by 11 · see citing works
Causes and Reasons in Failures to Perceive Fearful Faces
de Bonis, M. (2003) · Springer eBooks · doi:10.1007/978-1-4615-1063-5_8 · Cited by 10 · see citing works
Djilali, Y. A. D., Narayan, S., Boussaid, H., Almazrouei, E., & Debbah, M. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01268 · Cited by 10 · see citing works
“[56, 37, 6].”
Evidence for a Generic Process Underlying Multisensory Integration
Tremblay, C., Champoux, F., Bacon, B. A., & Théoret, H. (2007) · The Open Behavioral Science Journal · doi:10.2174/1874230000701010001 · Cited by 9 · see citing works
“Subsequent studies have confirmed that the McGurk effect is a very robust illusion [2] [3].”
Speaker Variations Influence Speechreading Speed for Dynamic Faces
Kaufmann, J. M., & Schweinberger, S. R. (2005) · Perception · doi:10.1068/p5104 · Cited by 9 · see citing works
Aleksic, P. S., Potamianos, G., & Katsaggelos, A. K. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374456-2.00024-4 · Cited by 9 · see citing works
Why Complex Signals Matter, Sometimes
Rubi, T. L., & Stephens, D. W. (2016) · Animal signals and communication · doi:10.1007/978-3-319-48690-1_5 · Cited by 8 · see citing works
The Role of Facial Colour and Luminance in Visual and Audiovisual Speech Perception
Mccotter, M. V., & Jordan, T. R. (2003) · Perception · doi:10.1068/p3316 · Cited by 7 · see citing works
Visible speech improves human language understanding: Implications for speech processing systems
Thompson, L. A., & Ogden, W. C. (1995) · Artificial Intelligence Review · doi:10.1007/bf00849044 · Cited by 7 · see citing works
Effects of horizontal viewing angle on visual and audiovisual speech perception
Jordan, T. R., Sergeant, P., Martin, C., Thomas, S. M., & Thow, E. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1997.638235 · Cited by 6 · see citing works
EFFECTS OF MOUTHING AND INTERLOCUTOR PRESENCE ON MOVEMENTS OF VISIBLE VS. NON-VISIBLE ARTICULATORS.
Bicevskis, K., de Vries, J., Green, L., Heim, J., Božič, J., D'Aquisto, J., Fry, M., Sadlier-Brown, E., Tkachman, O., Yamane, N., et al. (2016) · Canadian acoustics · Cited by 6 · see citing works
Effects of exhaustivity and uncertainty on audiovisual focus production.
Wollermann, C., & Schröder, B. (2009) · AVSP · Cited by 5 · see citing works
Spatial and temporal visual speech feature for Chinese phonemes
Jia, X., Han, Y., Powers, D. M. W., & -, B. X. (2012) · Journal of Information and Computational Science · Cited by 3 · see citing works
Audio-driven Facial Animation: A Survey
Hwang, J., & Park, K. (2022) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc55196.2022.9952573 · Cited by 3 · see citing works
“They later extended the model to include a tongue and to model co-articulation effects [9], [10]; co-articulation is a complex behavior of articulation of multiple viseme overlapping and affecting each other.”
BackgroundA Kind of Visual Speech Feature with the Geometric and Local Inner Texture Description
Jia, X., & Sun, Y. (2013) · TELKOMNIKA Indonesian Journal of Electrical Engineering · doi:10.11591/telkomnika.v11i2.2047 · Cited by 2 · see citing works
“Massaro et al. [15] have suggested that degree of lip opening could be a critical visual feature which acts as input to the audiovisual integration process.”
BackgroundSpeech synchronization between speech and lip shape movements for service robotics applications
Luo, R. C., Chien-Chieh, H., Chang, S.-R., & Tsai, Y.-J. (2011) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon.2011.6119661 · Cited by 2 · see citing works
“Cohen and Massaro [6] [7] proposed a lip-sync system with parametric model and studied the context of speech perception in 1990.”
MethodA New Test for Measuring Dynamic Contrast Sensitivity (DCS)
Zavod, M. J., & Long, G. M. (2001) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120104501825 · Cited by 2 · see citing works
Impairment of speech-reading in prosopagnosia
de Gelder, B., & Vroomen, J. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00052-1 · Cited by 2 · see citing works
Wannagat, W., Huestegge, L., Landmann, E., Nieding, G., & Huestegge, S. M. (2022) · Cognitive Development · doi:10.1016/j.cogdev.2022.101263 · Cited by 2 · see citing works
Auditory/visual speech in multimodal human interfaces
Massaro, D. W., & Cohen, M. M. (1994) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1994-135 · Cited by 2 · see citing works
“Top-down sources include contextual, semantic, syntactic, and phonological constraints; bottom-up sources include audible and visible features of the spoken word.”
BackgroundSelf-citationvon Steinbüchel, N., Steffen, A., & Wittmann, M. (1997) · Springer eBooks · doi:10.1007/978-3-662-40459-1_16 · Cited by 1 · see citing works
Learning effects in multimodal perception with real and simulated faces
Keough, M., Derrick, D., Rc, T., & Gick, B. (2019) · University of Canterbury Research Repository (University of Canterbury) · Cited by 1 · see citing works
Hearing Lips and Seeing Voices: Illusion and Serendipity in Auditory‐Visual Perception Research
Macdonald, J. B. (2006) · doi:10.1002/9780470777930.ch6 · Cited by 1 · see citing works
Johnson, J. A., Rosenblum, L. D., & Schmuckler, M. A. (1995) · The Journal of the Acoustical Society of America · doi:10.1121/1.411551 · Cited by 1 · see citing works
“The effect works with both elaborate and schematic synthetic visual stimuli (e.g., Massaro and Cohen, 1990; Summerfield, Macleod, McGrath, and Brooke, 1989).”
BackgroundNeural Tracking of Audiovisual Effects in Noise Using Deep Neural Network-Generated Virtual HumansPreprint
Cooper, J. K., Vanthornhout, J., Wieringen, A. V., & Francart, T. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.02.656280 · Cited by 1 · see citing works
Predicting Visual Intelligibility Gain in the SynFace Application
Hagström, P. (2009)
Facial Animation for Visual Speech
Guiard‐Marigny, T. (2009) · doi:10.1002/9780470611180.ch4
A Visual Speech Feature to Indentify the Speaking States from Video
Jia, X., Yin, B., & Sun, Y. (2010) · International Conference on Multimedia Technology (ICMT) · doi:10.1109/icmult.2010.5629829
Facial expressions of emotion : influences of configuration
Cook, F. L. (2007) · OpenGrey (Institut de l'Information Scientifique et Technique)
Albasri, S. (2019) · ORCA Online Research @Cardiff (Cardiff University)
Visible Speech Improves Human Language Understanding: Implications for Speech Processing Systems
Thompson, L. A., & Ogden, W. C. (1995) · Springer eBooks · doi:10.1007/978-94-009-1639-5_8
Improving Naturalness in Speech Synthesis Using Fuzzy Logic
Shah, B. G., & Sajja, S. P. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-0769-4_22
- Authors
- Massaro, D. W., & Cohen, M. M.
- Affiliations
- University of California, Santa Cruz (US)
- ORCID
- Massaro, Dominic W.
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 1990-01
- Volume
- 1
- Issue
- 1
- Pages
- 55-63
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2012077291
- Cited by
- 94
- References
- 28
- Citations by year
- 2025: 1 · 2023: 2 · 2022: 4 · 2020: 2 · 2019: 2 · 2018: 1 · 2016: 2 · 2015: 1 · 2014: 1 · 2013: 1 · 2012: 4
- FWCI
- 5.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Science (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech perception, synthetic speech, lip reading, audiovisual integration, categorical perception, syllable onset, hearing-impaired listeners
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
