Intermodal timing relations and audio-visual speech recognition by normal-hearing adults

“The effect was biggest when auditory vowels were synchronized with the original mouth movements (McGrath & Summerfield, 1985), but the effect survives, even if audition lagged vision by 180 msec (see also Soto-Faraco & Alsius, 2007, 2009; these studies show that participants can still perceive a…”

“4 ms, well below the 80 ms range required for observers to detect an audio-visual asynchrony (McGrath and Summerfield, 1985).”

“Within this window, the combination of information from two modalities results in significant changes in neural, behavioral and perceptual responses (Dixon and Spitz, 1980; McGrath and Summerfield, 1985; Meredith et al., 1987; Lewkowicz, 1996; Colonius and Diederich, 2004).”

“This particular range of SOAs was selected on the basis of previous research showing that people can typically discriminate the temporal order of briefly presented, matched auditory and visual speech stimuli at a 75% correct level (i.e., the conventionally defined JND; Spence et al., 2001) at SOAs of approximately 80…”

“Precise timing of miming was achieved by a system of computer-controlled ''ready—go'' red and green lights controlled by the microprocessor (see Grant, van Wassenhove, & Poeppel, 2003; McGrath & Summerfield, 1985, regarding perceptual limits for temporal asynchrony in auditory–visual dubbing).”

“For example, researchers working on the design of hearing aids (McGrath and Summerfield 1985; Pandev et al. 1986) have to tradeoff the potential benefits of increasing the quality of auditory signal processing provided by the hearing aid against the increased temporal delays (and consequent increase in the asynchrony…”

“Temporal discrepancy between auditory and visual speech signals also affects one’s percept of unity [70].”

“Previous research on AV synchrony detection (Dixon and Spitz, 1980; Lewkowicz, 1996; Massaro and Cohen, 1993; Massaro et al., 1996; McGrath and Summerfield, 1985; Pandey et al., 1986) and pilot studies in our lab indicated that most normal-hearing young adult participants were able to judge AV…”

“One interesting feature of this temporal binding window in adults is that it differs depending on whether the visual or auditory stimuli are leading or lagging (e.g. Dixon & Spitz, 1980; McGrath & Summerfield, 1985; Powers et al., 2009).”

“…documenting the conditions under which these effects hold (e.g., Green, in press; Green & Gerdeman, 1995; Green, Kuhl, & Meltzoff, 1988; Green, Kuhl, Meltzoff, & Stevens, 1991; McGrath & Summerfield, 1985; Saldaña, Rosenblum, & Osinga, 1992; Summerfield, 1979; Werker, Frost, & McGurk 1992).”

“The drop in perceived synchrony was greater with V-A than with A-V stimulus presentations, replicating earlier studies of asynchrony detection (Conrey and Pisoni, 2006; Conrey and Pisoni, 2004; Dixon and Spitz, 1980; Grant and Greenberg, 2001; Grant et al., 2004; McGrath and Summerfield, 1985; Miller and D'Esposito, 2…”

“Reinforcing this, [119] proposed a lip-reading task and showed that the performance of users dropped significantly when the soundtrack was delayed by 160ms.”

“Within each of these areas, research has examined various aspects of the temporal synchronization patterns that characterize multimodal integration [5, 6, 12].”

“At this point, it is important to note that perceptual thresholds for detecting asynchrony in audiovisual speech appear to be relatively large at around 260 ms when the voice is delayed relative to the video and around 130 ms when the voice is advanced relative to the video (Munhall & Vatikiotis-Bateson, 2004 )—althou…”

“…to provide a more sensitive description of the limits of cross-modal temporal synchrony than those derived from speech identification experiments (McGrath and Summerfield, 1985; Pandey et al., 1986; Massaro et al., 1996; Silipo et al., 1999; Grant and Greenberg , 2001; van Wassenhove et al.,…”

“Not surprisingly, empirical evidence indicates that adults are sensitive to these kinds of intersensory temporal synchrony relations (Dixon & Spitz, 1980; Massaro, 1998; McGrath & Summerfield, 1985; Radeau, 1994).”

“A constant delay across frequency should not affect perception of a speech source external to the listener, except for rather long delays (above 40 msec), when lipreading may be affected (McGrath & Summerfield, 1985).”

“…a host of perceptual and behavioral benefits (Calvert et al., 2004; Murray & Wallace, 2012; Stevenson et al., 2014a; Noel & Wallace, 2016), such as enhanced speech intelligibility (Sumby & Pollack, 1954; McGrath & Summerfield, 1985; Rosenblum et al., 1996; Schwartz et al., 2004; Ma et al., 2009).”

“The literature on lip-reading (McGrath & Summerfield, 1985) suggests a limit of a 40 msec; beyond that audio-visual integration may be disrupted.”

“Additionally, it is interesting to notice that the perceived best alignment between the visual and auditory stimuli (PSS) occurred when the audio was delayed with respect to the video (Dixon & Spitz, 1980; McGrath & Summerfield, 1985; Arrighi et al., 2006) in all the display conditions but one.”

“(Dixon & Spitz, 1980; Massaro, 1998; McGrath & Summerfield, 1985; Radeau, 1994) and infants (Bahrick, 1992; Lewkowicz, 1992a, 1996b; Spelke, Born, & Chu, 1983; Walker, 1982) has shown that temporal AV relations contribute in important ways to intersensory integration.”

“Audio-visual delays of up to 80ms turned out to have little effect on sentence identification (McGrath and Summerfield, 1985).”

“the visible lip movements, the visual stimulus must lead the auditory stimulus by 140 ms more (McGrath and SummerWeld 1985).”

“In contrast, research on multimodal speech and lip movements has been driven largely by cognitive science interest in intersensory audio-visual perception, and the coordination of speech output with lip and facial movements [5,7,27-36].”

“McGrath and Summerfield (1985) added a soundtrack of rectangular pulses synchronized to the closing of the talker's vocal track.”

“…relatively large temporal asynchronies in the auditory and visual stimuli of congruent speech (Dixon & Spitz, 1980; Massaro, Cohen, & Smeele, 1996; McGrath & Summerfield, 1985) and McGurk stimuli (Kösem & van Wassenhove, 2012; Munhall et al., 1996) may still be perceived as synchronous, and…”

“This finding is contrary to the conclusion reached by McGrath and Summerfield (1985), who argued that beyond minimal levels of intelligence and verbal ability, no other abilities were important for lipreading. It is also contrary to the view expressed by Summerfield (1992) in a review of the literature on lipreading a…”

“…2001; Vatakis and Spence, 2006a) becoming broader/wider when stimuli are more complex (consisting of semantic content; Dixon and Spitz, 1980; McGrath and Summerfield, 1985; Spence and Squire, 2003; Miller and D’Esposito, 2005; Vatakis and Spence, 2006b, see also Vroomen and Keetels, 2010…”

“Further, in multisensory speech identification, a temporal separation of more than 150 ms generally has little or no audio-visual benefit behaviorally when compared with visual alone (McGrath and Summerfield, 1985), whereas there is a clear benefit at SOAs of less than this separation.”

“The practical implication of the results is that delays in audio or video signals of communication systems would affect hearing-impaired listeners in a manner similar to normal-hearing listeners, and due to the importance of visual cues for the hearingimpaired listeners, special attention should be…”

“... on the multisensory perception of synchrony under more realistic conditions, however, one has to move away from the study of simple transitory stimuli of low informational content (e.g., Hirsh and Sherrick, 1961; Zampini et al., 2003b) toward the use of more ecologically valid and complex stimuli (such as speech,…”

“Several studies (e.g., McGrath and Summerfield 1985; Munhall et al. 1996; van Wassenhove et al. 2007) have shown that audiovisual integration in identification tasks is fairly robust to audiovisual asynchrony.”

“Within the hearing population, the examination of multimodal speech has revealed that speech understanding can occur with degraded asynchronous audiovisual (AV) speech (McGrath & Summerfield, 1985; Grant & Greenberg, 2001; Pandey et al, 1986).”

“In addition, that visual cues have been shown to be beneficial even when speech was clearly presented (Jesse & Massaro, 2010; McGrath & Summerfield, 1985) seems to provide evidence that multimodal perception is the primary mode of speech perception, whereas auditory-only speech perception is an…”

“…and visual stimuli is somewhat different from other multisensory studies, whether behavioral (Colonius and Diederich, 2004; Dixon and Spitz, 1980; McGrath and Summerfield, 1985; Powers et al., 2009) or imaging (Dhamala et al., 2007a; Noesselt et al., 2007; van Atteveldt et al., 2007a; Watkins et…”

“Soc modality when that information is coupled with information via another modality (e.g., McGrath and Summerfield, 1985).”

“However, in order for face and voice to be perceived coherently, the acoustic signal also has to be delayed by the same amount (McGrath and Summerfield, 1985).”

“…on the impact of temporal synchrony on complex2 audiovisual integration (Vatakis & Spence, 2006a, 2006c, 2008; van Wassenhove et al., 2007; Grant, van Wassenhove, & Poeppel, 2004; Massaro, Cohen, & Smeele, 1996; Munhall et al., 1996; Steinmetz, 1996; Rihs, 1995; McGrath & Summerfield, 1985).”

“According to McGrath and Summerfield (1985), delays on the order of 80 msec or more are likely to affect AV performance for average speechreaders, and delays as small as 40 msec can affect AV performance for more experienced speechreaders.”

“From psychophysics it is known that relative shifts between audio and visual signals that are smaller than the duration of a video frame are essentially imperceptible and do not seem to affect audiovisual integration [4, 50 , 51].”

“…stable when the auditory and the visual cues are separated temporally (up to 180ms) or spatially (Jones & Munhall, 1997; Massaro & Cohen, 1993; McGrath & Summerfield, 1985; Munhall, Gribble, Sacco, & Ward, 1996), and when subjects do not fixate the speaker s lips (Pare, Richler, ten Hove, &…”

“…of several tens to several hundreds of milliseconds (Bushara, Grafman, & Hallett, 2001; Dixon & Spitz, 1980; Grant, van Wassenhove, & Poeppel, 2004; McGrath & Summerfield, 1985; Petrini, Dahl, et al., 2009; Petrini, Halt, & Pollick, 2010; Petrini, Russell, & Pollick, 2009; Van Wassenhove, Grant, &…”

“Introduction Current knowledge about multisensory processes and percepts stems from a long line of well-designed and well-controlled research studies (McGrath and Summerfield, 1985; Lewkowicz, 1996; Stone et al., 2001; Zampini et al., 2003; Fujisaki and Nishida, 2005; Zampini et al., 2005).”

“In the multimodal literature on speech and lip movements, similar robustness advantages have been documented when humans process dual input signals ([3], [12], [13], [28]), and more recently such robustness advantages also have been demonstrated for multimodal systems ([1], [29]).”

“The McGurk effect is a remarkably robust phenomenon that can be resistant to such manipulations as temporal asynchrony [51] and gender mismatch [52] of the visual and auditory source components, and can occur even when participants are aware of the dubbing procedure [49].”

“A Low-Latency Lip-Synchronized Videoconferencing System Milton Chen Interactivity Laboratory and Computer Graphics Laboratory Stanford University miltchen@graphics.stanford.edu”

“2003), and also on the role of temporal synchrony, when the audio and video stimuli are congruent but temporally misaligned (Dixon and Spitz 1980; McGrath and Summerfield 1985).”