The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.
Multimodal Interfaces
Abstract
This chapter reviews the main types of multimodal interfaces, their advantages and cognitive science underpinnings, primary features and architectural characteristics, and general research in the field of multimodal interaction and interface design. Multimodal systems have developed rapidly during the past decade, with steady progress toward building more general and robust systems, as well as more transparent human interfaces than ever before. Major developments have occurred in the hardware and software needed to support key component technologies incorporated within multimodal systems, as well as in techniques for integrating parallel input streams. Current natural language-processing algorithms typically rely heavily on the specification of determiners in definite and indefinite references in order to represent and resolve noun phrase reference. Over the past decade, numerous advantages of multimodal interface design have been documented. Unlike a traditional keyboard-and-mouse interface or a unimodal recognition-based interface, multimodal interfaces permit flexible use of input modes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Baddeley, A. (1992) · Science · doi:10.1126/science.1736359 · Cited by 5,258
Hand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583
Merging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906
Embodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Visual interpretation of hand gestures for human-computer interaction: a review
Pavlovic, V. I., Sharma, R., & Huang, T. S. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598226 · Cited by 1,791
“Put-that-there”: Voice and gesture at the graphics interface
Bolt, R. A. (1980) · Proceedings of the 7th annual conference on Computer graphics and interactive techniques - SIGGRAPH '80 · doi:10.1145/800250.807503 · Cited by 1,445
Human facial expression: an evolutionary view
(1995) · Choice Reviews Online · doi:10.5860/choice.32-3890 · Cited by 1,279
McNeill, D. (2011) · Advances in Visual Semiotics · doi:10.1515/9783110874259.351 · Cited by 927
“Put-that-there”: Voice and gesture at the graphics interface
Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921
Facial Expressions of Emotion: New Findings, New Questions
Ekman, P. (1992) · Psychological Science · doi:10.1111/j.1467-9280.1992.tb00253.x · Cited by 917
Wizard of Oz studies: why and how
Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993) · Proceedings of the 1st international conference on Intelligent user interfaces - IUI '93 · doi:10.1145/169891.169968 · Cited by 742
Ten myths of multimodal interaction
Oviatt, S. (1999) · Communications of the ACM · doi:10.1145/319382.319398 · Cited by 689
Self-citationManual and gaze input cascaded (MAGIC) pointing
Zhai, S., Morimoto, C., & Ihde, S. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303053 · Cited by 680
The logic of typed feature structures
Carpenter, B. (1992) · Cited by 600
Audio-visual speech modeling for continuous speech recognition
Dupont, S., & Luettin, J. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865479 · Cited by 598
The open agent architecture: A framework for building distributed software systems
Martin, D. L., Cheyer, A. J., & Moran, D. B. (1999) · Applied Artificial Intelligence · doi:10.1080/088395199117504 · Cited by 492
QuickSet: multimodal interaction for distributed applications
Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., & Clow, J. (1997) · Proceedings of the fifth ACM international conference on Multimedia - MULTIMEDIA '97 · doi:10.1145/266180.266328 · Cited by 466
Self-citationQuantifying the contribution of vision to speech perception in noise
Macleod, A., & Summerfield, Q. (1987) · British Journal of Audiology · doi:10.3109/03005368709077786 · Cited by 426
Automatic lipreading to enhance speech recognition (speech reading)
Petajan, E. D. (1984) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 420
Compatibility and Resource Competition between Modalities of Input, Central Processing, and Output
Wickens, C. D., Sandry, D. L., & Vidulich, M. (1983) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872088302500209 · Cited by 402
Integration and synchronization of input modes during multimodal human-computer interaction
Oviatt, S., DeAngeli, A., & Kuhn, K. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258821 · Cited by 360
Self-citationLipreading and audio-visual speech perception
Summerfield, Q. (1992) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1992.0009 · Cited by 358
Mutual disambiguation of recognition errors in a multimodel architecture
Oviatt, S. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303163 · Cited by 315
Self-citationHuman-centered design meets cognitive load theory: designing interfaces that help people think
Oviatt, S. (2006) · Proceedings of the 14th ACM international conference on Multimedia · doi:10.1145/1180639.1180831 · Cited by 315
Self-citation"Eigenlips" for robust speech recognition
Bregler, C., & Konig, Y. (2002) · Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.1994.389567 · Cited by 312
Toward multimodal human-computer interface
Sharma, R., Pavlovic, V. I., & Huang, T. S. (1998) · Proceedings of the IEEE · doi:10.1109/5.664275 · Cited by 302
Taming recognition errors with a multimodal interface
Oviatt, S. (2000) · Communications of the ACM · doi:10.1145/348941.348979 · Cited by 299
Self-citationWhen do we interact multimodally?: cognitive load and multimodal communication patterns
Oviatt, S., Coulston, R., & Lunsford, R. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027957 · Cited by 285
Self-citationPerceptual user interfaces: multimodal interfaces that process what comes naturally
Oviatt, S., & Cohen, P. (2000) · Communications of the ACM · doi:10.1145/330534.330538 · Cited by 278
Self-citationSekiyama, K., & Tohkura, Y. (1991) · The Journal of the Acoustical Society of America · doi:10.1121/1.401660 · Cited by 263
Speechreading by Humans and Machines: Models, Systems, and Applications
(1996) · NATO ASI Series · doi:10.1007/978-3-662-13015-5 · Cited by 226
The role of voice input for human-machine communication.
Cohen, P. R., & Oviatt, S. L. (1995) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.92.22.9921 · Cited by 203
Self-citationIntermodal timing relations and audio-visual speech recognition by normal-hearing adults
McGrath, M., & Summerfield, Q. (1985) · The Journal of the Acoustical Society of America · doi:10.1121/1.392336 · Cited by 201
Multimodal integration-a statistical view
Wu, L., Oviatt, S. L., & Cohen, P. R. (1999) · IEEE Transactions on Multimedia · doi:10.1109/6046.807953 · Cited by 190
Self-citationBrandl, P., Forlines, C., Wigdor, D., Haller, M., & Shen, C. (2008) · Working Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/1385569.1385595 · Cited by 180
Speech and gestures for graphic image manipulation
Hauptmann, A. G. (1989) · Proceedings of the SIGCHI conference on Human factors in computing systems Wings for the mind - CHI '89 · doi:10.1145/67449.67496 · Cited by 168
Perceptual user interfaces (introduction)
Turk, M., & Robertson, G. (2000) · Communications of the ACM · doi:10.1145/330534.330535 · Cited by 163
Synergistic use of direct manipulation and natural language
Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. N., & Sullivan, J. (1989) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/67449.67494 · Cited by 143
An Analysis of Behavioral Organization
Condon, W. S. (1976) · Sign language studies · doi:10.1353/sls.1976.0001 · Cited by 130
Improving connected letter recognition by lipreading
Bregler, C., Hild, H., Manke, S., & Waibel, A. H. (1993) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1993.319179 · Cited by 125
Error resolution during multimodal human-computer interaction
Oviatt, S. L., & VanGent, R. (1996) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607077 · Cited by 122
Self-citationUser and discourse models for multimodal communication
Wahlster, W. (1991) · Intelligent User Interfaces · doi:10.1145/107215.128691 · Cited by 112
Intelligent multi-media interface technology
Neal, J. G., & Shapiro, S. C. (1991) · Intelligent User Interfaces · doi:10.1145/107215.128690 · Cited by 112
Integrating audio and visual information to provide highly robust speech recognition
Tomlinson, M. J., Russell, M. J., & Brooke, N. M. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543247 · Cited by 108
Robert-Ribes, J., Schwartz, J., Lallouache, M.-T., & Escudier, P. (1998) · The Journal of the Acoustical Society of America · doi:10.1121/1.423069 · Cited by 107
Towards a fault-tolerant multi-agent system architecture
Kumar, S., & Cohen, P. R. (2000) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/336595.337570 · Cited by 105
Kinematics of head movements accompanying speech during conversation
Hadar, U., Steiner, T. J., Grant, E. C. G., & Rose, F. C. (1983) · Human Movement Science · doi:10.1016/0167-9457(83)90004-0 · Cited by 103
Perception of Synthesized Audible and Visible Speech
Massaro, D. W., & Cohen, M. M. (1990) · Psychological Science · doi:10.1111/j.1467-9280.1990.tb00068.x · Cited by 94
Multimodal system processing in mobile environments
Oviatt, S. L. (2000) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/354401.354408 · Cited by 92
Self-citationComputer-human interface solutions for emergency medical care
Holzman, T. G. (1999) · interactions · doi:10.1145/301153.301160 · Cited by 83
Linguistic Adaptations During Spoken and Multimodal Error Resolution
Oviatt, S. L., Bernard, J., & Levow, G. (1998) · Language and Speech · doi:10.1177/002383099804100409 · Cited by 80
Self-citationVo, M. T., & Wood, C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.550794 · Cited by 80
Adaptive bimodal sensor fusion for automatic speechreading
Meier, U., Hürst, W., & Duchnowski, P. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543250 · Cited by 77
Oviatt, S. L., Cohen, P. R., & Wang, M. (1994) · Speech Communication · doi:10.1016/0167-6393(94)90079-5 · Cited by 76
Self-citationIndividual differences in multimodal integration patterns: what are they and why do they exist?
Oviatt, S. L., Lunsford, R., & Coulston, R. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055006 · Cited by 74
Self-citationInteractive simulation in a multi-person virtual world
Codella, C. F., Sweeney, P., Turk, G., Jalili, R., Koved, L., Lewis, J. B., Ling, D. T., Lipscomb, J. S., Rabenhorst, D. A., Wang, C. P., et al. (1992) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/142750.142825 · Cited by 72
Integration themes in multimodal human-computer interaction
Oviatt, S. L., & Olsen, E. (1994) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1994-140 · Cited by 69
Self-citationCombining deictic gestures and natural language for referent identification
Kobsa, A., Allgayer, J., Reddig, C., Reithinger, N., Schmauks, D., Harbusch, K., & Wahlster, W. (1986) · International Conference on Computational Linguistics · doi:10.3115/991365.991471 · Cited by 67
Discourse structure and performance efficiency in interactive and non-interactive spoken modalities
Oviatt, S. L., & Cohen, P. R. (1991) · Computer Speech & Language · doi:10.1016/0885-2308(91)90001-7 · Cited by 65
Self-citation“Finger-Pointer”: Pointing interface by image processing
Fukumoto, M., Suenaga, Y., & Mase, K. (1994) · Computers & Graphics · doi:10.1016/0097-8493(94)90157-0 · Cited by 65
Differences in Visual Intelligibility Across Talkers
Kricos, P. B. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_4 · Cited by 61
A rapid semi-automatic simulation technique for investigating interactive speech and handwriting
Oviatt, S. L., Cohen, P. R., Fong, M. W., & Frank, M. (1992) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1992-362 · Cited by 61
Self-citationWhich components of the face do humans and machines best speechread?
Benoı̂t, C., Guiard‐Marigny, T., Goff, B. L., & Adjoudani, A. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_24 · Cited by 58
Benoı̂t, C., & Goff, B. L. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00045-4 · Cited by 57
Adaptive fusion of acoustic and visual sources for automatic speech recognition
Rogozan, A., & Deléglise, P. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00056-9 · Cited by 56
Unification-based multimodal integration
Johnston, M. O., Cohen, P. R., McGee, D. R., Oviatt, S. L., Pittman, J. A., & Smith, I. (1997) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/979617.979653 · Cited by 56
Self-citationAgreement detection in multiparty conversation
Germesin, S., & Wilson, T. (2009) · International Conference on Multimodal Interfaces · doi:10.1145/1647314.1647319 · Cited by 54
Perception of Conflicting Audio-Visual Speech: an Examination across Spanish and German
Fuster-Duran, A. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_9 · Cited by 49
Context based multimodal fusion
Pfleger, N. (2004) · International Conference on Multimodal Interfaces · doi:10.1145/1027933.1027977 · Cited by 47
Speech/gesture interface to a visual computing environment for molecular biologists
Sharma, R., Huang, T. S., Pavlović, V., Zhao, Y., Lo, Z., Chu, S., & Schul, K. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.547311 · Cited by 45
Integration of eye-gaze, voice and manual response in multimodal user interface
Wang, J. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1995.538404 · Cited by 35
For speech perception by humans or machines, three senses are better than one
Bernstein, L. E., & Benoı̂t, C. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607895 · Cited by 34
Toward High-Performance Communications Interfaces for Science Problem Solving
Oviatt, S. L., & Cohen, A. O. (2010) · Journal of Science Education and Technology · doi:10.1007/s10956-010-9218-7 · Cited by 34
Self-citationCheyer, A. J., & Julia, L. E. (1998) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/268389.268399 · Cited by 31
Audio-visual and multimodal speech-based systems
Gibbon, D., Mertins, I., & Moore, R. K. (2000) · Springer eBooks · doi:10.1007/978-1-4615-4501-9_2 · Cited by 29
Toward open-microphone engagement for multiparty interactions
Lunsford, R., Oviatt, S. L., & Arthur, A. M. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181049 · Cited by 27
Self-citationNeti, C., Iyengar, G., Potamianos, G., et al. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-466 · Cited by 25
Referential features and linguistic indirection in multimodal language
Oviatt, S. L., & Kuhn, K. (1998) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1998-277 · Cited by 25
Self-citationMultimodal interaction in speech systems
Rudnicky, A. I., & Hauptmann, A. G. (1992) · doi:10.1145/146022.146049 · Cited by 23
Integrating multimodal language processing with speech recognition
Bangalore, S., & Johnston, M. O. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-225 · Cited by 23
Multimodal Interfaces for Multimedia Information Agents
Waibel, A., Suhm, B., Vo, M. T., & Yang, J. (1999) · Springer eBooks · doi:10.1007/978-3-642-60087-6_35 · Cited by 22
Bimodal Speech Perception: A Progress Report
Massaro, D. W. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_6 · Cited by 21
Integration of audio/visual information for use in human-computer intelligent interaction
Pavlović, V., Berry, G. A., & Huang, T. S. (2002) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.1997.647399 · Cited by 20
Multimodal signal processing in naturalistic noisy environments
Oviatt, S. L. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-364 · Cited by 19
Self-citationAudio-visual cues distinguishing self- from system-directed speech in younger and older adults
Lunsford, R., Oviatt, S. L., & Coulston, R. (2005) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1088463.1088494 · Cited by 18
Self-citationTalking to Digital Fish: Designing Effective Conversational Interfaces for Educational Software
Darves, C., & Oviatt, S. L. (2004) · Human-computer interaction series · doi:10.1007/1-4020-2730-3_10 · Cited by 16
Self-citationUsing the Baby-Babble-Blanket for infants with motor problems
Fell, H. J., Delta, H., Peterson, R., Ferrier, L. J., Mooraj, Z., & Valleau, M. (1994) · doi:10.1145/191028.191049 · Cited by 14
Multimodal Interfaces for Cell Phones and Mobile Technology
Oviatt, S. L., & Lunsford, R. (2005) · International Journal of Speech Technology · doi:10.1007/s10772-005-2164-8 · Cited by 12
Self-citation
Cited by
Augmented reality technologies, systems and applications
Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., & Ivkovic, M. (2011) · Multimedia Tools and Applications · doi:10.1007/s11042-010-0660-6 · Cited by 1,547
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Multimodal human–computer interaction: A survey
Jaimes, A., & Sebe, N. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.019 · Cited by 965
Intelligent multi-camera video surveillance: A review
Wang, X. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.07.005 · Cited by 694
Augmented Reality: An Overview
Carmigniani, J., & Furht, B. (2011) · Handbook of Augmented Reality · doi:10.1007/978-1-4614-0064-6_1 · Cited by 627
Metadata
- Authors
- Oviatt, S.
- Venue
- Human Factors and Ergonomics
- Publisher
- CRC Press
- Published
- 2007-09-19
- Pages
- 413-432
- ISSN
- 2154-6096 (ISSN-L)
- OpenAlex
- W2911240772
- Cited by
- 485
- References
- 91
- FWCI
- 69.94
- Type
- book chapter
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.