The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Toward multimodal human-computer interface
Abstract
Recent advances in various signal processing technologies, coupled with an explosion in the available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities: speech, vision-based gesture recognition, eye tracking, electroencephalograph, etc. Successful embodiment of these modalities into an interface has the potential of easing the HCI bottleneck that has become noticeable with the advances in computing and communication. It has also become increasingly evident that the difficulties encountered in the analysis and interpretation of individual sensing modalities may be overcome by integrating them into a multimodal human-computer interface. We examine several promising directions toward achieving multimodal HCI. We consider some of the emerging novel input modalities for HCI and the fundamental issues in integrating them at various levels, from early signal level to intermediate feature level to late decision level. We discuss the different computational approaches that may be applied at the different levels of modality integration. We also briefly review several demonstrated multimodal HCI systems and applications. Despite all the recent developments, it is clear that further research is needed for interpreting and fitting multiple sensing modalities in the context of HCI. This research can benefit from many disparate fields of study that increase our understanding of the different human communication modalities and their potential role in HCI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743
Fundamentals of speech recognition
Rabiner, L. R., & Juang, B.-H. F. (1993) · Cited by 7,680
Hand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
Artificial neural networks: a tutorial
Jain, A. K., Mao, J., & Mohiuddin, K. M. (1996) · Computer · doi:10.1109/2.485891 · Cited by 4,946
A FRAMEWORK FOR REPRESENTING KNOWLEDGE
Minsky, M. (1988) · Readings in Cognitive Science · doi:10.1016/b978-1-4832-1446-7.50018-2 · Cited by 4,536
Pfinder: real-time tracking of the human body
Wren, C. R., Azarbayejani, A., Darrell, T., & Pentland, A. P. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598236 · Cited by 4,169
An introduction to multisensor data fusion
Hall, D. L., & Llinas, J. (1997) · Proceedings of the IEEE · doi:10.1109/5.554205 · Cited by 2,629
Telerobotics, Automation, and Human Supervisory Control
Sheridan, T. B. (1992) · Medical Entomology and Zoology · Cited by 1,949
Rheingold, H. (1991) · Cited by 1,867
Visual interpretation of hand gestures for human-computer interaction: a review
Pavlovic, V. I., Sharma, R., & Huang, T. S. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598226 · Cited by 1,791
Self-citationDistributed Detection and Data Fusion
Varshney, P. K. (1997) · Springer eBooks · doi:10.1007/978-1-4612-1904-0 · Cited by 1,788
A Tutorial on Learning with Bayesian Networks
Heckerman, D. (2008) · Studies in Computational Intelligence · doi:10.1007/978-3-540-85066-3_3 · Cited by 1,693
“Put-that-there”: Voice and gesture at the graphics interface
Bolt, R. A. (1980) · Proceedings of the 7th annual conference on Computer graphics and interactive techniques - SIGGRAPH '80 · doi:10.1145/800250.807503 · Cited by 1,445
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
Kalman Filtering with Real-Time Applications
Chui, C. K., & Chen, G. (1987) · Springer Series in Information Sciences · doi:10.1007/978-3-662-02508-6 · Cited by 1,015
Generalization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 944
“Put-that-there”: Voice and gesture at the graphics interface
Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921
Coding, analysis, interpretation, and recognition of facial expressions
Essa, I. A., & Pentland, A. P. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598232 · Cited by 911
Sturman, D. J., & Zeltzer, L. K. (1994) · IEEE Computer Graphics and Applications · doi:10.1109/38.250916 · Cited by 727
A brief history of human-computer interaction technology
Myers, B. A. (1998) · Interactions · doi:10.1145/274430.274436 · Cited by 669
Introduction to random signal analysis and Kalman filtering
Brown, R. G. (1983) · Medical Entomology and Zoology · Cited by 634
Magnetic Position and Orientation Tracking System
Raab, F., Blood, E., Steiner, T., & Jones, H. (1979) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.1979.308860 · Cited by 614
Connectionist Speech Recognition
Bourlard, H. A., & Morgan, N. (1994) · Springer eBooks · doi:10.1007/978-1-4615-3210-1 · Cited by 585
Wearable computing: a first step toward personal imaging
Mann, S. (1997) · Computer · doi:10.1109/2.566147 · Cited by 510
Sensor fusion potential exploitation-innovative architectures and illustrative applications
Dasarathy, B. V. (1997) · Proceedings of the IEEE · doi:10.1109/5.554206 · Cited by 498
Charade: remote control of objects using free-hand gestures
Baudel, T., & Beaudouin-Lafon, M. (1993) · Communications of the ACM · doi:10.1145/159544.159562 · Cited by 473
Gesture as communication I: Its coordination with gaze and speech
Streeck, J. (1993) · Communication Monographs · doi:10.1080/03637759309376314 · Cited by 401
Glove-Talk: a neural network interface between a data-glove and a speech synthesizer
Fels, S. S., & Hinton, G. E. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.182690 · Cited by 374
Integration and synchronization of input modes during multimodal human-computer interaction
Oviatt, S., DeAngeli, A., & Kuhn, K. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258821 · Cited by 360
An introduction to software agents
Bradshaw, J. M. (1997) · The MIT Press eBooks · Cited by 359
Picard, R. W., & Healey, J. (1997) · Personal Technologies · doi:10.1007/bf01682026 · Cited by 287
Multiscale recursive estimation, data fusion, and regularization
Chou, K. C., Willsky, A. S., & Benveniste, A. (1994) · IEEE Transactions on Automatic Control · doi:10.1109/9.280746 · Cited by 281
Self-Regulation of the Brain and Behavior
Willis, A., Elbert, T., Rockstroh, B., Lutzenberger, W., & Bribaumer, N. (1985) · The American Journal of Psychology · doi:10.2307/1422455 · Cited by 251
Tracking requirements for augmented reality
Azuma, R. (1993) · Communications of the ACM · doi:10.1145/159544.159581 · Cited by 251
Evidence Theory and Its Applications
Guan, J., & Bell, D. A. (1991) · Elsevier eBooks · Cited by 214
Speech recognition in adverse environments
Juang, B. H. (1991) · Computer Speech & Language · doi:10.1016/0885-2308(91)90011-e · Cited by 198
Invariant features for 3-D gesture recognition
Campbell, L. W., Becker, D. A., Azarbayejani, A., Bobick, A. F., & Pentland, A. (2002) · Proceedings of the Second International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1996.557258 · Cited by 187
An overview of agent-oriented programming
Shoham, Y. (1997) · Cited by 173
The ALIVE system: wireless, full-body interaction with autonomous agents
Maes, P., Darrell, T., Blumberg, B., & Pentland, A. (1997) · Multimedia Systems · doi:10.1007/s005300050046 · Cited by 167
Pentland, A. (1996) · Scientific American · doi:10.1038/scientificamerican0496-68 · Cited by 140
Applying the Wizard of Oz technique to the study of multimodal systems
Salber, D., & Coutaz, J. (1993) · Lecture notes in computer science · doi:10.1007/3-540-57433-6_51 · Cited by 127
Gestures with Speech for Graphic Manipulation
Hauptmann, A., & McAvinney, P. (1993) · Int. J. Man Mach. Stud. · doi:10.1006/imms.1993.1011 · Cited by 110
Self-Regulation of the Brain and Behavior
Elbert, T. R., Rockstroh, B. S., Lutzenberger, W., & Birbaumer, N. (1984) · Springer eBooks · doi:10.1007/978-3-642-69379-3 · Cited by 108
An EEG-based method for graded cursor control
McFarland, D. J., Neat, G. W., Read, R. F., & Wolpaw, J. R. (1993) · Psychobiology · doi:10.3758/bf03327130 · Cited by 107
Visionary Speech: Looking Ahead to Practical Speechreading Systems
Hennecke, M. E., Stork, D. G., & Prasad, K. V. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_25 · Cited by 106
Biological and cognitive foundations of intelligent sensor fusion
Murphy, R. R. (1996) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.477859 · Cited by 103
Recording and analyzing eye-position data using a microcomputer workstation
Nodine, C. F., Kundel, H. L., Toto, L. C., & Krupinski, E. A. (1992) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203584 · Cited by 91
Integrating time alignment and neural networks for high performance continuous speech recognition
Haffner, P., Franzini, M. A., & Waibel, A. H. (1991) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1991.150289 · Cited by 89
Surgical Simulation: An Emerging Technology for Training in Emergency Medicine
Delp, S. L., Loan, P., Başdoğan, Ç., & Rosen, J. M. (1997) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres.1997.6.2.147 · Cited by 80
Vo, M. T., & Wood, C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.550794 · Cited by 80
Adam, J. (1993) · IEEE Spectrum · doi:10.1109/6.237580 · Cited by 78
Adaptive bimodal sensor fusion for automatic speechreading
Meier, U., Hürst, W., & Duchnowski, P. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543250 · Cited by 77
Interactive simulation in a multi-person virtual world
Codella, C. F., Sweeney, P., Turk, G., Jalili, R., Koved, L., Lewis, J. B., Ling, D. T., Lipscomb, J. S., Rabenhorst, D. A., Wang, C. P., et al. (1992) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/142750.142825 · Cited by 72
Controlling Computers with Neural Signals
Lusted, H. S., & Knapp, R. B. (1996) · Scientific American · doi:10.1038/scientificamerican1096-82 · Cited by 71
Man-machine communications through brain-wave processing
Keirn, Z. A., & Aunon, J. I. (1990) · IEEE Engineering in Medicine and Biology Magazine · doi:10.1109/51.62907 · Cited by 71
Multimodal user interfaces in the Open Agent Architecture
Moran, D. B., Cheyer, A. J., Julia, L. E., Martin, D. L., & Park, S. (1997) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/238218.238290 · Cited by 71
Waibel, A., Vo, M. T., Duchnowski, P., & Manke, S. (1996) · Artificial Intelligence Review · doi:10.1007/bf00127684 · Cited by 69
Virtual Environments for Medical Training and Education
Satava, R. M., & Jones, S. B. (1997) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres.1997.6.2.139 · Cited by 65
Cohen, P. R., Johnston, M. O., McGee, D. R., Oviatt, S. L., Pittman, J., Smith, I., Chen, L., & Clow, J. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974557.974562 · Cited by 65
“Finger-Pointer”: Pointing interface by image processing
Fukumoto, M., Suenaga, Y., & Mase, K. (1994) · Computers & Graphics · doi:10.1016/0097-8493(94)90157-0 · Cited by 65
MDScope — a visual computing environment for structural biology
Nelson, M., Humphrey, W., Kufrin, R., Gürsoy, A., Dalke, A., Kalé, L. V., Skeel, R. D., & Schulten, K. J. (1995) · Computer Physics Communications · doi:10.1016/0010-4655(95)00045-h · Cited by 54
Hot topics-eye-gaze computer interfaces: what you look at is what you get
Jacob, R. J. K. (1993) · Computer · doi:10.1109/mc.1993.274943 · Cited by 54
Speech/gesture interface to a visual computing environment for molecular biologists
Sharma, R., Huang, T. S., Pavlović, V., Zhao, Y., Lo, Z., Chu, S., & Schul, K. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.547311 · Cited by 45
Self-citationIntegration of eye-gaze, voice and manual response in multimodal user interface
Wang, J. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1995.538404 · Cited by 35
Jordan, M. I., & Bishop, C. (1996) · ACM Computing Surveys · doi:10.1145/234313.234348 · Cited by 33
Haptic interfaces: the study of force and tactile feedback systems
Bergamasco, M. (2002) · IEEE RO-MAN · doi:10.1109/roman.1995.531929 · Cited by 30
Real-time computer control using pattern recognition of the electromyogram
Putnam, W., & Knapp, R. B. (2005) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.1993.979112 · Cited by 26
Integration of audio/visual information for use in human-computer intelligent interaction
Pavlović, V., Berry, G. A., & Huang, T. S. (2002) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.1997.647399 · Cited by 20
Self-citationEye-gaze computer interfaces: computers that sense eye position on the display
Hutchinson, T. F. (1993) · Computer · doi:10.1109/mc.1993.620436 · Cited by 13
Fusion of visual and acoustic signals for command-word recognition
Kober, R., Harz, U., & Schiffers, J. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1997.596233 · Cited by 7
(1995) · DSpace@MIT (Massachusetts Institute of Technology)
Cited by
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893
Recent developments in human motion analysis
Wang, L., Hu, W., & Tan, T. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00100-0 · Cited by 962
Toward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Oviatt, S. (2007) · Human Factors and Ergonomics · doi:10.1201/9781410615862.ch21 · Cited by 485
Looking at people: sensing for ubiquitous and wearable computing
Pentland, A. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.824823 · Cited by 403
Metadata
- Authors
- Sharma, R., Pavlovic, V. I., & Huang, T. S.
- Venue
- Proceedings of the IEEE
- Published
- 1998-05
- Volume
- 86
- Issue
- 5
- Pages
- 853-869
- ISSN
- 0018-9219 (ISSN-L)
- OpenAlex
- W2170326347
- Cited by
- 302
- References
- 71
- FWCI
- 14.59
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.