Audiovisual Information Fusion in Human–Computer Interfaces and Intelligent Environments: A Survey
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Microphones and cameras have been extensively used to observe and detect human activity and to facilitate natural modes of interaction between humans and intelligent systems. Human brain processes the audio and video modalities, extracting complementary and robust information from them. Intelligent systems with audiovisual sensors should be capable of achieving similar goals. The audiovisual information fusion strategy is a key component in designing such systems. In this paper, we exclusively survey the fusion techniques used in various audiovisual information fusion tasks. The fusion strategy used tends to depend mainly on the model, probabilistic or otherwise, used in the particular task to process sensory information to obtain higher level semantic information. The models themselves are task oriented. In this paper, we describe the fusion strategies and the corresponding models used in audiovisual tasks such as speech recognition, tracking, biometrics, affective state recognition, and meeting scene analysis. We also review the challenges and existing solutions and also unresolved or partially resolved issues in these fields. Specifically, we discuss established and upcoming work in hierarchical fusion strategies and cross-modal learning techniques, identifying these as critical areas of research in the future development of intelligent systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“THE Turing test [1] suggests that a machine can be considered intelligent if a human judge involved in a natural conversation with a human and the machine, cannot distinguish between the two.”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
An introduction to biometric recognition
Jain, A. K., Ross, A., & Prabhakar, S. (2004) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2003.818349 · Cited by 4,974 · cited 1×
“infrared thermograms, DNA, ears, faces, gait, voice and signature, the audio and video modalities facilitate unobtrusive, user friendly, lower-costing-sensor based person identification [117].”
BackgroundMartinez, A. M., & Kak, A. C. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908974 · Cited by 3,249
A survey of socially interactive robots
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(02)00372-x · Cited by 3,162 · cited 1×
“Human-robot interaction [50][36] and avatar”
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 2×
“More recent work includes the use of discriminative classifiers like SVMs[110][111] and boosting and more expressive generative classifiers like Bayesian networks [112] to fuse audio-visual information.”
Facial emotion recognition using multi-modal information
De Silva, L. C., Miyasato, T., & Nakatsu, R. (2002) · Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat. No.97TH8237) · doi:10.1109/icics.1997.647126 · Cited by 319
Layered representations for human activity recognition
Oliver, N., Horvitz, E., & Garg, A. (2003) · Proceedings. Fourth IEEE International Conference on Multimodal Interfaces · doi:10.1109/icmi.2002.1166960 · Cited by 312 · cited 2×
“In [146], the authors propose a hierarchial HMM framework for modeling human activity.”
BackgroundIntegration of acoustic and visual speech signals using neural networks
Yuhas, B. P., Goldstein, M. H., & Sejnowski, T. J. (1989) · IEEE Communications Magazine · doi:10.1109/35.41402 · Cited by 209
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202 · cited 2×
“[112] 2006 Multistream Fused HMM Am.”
Cited by
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296
“, [21], [27], [58] (biomedical imaging) and [10], [11] (HMI).”
BackgroundInfluentialA Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 566
Lingenfelser, F., Wagner, J., Deng, J., Brueckner, R., Schuller, B., & Andre, E. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2635124 · Cited by 23
“[33] differentiate between fusion approaches at the signal, feature, model, decision and semantic level.”
BackgroundExploring ROI size in deep learning based lipreading
Koumparoulis, A., Potamianos, G., Mroueh, Y., & Rennie, S. J. (2017) · The 14th International Conference on Auditory-Visual Speech Processing · doi:10.21437/avsp.2017-13 · Cited by 21
Metadata
- Authors
- Shivappa, S. T., Trivedi, M. M., & Rao, B. D.
- Venue
- Proceedings of the IEEE
- Published
- 2010-10
- Volume
- 98
- Issue
- 10
- Pages
- 1692-1715
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W2055098062
- Cited by
- 128
- References
- 14
- FWCI
- 15.24
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Audiovisual Information Fusion in Human–Computer Interfaces and Intelligent Environments: A Survey. https://app.cochat.ai/papers/10.1109/jproc.2010.2057231
- The paper
- Shivappa, S. T., Trivedi, M. M., & Rao, B. D. (2010). Audiovisual Information Fusion in Human–Computer Interfaces and Intelligent Environments: A Survey. Proceedings of the IEEE. https://doi.org/10.1109/jproc.2010.2057231
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.