Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 628

The DOI matches its Crossref record; 136 of the 136 references it cites were found in a registry or an index.

A Review and Meta-Analysis of Multimodal Affect Detection Systems

D'mello, Sidney K., Kory, Jacqueline·2015·ACM Computing Surveys
DOI 10.1145/2682899
Open this paper in CoChat — free account

Abstract

Affect detection is an important pattern recognition problem that has inspired researchers from several areas. The field is in need of a systematic review due to the recent influx of Multimodal (MM) affect detection systems that differ in several respects and sometimes yield incompatible results. This article provides such a survey via a quantitative review and meta-analysis of 90 peer-reviewed MM systems. The review indicated that the state of the art mainly consists of person-dependent models (62.2% of systems) that fuse audio and visual (55.6%) information to detect acted (52.2%) expressions of basic emotions and simple dimensions of arousal and valence (64.5%) with feature- (38.9%) and decision-level (35.6%) fusion techniques. However, there were also person-independent systems that considered additional modalities to detect nonbasic emotions and complex dimensions using model-level fusion techniques. The meta-analysis revealed that MM systems were consistently (85% of systems) more accurate than their best unimodal counterparts, with an average improvement of 9.83% (median of 6.60%). However, improvements were three times lower when systems were trained on natural (4.59%) versus acted data (12.7%). Importantly, MM accuracy could be accurately predicted (cross-validated R 2 of 0.803) from unimodal accuracies and two system-level factors. Theoretical and applied implications and recommendations are discussed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/2682899

Cites

136 of 136 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 136 references

Research works printed but not linked: 41

  1. Baltrušaitis T., Proceedings of the International Conference on Multimedia and Expo (Workshop on Affective Analysis in Multimedia).
  2. Banda N., Proceedings of the 11th International Conference on Multimodal Interaction (ICMI).
  3. Chen C.-Y., Proceedings of the IEEE International Conference on Multimedia and Expo. IEEE
  4. Chetty G., Proceedings of the International Conference on Auditory-Visual Speech Processing, 115--120
  5. 10.1037/0033-2909.112.1.155 · doi:10.1037/0033-2909.112.1.155 · Paper page
  6. D'Mello S., Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Lukin et al. (Eds.). IOS Press
  7. S. K. D'Mello and A. C. Graesser. 2014. Confusion. In International Handbook of Emotions in Education R. Pekrun and L. Linnenbrink-Garcia (Eds.). Routledge New York NY 289--310. S. K. D'Mello and A. C. Graesser. 2014. Confusion. In International Handbook of Emotions in Education R. Pekrun and L. Linnenbrink-Garcia (Eds.). Routledge New York NY 289--310.
  8. Dy M., Proceedings of the 3rd International Conference on Human-Centric Computing. IEEE
  9. Emerich S., 2009, Proceedings of the 17th European Signal Processing Conference (EUSIPCO
  10. Eyben F., 2011, Ninth IEEE International Conference on Automatic Face and Gesture Recognition (FG
  11. Forbes-Riley K., Proceedings of the 4th Meeting of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 201--208
  12. Glodek M., Proceedings of the 11th International Workshop on Multiple Classifier Systems, Z.-H. Zhou, F. Roli, and J. Kittler (Eds.). Springer
  13. Graesser A., Proceedings of the 28th Annual Conference of the Cognitive Science Society, R. Sun and N. Miyake (Eds.). Cognitive Science Society
  14. Han M., 2007, Proceedings of the IEEE International Conference on Systems, Man and Cybernetics. IEEE
  15. Haq S., Proceedings of International Conference on Auditory-Visual Speech Processing, 53--58
  16. Haq S., Proceedings of the International Conference on Auditory-Visual Speech Processing, 185--190
  17. Hoch S., Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE
  18. Hoque M., Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG'11)
  19. Hussain M., Proceedings of the Australasian Data Mining Conference.
  20. Jiang D., Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, S. D'Mello, A. Graesser, B. Schuller, and J. Martin (Eds.). Springer-Verlag, 609--618
  21. Joo J.-T., 2007, Proceedings of the 8th Symposium on Advanced Intelligent Systems. 200--204
  22. Kanluan I., Proceedings of the 16th European Signal Processing Conference.
  23. Khalali Z., Proceedings of International Joint Conference on Neural Networks. IEEE
  24. Kim J., Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing. BIOSTEC, 460--463
  25. 10.1016/j.specom.2005.09.008 · DOI
  26. Lu K., Proceedings of the 21st International Conference on Pattern Recognition. IEEE
  27. Monkaresi H., Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE
  28. Pal P., Proceedings. of the 2006 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE
  29. Picard R., Affective Computing
  30. Rigoll G., Proceedings of the 10th International Conference Speech and Computer. 61--67
  31. 10.1037/1089-2680.2.3.247 · doi:10.1037/1089-2680.2.3.247 · Paper page
  32. Rozgic V., Proceedings of the Signal and Information Processing Association Annual Summit and Conference. IEEE
  33. Seppi D., Proceedings of the 9th Annual Conference of the International Speech Communication Association, 601--604
  34. Sutdiffe A., The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications
  35. Tomkins S. S., Affect Imagery Consciousness
  36. Tu B., Proceedings of the 6th International Conference on Intelligent Systems and Knowledge. Springer
  37. Vu H., IEEE International Conference on Fuzzy Systems. IEEE
  38. Walter S., Proceedings of the International Conference on Human-Computer Interaction, J. Jacko (Ed.). Springer
  39. Wang Y., Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE
  40. Wimmer M., Proceedings of the 3rd International Conference on Computer Vision Theory and Applications, 145--151
  41. Wöllmer M., Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH'10)

Cited by

See all 628

Metadata

Authors
D'mello, S. K., & Kory, J.
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2015-02-17
Volume
47
Issue
3
Pages
1-36
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/2682899
OpenAlex
W1985867508
Cited by
628
References
136
FWCI
4.42
Type
journal article
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Funders
  • NSF Graduate Research Fellowship under 1122374
  • Bill & Melinda Gates Foundation
  • National Science Foundation (NSF) (ITR 0325428, HCC 0834847, DRL 1235958)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Review and Meta-Analysis of Multimodal Affect Detection Systems. https://app.cochat.ai/papers/10.1145/2682899
The paper
D'mello, S. K., & Kory, J. (2015). A Review and Meta-Analysis of Multimodal Affect Detection Systems. ACM Computing Surveys. https://doi.org/10.1145/2682899

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.