Multimodal Data Fusion in Learning Analytics: A Systematic Review
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Multimodal learning analytics (MMLA), which has become increasingly popular, can help provide an accurate understanding of learning processes. However, it is still unclear how multimodal data is integrated into MMLA. By following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, this paper systematically surveys 346 articles on MMLA published during the past three years. For this purpose, we first present a conceptual model for reviewing these articles from three dimensions: data types, learning indicators, and data fusion. Based on this model, we then answer the following questions: 1. What types of data and learning indicators are used in MMLA, together with their relationships; and 2. What are the classifications of the data fusion methods in MMLA. Finally, we point out the key stages in data fusion and the future research direction in MMLA. Our main findings from this review are (a) The data in MMLA are classified into digital data, physical data, physiological data, psychometric data, and environment data; (b) The learning indicators are behavior, cognition, emotion, collaboration, and engagement; (c) The relationships between multimodal data and learning indicators are one-to-one, one-to-any, and many-to-one. The complex relationships between multimodal data and learning indicators are the key for data fusion; (d) The main data fusion methods in MMLA are many-to-one, many-to-many and multiple validations among multimodal data; and (e) Multimodal data fusion can be characterized by the multimodality of data, multi-dimension of indicators, and diversity of methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.12.005 · Cited by 416 · cited 8×
“Blood blood volume pulse BVP Recognition of emotions [71]”
BackgroundMethodInfluentialSpikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12263 · Cited by 180 · cited 6×
“Many-to-One FE, PPG Emotion [73,134,135] AU, FA, LOG, HA Learning performance [37,86,100] LOG, AU, BL, SR Collaboration [114,136] PS, AU Emotion [121,137] AU, FE, BL, EDA, VO Collaboration, engagement, learning performance [78,130,138] EM, AU, VB, MP Teaching behavior [139,140] FE, HER, EM Engagement [83,141] FR, HER,…”
BackgroundMethodInfluentialWhat multimodal data can tell us about the students’ regulation of their learning process?
Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2021) · Learning and Instruction · doi:10.1016/j.learninstruc.2019.04.004 · Cited by 163
Cognitive Heat: Exploring the Usage of Thermal Imaging to Unobtrusively Estimate Cognitive Load
Abdelrahman, Y., Velloso, E., Dingler, T., Schmidt, A., & Vetere, F. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3130898 · Cited by 154
From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics
Martinez-Maldonado, R., Echeverria, V., Fernandez Nieto, G., & Buckingham Shum, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376148 · Cited by 123
Affective learning: Improving engagement and enhancing learning with affect-aware feedback
Grawemeyer, B., Mavrikis, M., Holmes, W., Gutiérrez-Santos, S., Wiedmann, M., & Rummel, N. (2017) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-017-9188-z · Cited by 105 · cited 2×
“Integration Methods Data Type Learning Indicators Author Many-to-One FE, PPG Emotion [73,134,135] AU, FA, LOG, HA Learning performance [37,86,100] LOG, AU, BL, SR Collaboration [114,136] PS, AU Emotion [121,137] AU, FE, BL, EDA, VO Collaboration, engagement, learning performance [78,130,138] EM, AU, VB, MP Teaching be…”
Exploring the potential of LMS log data as a proxy measure of student engagement
Henrie, C. R., Bodily, R., Larsen, R., & Graham, C. R. (2018) · Journal of Computing in Higher Education · doi:10.1007/s12528-017-9161-1 · Cited by 92
Bannert, M., Molenaar, I., Azevedo, R., Järvelä, S., & Gašević, D. (2017) · Proceedings of the Seventh International Learning Analytics & Knowledge Conference · doi:10.1145/3027385.3029463 · Cited by 34 · cited 1×
“By leveraging sophisticated machine learning and artificial intelligence techniques [10], MMLA focuses mainly on its paradigms [11,12], framework [6,13], multimodal data [14,15], system [16,17], multimodal data value chain [18], and case studies [19,20].”
Background
Cited by
Ouyang, F., Xu, W., & Cukurova, M. (2023) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-023-09387-z · Cited by 124
Artificial intelligence in multimodal learning analytics: a systematic literature review
Mohammadi, M., Tajik, E., Martinez-Maldonado, R., Sadiq, S., Tomaszewski, W., & Khosravi, H. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100426 · Cited by 85
Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2021) · doi:10.35542/osf.io/sd23y · Cited by 15
Advanced Neural Networks for Multimodal Data Fusion in Interdisciplinary Research
Dhivya, K., Kumar, S. N., Victoria, D. R. S., Sherly, S. I., & Durgadevi, G. (2024) · Advances in IT Standards and Standardization Research · doi:10.4018/979-8-3693-4759-1.ch008 · Cited by 4
Metadata
- Authors
- Mu, S., Cui, M., & Huang, X.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2020-11-30
- Volume
- 20
- Issue
- 23
- Pages
- 6856
- ISSN
- 1424-8220 (ISSN-L)
- OpenAlex
- W3110511009
- Cited by
- 154
- References
- 11
- FWCI
- 1.89
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Data Fusion in Learning Analytics: A Systematic Review. https://app.cochat.ai/papers/10.3390/s20236856
- The paper
- Mu, S., Cui, M., & Huang, X. (2020). Multimodal Data Fusion in Learning Analytics: A Systematic Review. Sensors. https://doi.org/10.3390/s20236856
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.