CoChat
Sign Up
Cited by 81Online Learning and AnalyticsIntelligent Tutoring Systems and Adaptive LearningEducational Technology and Pedagogy

Multimodal Learning Analytics: Rationale, Process, Examples, and Direction

Open access
Ochoa, Xavier·2022·The Handbook of Learning Analytics
DOI 10.18608/hla22.006
Open this paper in CoChat — free accountOpen PDF

Abstract

This chapter is an introduction to the use of multiple modalities of learning trace data to better understand and feedback learning processes that occur both in digital and face-to-face contexts. First, it will explain the rationale behind the emergence of this type of study, followed by a brief explanation of what Multimodal Learning Analytics (MmLA) is based on current conceptual understandings and current state-of-the-art implementations. The majority of this chapter is dedicated to describing the general process of MmLA from the mapping of learning constructs to low-level multimodal learning traces to the reciprocal implementation of multimedia recording, multimodal feature extraction, analysis, and fusion to detect behavioral markers and estimate the studied constructs. This process is illustrated by the detailed dissection of a real-world example. This chapter concludes with a discussion of the current challenges facing the field and the directions in which the field is moving to address them.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18608/hla22.006

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Challenges in comprehending VLA may intensify as VLA becomes more intricate with advancements in AI and the integration of multimodal sensory data.

    “These challenges may intensify as VLA becomes more intricate with advancements in artificial intelligence (AI) and the integration of multimodal sensory data (Ochoa, 2022).”

    From The effects of generative AI agents and scaffolding on enhancing students’ comprehension of visual learning analytics (2025) · cites this paper as Ochoa, 2022

    Section 4.2, “Technical Challenges,” says that AI-enabled multimodal learning analytics systems are technically difficult to design and implement, particularly because of sensor synchronization and data fusion. It does not address challenges in comprehending VLA or predict that those challenges will intensify as the systems advance. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Hamed S. Alavi and Pierre Dillenbourg. "An ambi- ent awareness tool for supporting supervised col- laborative problem solving". In: IEEE Transactions on Learning Technologies 5.3 (2012), pp. 264-274. DOI: 10.1109/TLT.2012.7. · DOI
  2. Alejandro Andrade. "Understanding student learn- ing trajectories using multimodal learning analyt- ics within an embodied-interaction learning envi- ronment". In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM, 2017, pp. 70-79. ISBN: 1-4503-4870-X.
  3. Sean Andrist, Dan Bohus, and Ashley Feniello. "Demonstrating a framework for rapid development of physically situated interactive systems". In: 2019 14th ACM/IEEE International Conference on Human- Robot Interaction (HRI). Mar. 2019, pp. 668-668. ISBN: 2167-2148. DOI: 10.1109/HRI.2019.8673067. · DOI
  4. Khaled Bachour, Frederic Kaplan, and Pierre Dillen- bourg. "An interactive table for supporting partic- ipation balance in face-to-face collaborative learn- ing". In: IEEE Transactions in Learning Technology 3.3 (2010), pp. 203-213. ISSN: 1939-1382. DOI: 10.1109/ tlt.2010.18. · DOI
  5. Paulo Blikstein. "Multimodal learning analytics". In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. Leuven, Bel- gium: Association for Computing Machinery, 2013, pp. 102-106. DOI: 10.1145/2460296.2460316. URL: https://doi.org/10.1145/2460296. 2460316. · DOI
  6. Paulo Blikstein. "Using learning analytics to assess students' behavior in open-ended programming tasks". In: Proceedings of the 1st International Confer- ence on Learning Analytics and Knowledge. Banff, Al- berta, Canada: Association for Computing Machin- ery, 2011, pp. 110-116. DOI: 10.1145/2090116. 2090132. URL: https://doi.org/10.1145/ 2090116.2090132. · DOI
  7. Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. "OpenPose: Realtime multi- person 2D pose estimation using Part Affinity Fields". In: IEEE transactions on pattern analysis and machine intelligence 43.1 (2019), pp. 172-186. ISSN: 0162-8828.
  8. Lei Chen, Chee Wee Leong, Gary Feng, and Chong Min Lee. "Using multimodal cues to analyze MLA'14 oral presentation quality corpus: Presen- tation delivery and slides quality". In: Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge. 2666640: ACM, 2014, pp. 45-52. DOI: 10.1145/2666633. 2666640. · DOI
  9. Paul Cobb, Jere Confrey, Andrea diSessa, Richard Lehrer, and Leona Schauble. "Design experiments in educational research". In: Educational Researcher 32.1 (Jan. 2003), pp. 9-13. ISSN: 0013-189X. DOI: 10 . 3102 / 0013189X032001009. URL: https : / / doi . org / 10 . 3102 / 0013189X032001009 (visited on 12/23/2020). · DOI · Paper page
  10. Lee J. Cronbach and Paul E. Meehl. "Construct validity in psychological tests". In: Psychologi- cal Bulletin 52.4 (1955), pp. 281-302. ISSN: 1939- 1455(Electronic),0033-2909(Print). DOI: 10.1037/ h0040957. · DOI · Paper page
  11. Mutlu Cukurova, Michail Giannakos, and Roberto Martinez-Maldonado. "The promise and challenges of multimodal learning analytics". In: British Journal of Educational Technology 51.5 (2020), pp. 1441-1449. ISSN: 0007-1013. DOI: 10.1111/bjet.13015. URL: https : / / bera -journals . onlinelibrary . wiley.com/doi/abs/10.1111/bjet.13015. · DOI · Paper page
  12. Daniele Di Mitri, Jan Schneider, Marcus Specht, and Hendrik Drachsler. "From signals to knowledge: A conceptual model for multimodal learning analyt- ics". In: Journal of Computer Assisted Learning 34.4 (Aug. 2018), pp. 338-349. ISSN: 0266-4909. DOI: 10. 1111/jcal.12288. URL: https://doi.org/ 10.1111/jcal.12288 (visited on 12/23/2020). · DOI
  13. Vanessa Echeverria. "Designing Feedback for Collo- cated Teams using Multimodal Learning Analytics". PhD thesis. University of Technology Sydney, 2020.
  14. Vanessa Echeverria, Roberto Martinez-Maldonado, Katherine Chiluiza, and Simon Buckingham Shum. "DBCollab: Automated feedback for face-to-face group database design". In: Proceedings of the 25th International Conference on Computers in Education, ICCE 2017-Main Conference Proceedings. 2017. ISBN: 986-94012-6-0.
  15. Maka Eradze, Maria Jesus Rodriguez Triana, and Mart Laanpere. "How to aggregate lesson observa- tion data into learning analytics datasets?" In: 6th Multimodal Learning Analytics (MMLA) Workshop and the 2nd Cross-LAK Workshop co-located with 7th Inter- national Learning Analytics and Knowledge Conference (LAK 2017). 2017.
  16. Michail N. Giannakos, Kshitij Sharma, Ilias O. Pap- pas, Vassilis Kostakos, and Eduardo Velloso. "Mul- timodal data as a means to understand the learning experience". In: International Journal of Information Management 48 (2019), pp. 108-119. ISSN: 0268-4012.
  17. Ruth Janning, Carlotta Schatten, and Lars Schmidt- Thieme. "Multimodal Affect recognition for adap- tive intelligent tutoring systems". In: International Journal of Information Management. 2014.
  18. René F. Kizilcec, Chris Piech, and Emily Schnei- der. "Deconstructing disengagement: analyzing learner sub-populations in massive open online courses". In: Educational Data Mining Conference (Workshops). Leuven, Belgium: Association for Com- puting Machinery, 2013, pp. 170-179. DOI: 10 . 1145/2460296.2460330. URL: https://doi. org/10.1145/2460296.2460330. · DOI · Paper page
  19. Kristine Lund. "The importance of gaze and ges- ture in interactive multimodal explanation". In: Lan- guage Resources and Evaluation 41.3 (2007), pp. 289- 303. ISSN: 1574020X, 15728412. URL: http : / / www . jstor . org / stable / 30204707 (visited on 12/23/2020).
  20. Taylor Martin, Ani Aghababyan, Jay Pfaffman, Jenna Olsen, Stephanie Baker, Philip Janisiewicz, Rachel Phillips, and Carmen Petrick Smith. "Nano- genetic learning analytics: illuminating student learning pathways in an online fraction game". In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. Leuven, Bel- gium: Association for Computing Machinery, 2013, pp. 165-169. DOI: 10.1145/2460296.2460328. URL: https://doi.org/10.1145/2460296. 2460328. · DOI
  21. Roberto Martinez-Maldonado, Vanessa Echeverria, Doug Elliott, Carmen Axisa, Tamara Power, and Simon Buckingham Shum. "Making the design of CSCL analytics interfaces a co-design process: The case of multimodal teamwork in healthcare". In: Computer Supported Collaborative Learning 2019. In- ternational Society of the Learning Sciences, 2019, pp. 859-860.
  22. Xavier Ochoa and Federico Dominguez. "Con- trolled evaluation of a multimodal system to im- prove oral presentation skills in a real learning set- ting". In: British Journal of Educational Technology 51.5 (2020), pp. 1615-1630. ISSN: 0007-1013. DOI: 10.1111/bjet.12987. URL: https://bera- journals.onlinelibrary.wiley.com/doi/ abs/10.1111/bjet.12987. · DOI
  23. Xavier Ochoa, Federico Domínguez, Bruno Guamán, Ricardo Maya, Gabriel Falcones, and Jaime Castells. "The RAP system: automatic feedback of oral presentation skills using mul- timodal analysis and low-cost sensors". In: Proceedings of the 8th International Conference on Learning Analytics and Knowledge. Sydney, New South Wales, Australia: Association for Computing Machinery, 2018, pp. 360-364. DOI: 10 . 1145 / 3170358 . 3170406. URL: https : //doi.org/10.1145/3170358.3170406. · DOI
  24. Xavier Ochoa and Marcelo Worsley. "Augmenting learning analytics with multimodal sensory data". In: Journal of Learning Analytics 3.2 (2016), pp. 213- 219. ISSN: 1929-7750.
  25. Luis P. Prieto, Kshitij Sharma, Pierre Dillenbourg, and María Jesús. "Teaching analytics: Towards auto- matic extraction of orchestration graphs using wear- able sensors". In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM, 2016, pp. 148-157. ISBN: 1-4503-4190-X.
  26. Stefan Scherer, Marcelo Worsley, and Louis- Philippe Morency. "1st international workshop on multimodal learning analytics: extended abstract". In: Proceedings of the 14th ACM international confer- ence on Multimodal interaction. Santa Monica, Cali- fornia, USA: Association for Computing Machin- ery, 2012, pp. 609-610. DOI: 10.1145/2388676. 2388803. URL: https://doi.org/10.1145/ 2388676.2388803. · DOI
  27. Jan Schneider, Dirk Börner, Peter Van Rosmalen, and Marcus Specht. "Presentation trainer, your pub- lic speaking multimodal coach". In: Proceedings of the 2015 ACM on International Conference on Multi- modal Interaction. acm, 2015, pp. 539-546. ISBN: 1- 4503-3912-3.
  28. Neil Selwyn. "What's the problem with learning analytics?" In: Journal of Learning Analytics 6.3 (2019), pp. 11-19. DOI: 10 . 18608 / jla . 2019 . 63 . 3. URL: https://learning-analytics.info/ index.php/JLA/article/view/6386 (visited on 12/23/2020). · DOI
  29. Kshitij Sharma, Zacharoula Papamitsiou, and Michail Giannakos. "Building pipelines for educa- tional data using AI and multimodal analytics: A "grey-box" approach". In: British Journal of Educa- tional Technology 50.6 (2019), pp. 3004-3031. ISSN: 0007-1013. DOI: 10 . 1111 / bjet . 12854. URL: https : / / bera -journals . onlinelibrary . wiley.com/doi/abs/10.1111/bjet.12854. · DOI
  30. Johannes Wagner, Florian Lingenfelser, Tobias Baur, Ionut Damian, Felix Kistler, and Elisabeth Andr. "The social signal interpretation (SSI) framework: Multimodal signal processing and recognition in real-time". In: Proceedings of the 21st ACM interna- tional conference on Multimedia. 2502223: ACM, 2013, pp. 831-834. DOI: 10.1145/2502081.2502223. · DOI
  31. Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. "Convolutional pose machines". In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2016, pp. 4724-4732.
  32. Marcelo Worsley. "(Dis) engagement matters: Identi- fying efficacious learning practices with multimodal learning analytics". In: Proceedings of the 8th Interna- tional Conference on Learning Analytics and Knowledge. ACM, 2018, pp. 365-369. ISBN: 1-4503-6400-4.
  33. Marcelo Worsley, Dor Abrahamson, Paulo Blikstein, Shuchi Grover, Bertrand Schneider, and Mike Tis- senbaum. "Situating multimodal learning analyt- ics". In: Proceedings of the Third International Confer- ence on Learning Analytics and Knowledge. Vol. 2. 2016, pp. 1346-1349. ISBN: 0-9903550-8-X.
  34. Marcelo Worsley and Paulo Blikstein. "Towards the development of multimodal action based assess- ment". In: Proceedings of International Conference of the Learning Sciences. Leuven, Belgium: Association for Computing Machinery, 2013, pp. 94-101. DOI: 10.1145/2460296.2460315. URL: https:// doi.org/10.1145/2460296.2460315. · DOI

Cited by

Sorted by citations

Authors
Ochoa, X.
Venue
The Handbook of Learning Analytics
Publisher
SOLAR
Published
2022
Pages
54-65
DOI
10.18608/hla22.006
OpenAlex
W4381145042
Cited by
81
References
2
Citations by year
2026: 21 · 2025: 27 · 2024: 26 · 2023: 5 · 2022: 2
FWCI
55.96
Type
book chapter
Language
en
Open access
Licence
cc-by
Locations
Solar eBooks (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Educational Technology and Pedagogy
Keywords
multimodal learning analytics, behavioral markers, multimodal feature extraction, multimodal fusion, multimedia recording, feedback in learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review