Challenges and opportunities of multimodal data in human learning: The computer science students' perspective
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Multimodal data have the potential to explore emerging learning practices that extend human cognitive capacities. A critical issue stretching in many multimodal learning analytics (MLA) systems and studies is the current focus aimed at supporting researchers to model learner behaviours, rather than directly supporting learners. Moreover, many MLA systems are designed and deployed without learners' involvement. We argue that in order to create MLA interfaces that directly support learning, we need to gain an expanded understanding of how multimodal data can support learners' authentic needs. We present a qualitative study in which 40 computer science students were tracked in an authentic learning activity using wearable and static sensors. Our findings outline learners' curated representations about multimodal data and the non‐technical challenges in using these data in their learning practice. The paper discusses 10 dimensions that can serve as guidelines for researchers and designers to create effective and ethically aware student‐facing MLA innovations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Verification strategies for establishing reliability and validity in qualitative research
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100202 · Cited by 4,978
Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars.
Maguire, M., & Delahunt, B. (2017) · All Ireland Journal of Higher Education · doi:10.62707/aishej.v9i3.335 · Cited by 2,099
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1×
“This is aligned with the prospective usefulness of eye-trackers and wrist-mounted sensors in educational technology research (Calvo & D'Mello, 2010; Was et al., 2016).”
BackgroundRoutledge International Handbook of Participatory Design
(2012) · doi:10.4324/9780203108543 · Cited by 1,606
Solove, D. J. (2006) · University of Pennsylvania Law Review · doi:10.2307/40041279 · Cited by 1,388
Learning Analytics: The Emergence of a Discipline
Siemens, G. (2013) · American Behavioral Scientist · doi:10.1177/0002764213498851 · Cited by 1,284 · cited 2×
“Concerns about data quality, sufficient scope of the data captured to reflect accurately the learning experience, privacy, and ethics of analytics are among the most significant concerns ’ (Siemens, 2013, p. 394).”
BackgroundLearning Analytics: Ethical Issues and Dilemmas
Slade, S., & Prinsloo, P. (2013) · American Behavioral Scientist · doi:10.1177/0002764213479366 · Cited by 1,078
The current landscape of learning analytics in higher education
Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.07.027 · Cited by 766 · cited 2×
“Fundamental to these competences is learners' ability to interpret, comprehend, and generate inference from analytics derived from multimodal data (Mangaroska & Giannakos, 2018; Viberg et al., 2018).”
BackgroundInterest, Prior Knowledge, and Learning
Tobias, S. (1994) · Review of Educational Research · doi:10.3102/00346543064001037 · Cited by 713 · cited 1×
“Therefore, neglecting prior knowledge can cause students to lose interest (Tobias, 1994), develop misconceptions (Mladenovic et al., 2016), or learn something opposed to the educator's intentions (Roschelle, 1997).”
BackgroundEthical and privacy principles for learning analytics
Pardo, A., & Siemens, G. (2014) · British Journal of Educational Technology · doi:10.1111/bjet.12152 · Cited by 710 · cited 1×
“…of reference comes from the existing literature in LA and MLA (Azevedo, 2015; Beattie et al., 2014; Blikstein & Worsley, 2016; Järvelä et al., 2019; Pardo & Siemens, 2014; Slade & Prinsloo, 2013), and the forthcoming challenges of technological evolution addressed by Stephanidis et al. (2019).”
BackgroundA Wearable Sensor for Unobtrusive, Long-Term Assessment of Electrodermal Activity
Poh, M.-Z., Swenson, N. C., & Picard, R. W. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2009.2038487 · Cited by 693 · cited 1×
“This also suggests the need for more research focused on affective wear-ables (Picard & Healey, 1997) that are flexible, easy to manipulate, and painless (Bonato, 2003; Olguín Olguín, 2011; Poh et al., 2010).”
BackgroundBig Data and analytics in higher education: Opportunities and challenges
Daniel, B. (2015) · British Journal of Educational Technology · doi:10.1111/bjet.12230 · Cited by 610 · cited 1×
“There has been a growing interest in exploiting digital traces that learners leave behind while interacting with educational systems through artificial intelligence (AI), big data (Daniel, 2015), and learning analytics (Siemens, 2013) innovations.”
BackgroundMousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., & Ghazi Saeedi, M. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2018.1558257 · Cited by 605
Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., et al. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2019.1619259 · Cited by 524 · cited 6×
“…of reference comes from the existing literature in LA and MLA (Azevedo, 2015; Beattie et al., 2014; Blikstein & Worsley, 2016; Järvelä et al., 2019; Pardo & Siemens, 2014; Slade & Prinsloo, 2013), and the forthcoming challenges of technological evolution addressed by Stephanidis et al. (2019).”
BackgroundInfluentialAn Analysis of Mental Workload in Pilots During Flight Using Multiple Psychophysiological Measures
Wilson, G. F. (2002) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1201_2 · Cited by 488
Azevedo, R. (2015) · Educational Psychologist · doi:10.1080/00461520.2015.1004069 · Cited by 418 · cited 5×
“At present, engagement is one of the most misused and overgeneralised constructs in educational research (Azevedo, 2015), which can benefit from examination on a grain-sized continuum utilizing multimodal data , from an individual in the moment to a group of learners in a class (Sinatra et al.,…”
BackgroundMethodInfluentialDesign for classroom orchestration
Dillenbourg, P. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.04.013 · Cited by 386 · cited 1×
“Prior research has already shown the value of multimodal data for educators' role in orchestrating (e.g., designing, managing, adapting) learning activities at multiple social levels (Aslan et al., 2019; Dillenbourg, 2013; Prieto et al., 2018).”
BackgroundWearable sensors/systems and their impact on biomedical engineering
Bonato, P. (2003) · IEEE Engineering in Medicine and Biology Magazine · doi:10.1109/memb.2003.1213622 · Cited by 382 · cited 1×
“This also suggests the need for more research focused on affective wear-ables (Picard & Healey, 1997) that are flexible, easy to manipulate, and painless (Bonato, 2003; Olguín Olguín, 2011; Poh et al., 2010).”
BackgroundAffectAura: an intelligent system for emotional memory
McDuff, D., Karlson, A., Kapoor, A., Roseway, A., & Czerwinski, M. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208525 · Cited by 242
Student Vulnerability, Agency and Learning Analytics: An Exploration
Prinsloo, P., & Slade, S. (2016) · Journal of Learning Analytics · doi:10.18608/jla.2016.31.10 · Cited by 207 · cited 2×
“…to define principles and practices for multimodal data collection and processing to decrease the risks associated with misuse, lack of transparency, misinterpretation, and erosion of contextual integrity, that might result in stereotyping and discrimination (Henman, 2004; Prinsloo & Slade, 2016).”
BackgroundSpikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12263 · Cited by 180 · cited 1×
“…systems to identify incorrect postures in healthcare training (Di Mitri, 2019); creating hand tracking algorithms to predict group work quality (Spikol et al., 2018); and using positioning trackers to identify teaching strategies in the physical classroom (Martinez-Maldonado, Schulte, et al.,…”
BackgroundAutomatic Detection of Learning-Centered Affective States in the Wild
Bosch, N., D'Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., & Zhao, W. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701397 · Cited by 178 · cited 1×
“…Grand Challenges for living and interacting in technology-augmented environments (Stephanidis et al., 2019); thus, learners are entitled to ask for learning systems (Bosch et al., 2015) that are beneficial to humans, endorsing their values and expectations, and facilitating their well-being.”
BackgroundWhat multimodal data can tell us about the students’ regulation of their learning process?
Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2021) · Learning and Instruction · doi:10.1016/j.learninstruc.2019.04.004 · Cited by 163 · cited 4×
“Interaction traces occurring across various modes of communication constitute what we refer to as multimodal data in this paper (Järvelä et al., 2019).”
BackgroundMethodInfluentialMultimodal data to design visual learning analytics for understanding regulation of learning
Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.019 · Cited by 140 · cited 2×
“…systems and studies are aimed at supporting researchers to model learner behaviours (see reviews by Crescenzi-Lanna, 2020; Di Mitri et al., 2018; Noroozi et al., 2019) or offer technical infrastructures to interconnect sensors and systems (Huertas Celdrán et al., 2020; Shankar et al., 2020,…”
BackgroundMethodFrom Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics
Martinez-Maldonado, R., Echeverria, V., Fernandez Nieto, G., & Buckingham Shum, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376148 · Cited by 123
Henman, P. (2004) · International Sociology · doi:10.1177/0268580904042899 · Cited by 67
Reframing classroom sensing: promise and peril
Ogan, A. (2019) · Interactions · doi:10.1145/3358902 · Cited by 28 · cited 2×
“Moreover, although facial expressions were not mentioned by the participants, possibly due to the risks of pervasive surveillance (Ogan, 2019), wristband sensors and cameras from the instrument dimension, might become an unobtrusive and a powerful instrumentation that researchers can easily scale…”
Background
Cited by
Artificial intelligence in multimodal learning analytics: a systematic literature review
Mohammadi, M., Tajik, E., Martinez-Maldonado, R., Sadiq, S., Tomaszewski, W., & Khosravi, H. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100426 · Cited by 85
AI-assisted automated scoring of picture-cued writing tasks for language assessment
Zhao, R., Zhuang, Y., Zou, D., Xie, Q., & Yu, P. L. H. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11473-y · Cited by 40
Personalisation methods in e-learning-A literature review
Mikić, V., Ilić, M., Kopanja, L., & Vesin, B. (2022) · Computer Applications in Engineering Education · doi:10.1002/cae.22566 · Cited by 39
“Previous literature reviews of personalised learning addressed the significance of the adaptive features of the employed e‐learning system [170] as well as the customisation features for learners to take control of the learning process [26, 181, 186].”
BackgroundSelf-citation
Metadata
- Authors
- Mangaroska, K., Martinez‐Maldonado, R., Vesin, B., & Gašević, D.
- Venue
- Journal of Computer Assisted Learning
- Publisher
- Wiley
- Published
- 2021-03-02
- Volume
- 37
- Issue
- 4
- Pages
- 1030-1047
- ISSN
- 0266-4909 (ISSN-L)
- 1365-2729
- OpenAlex
- W3135540143
- Cited by
- 54
- References
- 28
- FWCI
- 8.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.