Multimodal data as a means to understand the learning experience
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Most work in the design of learning technology uses click-streams as their primary data source for modelling & predicting learning behaviour. In this paper we set out to quantify what, if any, advantages do physiological sensing techniques provide for the design of learning technologies. We conducted a lab study with 251 game sessions and 17 users focusing on skill development (i.e., user's ability to master complex tasks). We collected click-stream data, as well as eye-tracking, electroencephalography (EEG), video, and wristband data during the experiment. Our analysis shows that traditional click-stream models achieve 39% error rate in predicting learning performance (and 18% when we perform feature selection), while for fused multimodal the error drops up to 6%. Our work highlights the limitations of standalone click-stream models, and quantifies the expected benefits of using a variety of multimodal data coming from physiological sensing. Our findings help shape the future of learning technology research by pointing out the substantial benefits of physiological sensing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sweller, J., Ayres, P., & Kalyuga, S. (2011) · doi:10.1007/978-1-4419-8126-4 · Cited by 5,531
Sweller, J. (2011) · Psychology of Learning and Motivation · doi:10.1016/b978-0-12-387691-1.00002-8 · Cited by 4,856
Beyond the hype: Big data concepts, methods, and analytics
Gandomi, A., & Haider, M. (2015) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2014.10.007 · Cited by 4,226 · cited 1×
“Recent technological developments and data science advancements (Gandomi and Haider, 2015; Tursunbayeva et al., 2018) have boosted the growth of non-invasive high-frequency physiological-data collections (e.g., rapid development of wearable devices, dry electrode techniques etc.), acquired through…”
BackgroundWhat we can do and what we cannot do with fMRI
Logothetis, N. K. (2008) · Nature · doi:10.1038/nature06976 · Cited by 3,612 · cited 2×
“…to self-report certain conditions (Dimoka et al., 2012), or simply that the human body simply cannot represent the richness of psychometric measures (Logothetis, 2008), or psychometric measures are not complex enough to capture the richness of the interaction with the system (e.g., cognitive…”
BackgroundOne millisecond face alignment with an ensemble of regression trees
Kazemi, V., & Sullivan, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.241 · Cited by 2,831 · cited 2×
“For the facial landmark detection we use the method described in (Kazemi and Sullivan, 2014) using Dlib and Python.”
MethodThe pupil as a measure of emotional arousal and autonomic activation
Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J. (2008) · Psychophysiology · doi:10.1111/j.1469-8986.2008.00654.x · Cited by 1,919 · cited 1×
“For instance, pupil response has been successfully used to infer cognitive and emotional processes (Bradley et al., 2008), and pupil size has been used to infer cognitive listening load, user information processing and understanding (Zekveld et al., 2014).”
MethodPrinciples of sensorimotor learning
Wolpert, D. M., Diedrichsen, J., & Flanagan, J. R. (2011) · Nature Reviews Neuroscience · doi:10.1038/nrn3112 · Cited by 1,391 · cited 1×
“Skill acquisition (commonly termed also as movement-motor learning (Ericsson et al., 2018)) is a loosely de fi ned term that encompasses motor adaptation and decision-making (Wolpert et al., 2011; Krakauer and Mazzoni, 2011).”
BackgroundMultimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 1×
“The seamless integration and combination of different apparatuses, as well as the harmonization and sense-making of multimodal data-streams to support the learning experience, is an extremely challenging process (Lahat et al., 2015).”
BackgroundEmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276 · cited 3×
“eye-movements) has been reported as a promising approach (Zheng et al., 2018).”
BackgroundInfluentialOnline Learning: Adoption, Continuance, and Learning Outcome—A Review of Literature
Panigrahi, R., Srivastava, P. R., & Sharma, D. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.05.005 · Cited by 607
Fine motor skills and early comprehension of the world: two new school readiness indicators.
Grissmer, D., Grimm, K. J., Aiyer, S. M., Murrah, W. M., & Steele, J. S. (2010) · Developmental Psychology · doi:10.1037/a0020104 · Cited by 585
Human sensorimotor learning: adaptation, skill, and beyond
Krakauer, J. W., & Mazzoni, P. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.06.012 · Cited by 558 · cited 1×
“Skill acquisition (commonly termed also as movement-motor learning (Ericsson et al., 2018)) is a loosely de fi ned term that encompasses motor adaptation and decision-making (Wolpert et al., 2011; Krakauer and Mazzoni, 2011).”
BackgroundEpisodic and semantic memory: an analysis in the EEG theta and alpha band
Klimesch, W., Schimke, H., & Schwaiger, J. (1994) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(94)90164-3 · Cited by 441
Identifying emotional states using keystroke dynamics
Epp, C., Lippold, M., & Mandryk, R. L. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979046 · Cited by 364 · cited 4×
“Such keystroke dynamics are associated with users’ cognitive and affective states, like mood and emotions (Epp et al., 2011), and cognitive and physical stress (Vizer & Sears, 2017; Vizer, 2013).”
BackgroundInfluentialIntegrating cognitive load theory and concepts of human–computer interaction
Hollender, N., Hofmann, C., Deneke, M., & Schmitz, B. (2010) · Computers in Human Behavior · doi:10.1016/j.chb.2010.05.031 · Cited by 328 · cited 9×
“Brain data using EEG signals have also been used in HCI to detect shifts in engagement and workload.”
BackgroundInfluentialPeople analytics—A scoping review of conceptual boundaries and value propositions
Tursunbayeva, A., Di Lauro, S., & Pagliari, C. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.08.002 · Cited by 263 · cited 1×
“Recent technological developments and data science advancements (Gandomi and Haider, 2015; Tursunbayeva et al., 2018) have boosted the growth of non-invasive high-frequency physiological-data collections (e.g., rapid development of wearable devices, dry electrode techniques etc.), acquired through…”
BackgroundAutomatic Detection of Learning-Centered Affective States in the Wild
Bosch, N., D'Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., & Zhao, W. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701397 · Cited by 178 · cited 3×
“Such strategies can offer valuable information in advancing learning technologies (Bosch et al., 2015; Wiggins et al., 2015; Rana and Dwivedi, 2017), such as providing information that a user is engaged or delighted with the current task.”
BackgroundMethodCognitive Heat: Exploring the Usage of Thermal Imaging to Unobtrusively Estimate Cognitive Load
Abdelrahman, Y., Velloso, E., Dingler, T., Schmidt, A., & Vetere, F. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3130898 · Cited by 154 · cited 1×
“…data – and in particular physiological data – for understanding users ’ cognition, emotion, attention, information acquisition and more (Zheng et al., 2018; Kanjo et al., 2015; Hassib et al., 2016; Scheidt and Chung, 2018; Abdelrahman et al, 2017), research in this direction remains scarce.”
BackgroundSelf-citationParsing heterogeneity in attention‐deficit hyperactivity disorder using EEG ‐based subgroups
Loo, S. K., McGough, J. J., McCracken, J. T., & Smalley, S. L. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12814 · Cited by 73 · cited 1×
“, 1994), and finally, the beta band has been shown to reflect emotional process (Ray & Cole, 1985) and inattention (Roh et al., 2016; Loo et al., 2018).”
BackgroundBrainatwork: logging cognitive engagement and tasks in the workplace using electroencephalography
Hassib, M., Khamis, M., Friedl, S., Schneegass, S., & Alt, F. (2017) · Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia · doi:10.1145/3152832.3152865 · Cited by 24
Vizer, L. M., & Sears, A. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.03.001 · Cited by 12 · cited 2×
“Similarly to mouse behavior, keystroke dynamics seem promising for modeling user behavior (Epp et al., 2011; Vizer and Sears, 2017).”
Background
Cited by
Dwivedi, Y. K., Hughes, L., Baabdullah, A. M., Ribeiro-Navarrete, S., Giannakis, M., Al-Debei, M. M., Dennehy, D., Metri, B., Buhalis, D., Cheung, C. M. K., et al. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2022.102542 · Cited by 2,753
Self-citationEducational data mining and learning analytics: An updated survey
Romero, C., & Ventura, S. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1355 · Cited by 1,120
Artificial Intelligence in Education: The Three Paradigms
Ouyang, F., & Jiao, P. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100020 · Cited by 991
Revolutionizing education with AI: Exploring the transformative potential of ChatGPT
Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13152 · Cited by 975
“For example, a diverse range of multimodal data, encompassing physiological sensing, eye-tracking, and electroencephalography, have been employed to acquire a multifaceted comprehension of learners ’ levels, enabling a high-quality prediction of their learning performance (Giannakos et al., 2019).”
BackgroundOuyang, F., Zheng, L., & Jiao, P. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-10925-9 · Cited by 831
Metadata
- Authors
- Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E.
- Venue
- International Journal of Information Management
- Publisher
- Elsevier BV
- Published
- 2019-10
- Volume
- 48
- Pages
- 108-119
- ISSN
- 0268-4012 (ISSN-L)
- OpenAlex
- W2933627812
- Cited by
- 202
- References
- 21
- FWCI
- 25.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.