Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Students' on‐task engagement during adaptive learning activities has a significant effect on their performance, and at the same time, how these activities influence students' behavior is reflected in their effort exertion. Capturing and explaining effortful (or effortless) behavior and aligning it with learning performance within contemporary adaptive learning environments, holds the promise to timely provide proactive and actionable feedback to students. Using sophisticated machine learning (ML) algorithms and rich learner data, facilitates inference‐making about several behavioral aspects (including effortful behavior) and about predicting learning performance, in any learning context. Researchers have been using ML methods in a “black‐box” approach, ie, as a tool where the input data is the learner data and the output is a given class from the chosen construct. This work proposes a methodological shift from the “black‐box” approach to a “grey‐box” approach that bridges the hypothesis/literature‐driven (feature extraction) “white‐box” approach with the computation/data‐driven (feature fusion) “black‐box” approach. This will allow us to utilize data features that are educationally and contextually meaningful. This paper aims to extend current methodological paradigms, and puts into practice the proposed approach in an adaptive self‐assessment case study taking advantage of new, cutting‐edge, interdisciplinary work on building pipelines for educational data, using innovative tools and techniques. Practitioner Notes: What is already known about this topic Capturing and measuring learners' engagement and behavior using physiological data has been explored during the last years and exhibits great potential. Effortless behavioral patterns commonly exhibited by learners, such as “cheating,” “guessing” or “gaming the system” counterfeit the learning outcome. Multimodal data can accurately predict learning engagement, performance and processes. What this paper adds Generalizes a methodology for building machine learning pipelines for multimodal educational data, using a modularized approach, namely the “grey‐box” approach. Showcases that fusion of eye‐tracking, facial expressions and arousal data provide the best prediction of effort and performance in adaptive learning settings. Highlights the importance of fusing data from different channels to obtain the most suited combinations from the different multimodal data streams, to predict and explain effort and performance in terms of pervasiveness, mobility and ubiquity. Implications for practice and/or policy Learning analytics researchers shall be able to use an innovative methodological approach, namely the “grey‐box,” to build machine learning pipelines from multimodal data, taking advantage of artificial intelligence capabilities in any educational context. Learning design professionals shall have the opportunity to fuse specific features of the multimodal data to drive the interpretation of learning outcomes in terms of physiological learner states. The constraints from the educational contexts (eg, ubiquity, low‐cost) shall be catered using the modularized gray‐box approach, which can also be used with standalone data sources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nicol, D. J., & Macfarlane‐Dick, D. (2006) · Studies in Higher Education · doi:10.1080/03075070600572090 · Cited by 5,846
Pekrun, R. (2006) · Educational Psychology Review · doi:10.1007/s10648-006-9029-9 · Cited by 5,566 · cited 2×
“For example, attention is a key factor in achieving high performance (eg, Greenfield, DeWinstanley, Kilpatrick, & Kaye, 1994; Harris, Danoff Friedlander, Saddler, Frizzelle, & Graham, 2005; Sharma, Alavi, Jermann & Dillenbourg, 2016) and emotions also play a significant role in explaining learning processes (eg, Frenz…”
BackgroundWorden, M. S., Foxe, J. J., Wang, N., & Simpson, G. V. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-06-j0002.2000 · Cited by 1,268
Klivenyi, P., Andreassen, O. A., Ferrante, R. J., Dedeoglu, A., Mueller, G., Lancelot, E., Bogdanov, M., Andersen, J. K., Jiang, D., & Beal, M. F. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-01-00001.2000 · Cited by 1,111
Rethinking the Use of Tests: A Meta-Analysis of Practice Testing
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017) · Review of Educational Research · doi:10.3102/0034654316689306 · Cited by 544
Wright, R. A., & Kirby, L. D. (2001) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(01)80007-1 · Cited by 313
The Index of Cognitive Activity: measuring cognitive workload
Marshall, S. P. (2003) · Proceedings of the IEEE 7th Conference on Human Factors and Power Plants · doi:10.1109/hfpp.2002.1042860 · Cited by 266 · cited 5×
“Marshall (2002) attempted to quantify small discontinuities in pupil size that are related to cognitive activity.”
BackgroundResultInfluentialIdentification of personal traits in adaptive learning environment: Systematic literature review
Afini Normadhi, N. B., Shuib, L., Md Nasir, H. N., Bimba, A., Idris, N., & Balakrishnan, V. (2019) · Computers & Education · doi:10.1016/j.compedu.2018.11.005 · Cited by 204 · cited 2×
“Furthermore, adaptation has shown a positive impact on learners’ engagement with the activities (Normadhi et al., 2019), which is also reflected on learners’ performance (Barla et al.”
BackgroundSpikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12263 · Cited by 180 · cited 7×
“Pipeline from data collection to educationally meaningful interpretations ML in multimodal educational data As seen from the review of relevant literature, several channels of learners’ physiological data can be recorded during students’ participation in learning activities, aiming to help us understand and explain ed…”
BackgroundResultInfluentialInvestigating the effect of an adaptive learning intervention on students’ learning
Liu, M., McKelroy, E., Corliss, S. B., & Carrigan, J. (2017) · Educational Technology Research and Development · doi:10.1007/s11423-017-9542-1 · Cited by 138
Research works printed but not linked: 88
- 10.1123/jsp.9.4.326 · DOI · Paper page
- Amos B., 2016, Openface: A general‐purpose face recognition library with mobile applications
- 10.1145/3027385.3027429 · DOI
- 10.18608/jla.2016.32.14 · DOI
- 10.1007/978-3-540-69132-7_44 · DOI · Paper page
- Baker R. S., 2004, Proceedings of ACM CHI 2004: Computer‐Human Interaction, 383
- 10.1016/j.compedu.2010.03.016 · DOI
- 10.1111/jcal.12265 · DOI
- 10.1016/j.schres.2006.06.015 · DOI
- 10.18608/jla.2016.32.11 · DOI · Paper page
- 10.1145/2946837 · DOI
- Bull S., 2002, Proceedings of workshop on individual and group modelling methods that help learners understand themselves, International Conference on Intelligent Tutoring Systems, 5
- 10.1080/0144929X.2018.1499803 · DOI
- 10.1177/0013164410385110 · doi:10.1177/0013164410385110
- 10.1016/j.chb.2017.02.059 · DOI · Paper page
- 10.18608/jla.2016.32.13 · DOI
- 10.1093/oso/9780195169157.003.0014 · DOI · Paper page
- 10.1609/aimag.v34i3.2483 · DOI · Paper page
- 10.1145/3027385.3027447 · DOI · Paper page
- 10.1111/jcal.12288 · DOI · Paper page
- 10.1080/00461520.2017.1281747 · DOI · Paper page
- 10.1145/3107990.3107998 · DOI
- 10.1504/IJLT.2009.028805 · DOI · Paper page
- 10.1145/2723576.2723588 · DOI
- 10.1109/ACII.2009.5349483 · DOI
- 10.1007/BF03173468 · DOI · Paper page
- 10.1145/3272973.3274099 · doi:10.1145/3272973.3274099
- 10.1016/j.jsp.2014.04.001 · DOI
- 10.1016/j.ijinfomgt.2019.02.003 · DOI · Paper page
- Gilzenrat M. S., 2003, Pupil dynamics predict changes in task engagement mediated by locus coeruleus, Society for Neuroscience Abstracts, 515, 19
- Gowda S. M., 2011, Improving Models of Slipping, Guessing, and Moment‐By‐Moment Learning with Estimates of Skill Difficulty, Educational Data Mining, 2011, 198
- 10.1109/FG.2018.00037 · DOI
- 10.1016/0193-3973(94)90008-6 · DOI · Paper page
- 10.1177/00224669050390030201 · DOI · Paper page
- 10.1109/ICASSP.2007.367284 · doi:10.1109/icassp.2007.367284
- 10.1037/0022-0663.100.1.1 · DOI · Paper page
- 10.1037/0033-295X.91.2.153 · doi:10.1037/0033-295x.91.2.153 · Paper page
- 10.1016/j.compedu.2018.02.013 · DOI · Paper page
- 10.1111/jcal.12262 · DOI
- Kalsbeek J. W. H., 1963, Scored regularity of the heart rate pattern and the measurement of perceptual or mental load, Ergonomics, 6, 306
- 10.5014/ajot.2012.004010 · DOI
- 10.1016/j.proeng.2012.06.298 · DOI · Paper page
- 10.1016/j.edurev.2013.10.001 · DOI · Paper page
- 10.1007/978-3-030-02631-8_5 · DOI
- 10.1037/npe0000028 · DOI · Paper page
- 10.1196/annals.1325.048 · doi:10.1196/annals.1325.048
- 10.1007/0-306-47637-1_6 · DOI · Paper page
- Linnenbrink E. A., 2003, Annual Meeting of the American Educational Research Association
- 10.1111/jcal.12315 · DOI
- 10.1523/JNEUROSCI.2565-12.2013 · DOI · Paper page
- 10.1145/3170358.3170379 · DOI
- Mattingly S. M. Gregg J. M. Audia P. Bayraktaroglu A. E. Campbell A. T. Chawla N. V. …Gao G.(2019 April).The Tesserae project: Large‐scale longitudinal in situ multimodal sensing of information workers. InExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems(p.p. CS11).ACM. · DOI
- McMillan J. H., 2008, Student self‐assessment: The key to stronger student motivation and higher achievement, Educational Horizons, 87, 40
- 10.1207/S15326985EP3702_5 · DOI · Paper page
- 10.1109/TLT.2018.2883419 · doi:10.1109/tlt.2018.2883419
- 10.1109/T-AFFC.2012.6 · DOI · Paper page
- 10.1007/978-94-009-4448-0_12 · DOI · Paper page
- 10.1093/acprof:oso/9780195168716.003.0004 · DOI
- 10.1145/3170358.3170406 · DOI · Paper page
- 10.18608/jla.2016.32.10 · DOI · Paper page
- 10.1109/ICL.2015.7318194 · DOI
- 10.1007/978-3-319-39483-1_40 · DOI
- 10.1111/bjet.12747 · DOI
- 10.1109/TLT.2016.2639508 · DOI · Paper page
- 10.18608/hla17.014 · DOI · Paper page
- Pelánek R., 2015, Metrics for evaluation of student models, Journal of Educational Data Mining, 7, 1
- 10.1016/j.compedu.2016.03.017 · DOI · Paper page
- 10.1111/jcal.12271 · DOI
- 10.1145/2883851.2883927 · DOI · Paper page
- 10.1111/jcal.12232 · DOI · Paper page
- 10.18608/jla.2014.13.16 · DOI
- 10.3758/PBR.16.1.1 · DOI · Paper page
- Rudner L. M., 2003, Annual Meeting of the National Council on Measurement in Education
- 10.1016/S1388-2457(03)00008-7 · DOI
- Schneider B., 2015, Unraveling students' interaction around a tangible interface using multimodal learning analytics, Journal of Educational Data Mining, 7, 89
- 10.1145/2883851.2883902 · DOI
- Sharma K., 2015, Identifying styles and paths toward success in MOOCs
- Sharma K., 2019, Proceedings of 13th International Conference on Computer Supported Collaborative Learning, 684
- Smith C., 2016, Using multimodal learning analytics to identify patterns of interactions in a body‐based mathematics activity, Journal of Interactive Learning Research, 27, 355
- 10.1145/3242969.3242989 · DOI
- 10.1145/1028014.1028067 · DOI
- 10.1207/s15324818ame1802_2 · DOI · Paper page
- 10.1080/08957347.2019.1577248 · DOI
- 10.1145/2666633.2666634 · DOI
- 10.1145/3170358.3170420 · DOI
- 10.1145/2666633.2666637 · DOI
- Yen C. H., 2015, An analytics‐based approach to managing cognitive load by using log data of learning management systems and footprints of social media, Educational Technology & Society, 18, 141
- 10.1111/jcal.12247 · DOI · Paper page
Cited by
A Review of Artificial Intelligence (AI) in Education from 2010 to 2020
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021) · Complexity · doi:10.1155/2021/8812542 · Cited by 1,364
“[76] ADP Online adaptive selfassessment procedure with multimodal data Web technologies *irty-two undergraduate students EXP 0”
MethodChiu, T. K. F., Xia, Q., Zhou, X., Chai, C. S., & Cheng, M. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100118 · Cited by 1,181
Artificial intelligence innovation in education: A twenty-year data-driven historical analysis
Guan, C., Mou, J., & Jiang, Z. (2020) · International Journal of Innovation Studies · doi:10.1016/j.ijis.2020.09.001 · Cited by 665
Artificial intelligence in multimodal learning analytics: a systematic literature review
Mohammadi, M., Tajik, E., Martinez-Maldonado, R., Sadiq, S., Tomaszewski, W., & Khosravi, H. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100426 · Cited by 85
Multimodal Learning Analytics: Rationale, Process, Examples, and Direction
Ochoa, X. (2022) · The Handbook of Learning Analytics · doi:10.18608/hla22.006 · Cited by 81
Adaptive Learning in AI Agents for the Metaverse: The ALMAA Framework
Xia, Y., Shin, S.-Y., & Lee, H.-A. (2024) · Applied Sciences · doi:10.3390/app142311410 · Cited by 15
Reina-Parrado, M., Román-Graván, P., & Hervás-Gómez, C. (2025) · European Journal of Educational Research · doi:10.12973/eu-jer.14.1.249 · Cited by 13
Wang, X., Sun, D., Cheng, G., & Luo, H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1080294 · Cited by 13
Supporting Learners in a Crisis Context with Smart Self-Assessment
Papamitsiou, Z., Lunde, M., Westermoen, J., & Giannakos, M. N. (2021) · Lecture Notes in Educational Technology · doi:10.1007/978-981-15-7869-4_14 · Cited by 9
Self-citation
Metadata
- Authors
- Sharma, K., Papamitsiou, Z., & Giannakos, M.
- Venue
- British Journal of Educational Technology
- Publisher
- Wiley
- Published
- 2019-07-21
- Volume
- 50
- Issue
- 6
- Pages
- 3004-3031
- ISSN
- 0007-1013 (ISSN-L)
- 1467-8535
- OpenAlex
- W2963403519
- Cited by
- 130
- References
- 13
- FWCI
- 13.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.