CoChat

Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

Open access
Sharma, Kshitij, Papamitsiou, Zacharoula, Giannakos, Michail · 2019 · British Journal of Educational Technology
Cited by 130

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Abstract

Students' on‐task engagement during adaptive learning activities has a significant effect on their performance, and at the same time, how these activities influence students' behavior is reflected in their effort exertion. Capturing and explaining effortful (or effortless) behavior and aligning it with learning performance within contemporary adaptive learning environments, holds the promise to timely provide proactive and actionable feedback to students. Using sophisticated machine learning (ML) algorithms and rich learner data, facilitates inference‐making about several behavioral aspects (including effortful behavior) and about predicting learning performance, in any learning context. Researchers have been using ML methods in a “black‐box” approach, ie, as a tool where the input data is the learner data and the output is a given class from the chosen construct. This work proposes a methodological shift from the “black‐box” approach to a “grey‐box” approach that bridges the hypothesis/literature‐driven (feature extraction) “white‐box” approach with the computation/data‐driven (feature fusion) “black‐box” approach. This will allow us to utilize data features that are educationally and contextually meaningful. This paper aims to extend current methodological paradigms, and puts into practice the proposed approach in an adaptive self‐assessment case study taking advantage of new, cutting‐edge, interdisciplinary work on building pipelines for educational data, using innovative tools and techniques. Practitioner Notes: What is already known about this topic Capturing and measuring learners' engagement and behavior using physiological data has been explored during the last years and exhibits great potential. Effortless behavioral patterns commonly exhibited by learners, such as “cheating,” “guessing” or “gaming the system” counterfeit the learning outcome. Multimodal data can accurately predict learning engagement, performance and processes. What this paper adds Generalizes a methodology for building machine learning pipelines for multimodal educational data, using a modularized approach, namely the “grey‐box” approach. Showcases that fusion of eye‐tracking, facial expressions and arousal data provide the best prediction of effort and performance in adaptive learning settings. Highlights the importance of fusing data from different channels to obtain the most suited combinations from the different multimodal data streams, to predict and explain effort and performance in terms of pervasiveness, mobility and ubiquity. Implications for practice and/or policy Learning analytics researchers shall be able to use an innovative methodological approach, namely the “grey‐box,” to build machine learning pipelines from multimodal data, taking advantage of artificial intelligence capabilities in any educational context. Learning design professionals shall have the opportunity to fuse specific features of the multimodal data to drive the interpretation of learning outcomes in terms of physiological learner states. The constraints from the educational contexts (eg, ubiquity, low‐cost) shall be catered using the modularized gray‐box approach, which can also be used with standalone data sources.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/bjet.12854

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 88

  1. 10.1123/jsp.9.4.326 · DOI · Paper page
  2. Amos B., 2016, Openface: A general‐purpose face recognition library with mobile applications
  3. 10.1145/3027385.3027429 · DOI
  4. 10.18608/jla.2016.32.14 · DOI
  5. 10.1007/978-3-540-69132-7_44 · DOI · Paper page
  6. Baker R. S., 2004, Proceedings of ACM CHI 2004: Computer‐Human Interaction, 383
  7. 10.1016/j.compedu.2010.03.016 · DOI
  8. 10.1111/jcal.12265 · DOI
  9. 10.1016/j.schres.2006.06.015 · DOI
  10. 10.18608/jla.2016.32.11 · DOI · Paper page
  11. 10.1145/2946837 · DOI
  12. Bull S., 2002, Proceedings of workshop on individual and group modelling methods that help learners understand themselves, International Conference on Intelligent Tutoring Systems, 5
  13. 10.1080/0144929X.2018.1499803 · DOI
  14. 10.1177/0013164410385110 · doi:10.1177/0013164410385110
  15. 10.1016/j.chb.2017.02.059 · DOI · Paper page
  16. 10.18608/jla.2016.32.13 · DOI
  17. 10.1093/oso/9780195169157.003.0014 · DOI · Paper page
  18. 10.1609/aimag.v34i3.2483 · DOI · Paper page
  19. 10.1145/3027385.3027447 · DOI · Paper page
  20. 10.1111/jcal.12288 · DOI · Paper page
  21. 10.1080/00461520.2017.1281747 · DOI · Paper page
  22. 10.1145/3107990.3107998 · DOI
  23. 10.1504/IJLT.2009.028805 · DOI · Paper page
  24. 10.1145/2723576.2723588 · DOI
  25. 10.1109/ACII.2009.5349483 · DOI
  26. 10.1007/BF03173468 · DOI · Paper page
  27. 10.1145/3272973.3274099 · doi:10.1145/3272973.3274099
  28. 10.1016/j.jsp.2014.04.001 · DOI
  29. 10.1016/j.ijinfomgt.2019.02.003 · DOI · Paper page
  30. Gilzenrat M. S., 2003, Pupil dynamics predict changes in task engagement mediated by locus coeruleus, Society for Neuroscience Abstracts, 515, 19
  31. Gowda S. M., 2011, Improving Models of Slipping, Guessing, and Moment‐By‐Moment Learning with Estimates of Skill Difficulty, Educational Data Mining, 2011, 198
  32. 10.1109/FG.2018.00037 · DOI
  33. 10.1016/0193-3973(94)90008-6 · DOI · Paper page
  34. 10.1177/00224669050390030201 · DOI · Paper page
  35. 10.1109/ICASSP.2007.367284 · doi:10.1109/icassp.2007.367284
  36. 10.1037/0022-0663.100.1.1 · DOI · Paper page
  37. 10.1037/0033-295X.91.2.153 · doi:10.1037/0033-295x.91.2.153 · Paper page
  38. 10.1016/j.compedu.2018.02.013 · DOI · Paper page
  39. 10.1111/jcal.12262 · DOI
  40. Kalsbeek J. W. H., 1963, Scored regularity of the heart rate pattern and the measurement of perceptual or mental load, Ergonomics, 6, 306
  41. 10.5014/ajot.2012.004010 · DOI
  42. 10.1016/j.proeng.2012.06.298 · DOI · Paper page
  43. 10.1016/j.edurev.2013.10.001 · DOI · Paper page
  44. 10.1007/978-3-030-02631-8_5 · DOI
  45. 10.1037/npe0000028 · DOI · Paper page
  46. 10.1196/annals.1325.048 · doi:10.1196/annals.1325.048
  47. 10.1007/0-306-47637-1_6 · DOI · Paper page
  48. Linnenbrink E. A., 2003, Annual Meeting of the American Educational Research Association
  49. 10.1111/jcal.12315 · DOI
  50. 10.1523/JNEUROSCI.2565-12.2013 · DOI · Paper page
  51. 10.1145/3170358.3170379 · DOI
  52. Mattingly S. M. Gregg J. M. Audia P. Bayraktaroglu A. E. Campbell A. T. Chawla N. V. …Gao G.(2019 April).The Tesserae project: Large‐scale longitudinal in situ multimodal sensing of information workers. InExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems(p.p. CS11).ACM. · DOI
  53. McMillan J. H., 2008, Student self‐assessment: The key to stronger student motivation and higher achievement, Educational Horizons, 87, 40
  54. 10.1207/S15326985EP3702_5 · DOI · Paper page
  55. 10.1109/TLT.2018.2883419 · doi:10.1109/tlt.2018.2883419
  56. 10.1109/T-AFFC.2012.6 · DOI · Paper page
  57. 10.1007/978-94-009-4448-0_12 · DOI · Paper page
  58. 10.1093/acprof:oso/9780195168716.003.0004 · DOI
  59. 10.1145/3170358.3170406 · DOI · Paper page
  60. 10.18608/jla.2016.32.10 · DOI · Paper page
  61. 10.1109/ICL.2015.7318194 · DOI
  62. 10.1007/978-3-319-39483-1_40 · DOI
  63. 10.1111/bjet.12747 · DOI
  64. 10.1109/TLT.2016.2639508 · DOI · Paper page
  65. 10.18608/hla17.014 · DOI · Paper page
  66. Pelánek R., 2015, Metrics for evaluation of student models, Journal of Educational Data Mining, 7, 1
  67. 10.1016/j.compedu.2016.03.017 · DOI · Paper page
  68. 10.1111/jcal.12271 · DOI
  69. 10.1145/2883851.2883927 · DOI · Paper page
  70. 10.1111/jcal.12232 · DOI · Paper page
  71. 10.18608/jla.2014.13.16 · DOI
  72. 10.3758/PBR.16.1.1 · DOI · Paper page
  73. Rudner L. M., 2003, Annual Meeting of the National Council on Measurement in Education
  74. 10.1016/S1388-2457(03)00008-7 · DOI
  75. Schneider B., 2015, Unraveling students' interaction around a tangible interface using multimodal learning analytics, Journal of Educational Data Mining, 7, 89
  76. 10.1145/2883851.2883902 · DOI
  77. Sharma K., 2015, Identifying styles and paths toward success in MOOCs
  78. Sharma K., 2019, Proceedings of 13th International Conference on Computer Supported Collaborative Learning, 684
  79. Smith C., 2016, Using multimodal learning analytics to identify patterns of interactions in a body‐based mathematics activity, Journal of Interactive Learning Research, 27, 355
  80. 10.1145/3242969.3242989 · DOI
  81. 10.1145/1028014.1028067 · DOI
  82. 10.1207/s15324818ame1802_2 · DOI · Paper page
  83. 10.1080/08957347.2019.1577248 · DOI
  84. 10.1145/2666633.2666634 · DOI
  85. 10.1145/3170358.3170420 · DOI
  86. 10.1145/2666633.2666637 · DOI
  87. Yen C. H., 2015, An analytics‐based approach to managing cognitive load by using log data of learning management systems and footprints of social media, Educational Technology & Society, 18, 141
  88. 10.1111/jcal.12247 · DOI · Paper page

Cited by

Metadata

Authors
Sharma, K., Papamitsiou, Z., & Giannakos, M.
Venue
British Journal of Educational Technology
Publisher
Wiley
Published
2019-07-21
Volume
50
Issue
6
Pages
3004-3031
ISSN
  • 0007-1013 (ISSN-L)
  • 1467-8535
OpenAlex
W2963403519
Cited by
130
References
13
FWCI
13.34
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.