CoChat

Explanatory learner models: Why machine learning (alone) is not the answer

Open access
Rosé, Carolyn P., McLaughlin, Elizabeth A., Liu, Ran, Koedinger, Kenneth R. · 2019 · British Journal of Educational Technology
Cited by 136

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

Using data to understand learning and improve education has great promise. However, the promise will not be achieved simply by AI and Machine Learning researchers developing innovative models that more accurately predict labeled data. As AI advances, modeling techniques and the models they produce are getting increasingly complex, often involving tens of thousands of parameters or more. Though strides towards interpretation of complex models are being made in core machine learning communities, it remains true in these cases of “black box” modeling that research teams may have little possibility to peer inside to try understand how, why, or even whether such models will work when applied beyond the data on which they were built. Rather than relying on AI expertise alone, we suggest that learning engineering teams bring interdisciplinary expertise to bear to develop explanatory learner models that provide interpretable and actionable insights in addition to accurate prediction. We describe examples that illustrate use of different kinds of data (eg, click stream and discourse data) in different course content (eg, math and writing) and toward different goals (eg, improving student models and generating actionable feedback). We recommend learning engineering teams, shared infrastructure and funder incentives toward better explanatory learner model development that advances learning science, produces better pedagogical practices and demonstrably improves student learning. Practitioner Notes: What is already known about this topic Researchers in learning analytics and educational data mining have been successful in creating innovative models of data that optimize prediction. Some of these models produce scientific or practical insights and fewer have been put into use and demonstrated to enhance student learning. What this paper adds We provide examples of development of explanatory models of learners that not only accurately predict data but also provide scientific insights and yield practical outcomes. In particular, researchers with expertise in cognitive science and math education content use AI‐based data analytics to discover previously unrecognized barriers to geometry student learning. They use model‐derived insights to redesign an online tutoring system and “close‐the‐loop” by experimentally demonstrating that the new system produces better student learning than the original. Implications for practice and/or policy We define explanatory learning models and provide an articulation of a process for generating them that involves interdisciplinary teams employing human–computer interaction and learning engineering methods. Based on our experiences, we recommend learning engineering teams, shared infrastructure and funder incentives toward better explanatory learner model development that advances learning science, produces better pedagogical practices and demonstrably improves student learning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/bjet.12858

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Cited by

Metadata

Authors
Rosé, C. P., McLaughlin, E. A., Liu, R., & Koedinger, K. R.
Venue
British Journal of Educational Technology
Publisher
Wiley
Published
2019-08-13
Volume
50
Issue
6
Pages
2943-2958
ISSN
  • 0007-1013 (ISSN-L)
  • 1467-8535
OpenAlex
W2967076347
Cited by
136
References
7
FWCI
15.79
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.