CoChat

Towards investigating the validity of measurement of self-regulated learning based on trace data

Open access

The DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.

Abstract

Contemporary research that looks at self-regulated learning (SRL) as processes of learning events derived from trace data has attracted increasing interest over the past decade. However, limited research has been conducted that looks into the validity of trace-based measurement protocols. In order to fill this gap in the literature, we propose a novel validation approach that combines theory-driven and data-driven perspectives to increase the validity of interpretations of SRL processes extracted from trace-data. The main contribution of this approach consists of three alignments between trace data and think aloud data to improve measurement validity. In addition, we define the match rate between SRL processes extracted from trace data and think aloud as a quantitative indicator together with other three indicators (sensitivity, specificity and trace coverage), to evaluate the “degree” of validity. We tested this validation approach in a laboratory study that involved 44 learners who learned individually about the topic of artificial intelligence in education with the use of a technology-enhanced learning environment for 45 minutes. Following this new validation approach, we achieved an improved match rate between SRL processes extracted from trace-data and think aloud data (training set: 54.24%; testing set: 55.09%) compared to the match rate before applying the validation approach (training set: 38.97%; test set: 34.54%). By considering think aloud data as “reference point”, this improvement of the match rate quantified the extent to which validity can be improved by using our validation approach. In conclusion, the novel validation approach presented in this study used both empirical evidence from think aloud data and rationale from our theoretical framework of SRL, which now, allows testing and improvement of the validity of trace-based SRL measurements.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11409-022-09291-1

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Learning environment and different functional zones
Fig. 1 Learning environment and different functional zonesCC-BY · From the PDF
Image 2
CC-BY · From the PDF
Fig. 3 The process followed by the trace parser to extract SRL processes
Fig. 3 The process followed by the trace parser to extract SRL processesCC-BY · From the PDF
Fig. 4 Schematic diagram of first alignment
Fig. 4 Schematic diagram of first alignmentCC-BY · From the PDF
Fig. 5 Schematic diagram of the second alignment
Fig. 5 Schematic diagram of the second alignmentCC-BY · From the PDF
Image 6
CC-BY · From the PDF
Fig. 7 Example 1: process map of all First-reading, LC.F sessions ; The numbers are the frequency of actions or transitions of actions in this session group
Fig. 7 Example 1: process map of all First-reading, LC.F sessions ; The numbers are the frequency of actions or transitions of actions in this session groupCC-BY · From the PDF
Image 8
CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Fig. 10 Empirical evaluation of validity improvement of the trace-based SRL measurement results against the think aloud data
Fig. 10 Empirical evaluation of validity improvement of the trace-based SRL measurement results against the think aloud dataCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 53 references

Research works printed but not linked: 6

  1. Azevedo, R., & Witherspoon, A. (2009). Self-regulated learning with hypermedia. In Handbook of metacognition in education (pp. 331–351). Routledge.
  2. Biswas, G., Segedy, J. R., & Kinnebrew, J. S. (2014). ”A Combined Theory-and Data-Driven Approach for Interpreting Learners’ Metacognitive Behaviors in Open-Ended Tutoring Environments. Design recommendations for intelligent tutoring systems, 2, 135.
  3. Fan, Y., Lim, L., van der Graaf, J., Kilgour, J., Engelmann, K., Bannert, M., Molenaar, I., Moore, J., & Gasevic, D. (2020). Measuring Micro-Level Self-Regulated Learning Processes with Enhanced Log Data and Eye Tracking Data.pdf. In Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20) (pp. 433–436).
  4. Günther, C. W., & Rozinat, A. (2012). Disco: discover your processes. BPM (Demos), 940, 40–44. Publisher: Citeseer.
  5. Winne, P. H., & Hadwin, A. F. (2007). The weave of motivation and self-regulated learning. In Motivation and Self-Regulated Learning. Num Pages: 18. Routledge.
  6. Zimmerman, B. J. (2000). Chapter 2 - attaining self-regulation: a social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.) Handbook of Self-Regulation (pp. 13–39). San Diego: Academic Press.

Cited by

See all 126

Metadata

Authors
Fan, Y., van der Graaf, J., Lim, L., Raković, M., Singh, S., Kilgour, J., Moore, J., Molenaar, I., Bannert, M., & Gašević, D.
Affiliations
  • University of Edinburgh (GB)
  • Technical University of Munich (DE)
  • Monash University (AU)
  • Radboud University Nijmegen (NL)
Corresponding author
Fan, Yizhou
Venue
Metacognition and Learning
Published
2022-05-04
Volume
17
Issue
3
Pages
949-987
ISSN
  • 1556-1623 (ISSN-L)
  • 1556-1631
OpenAlex
W4225374280
Cited by
101
References
54
Citations by year
2026: 33 · 2025: 27 · 2024: 23 · 2023: 11 · 2022: 7
FWCI
13.13
Type
journal article
Language
en
Open access
Licence
Locations
  • Metacognition and Learning (publishedVersion, cc-by)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • Monash University Research Portal (Monash University) (submittedVersion, other-oa)
  • HAL (submittedVersion)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
Funders
  • Deutsche Forschungsgemeinschaft (BA20144/10-1)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO464.18.104)
  • Economic and Social Research Council (ES/S015701/1)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
student self-regulation, trace data, measurement validity, validation methodology, match rate, sensitivity and specificity, technology-enhanced learning environments, artificial intelligence in education, learning analytics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards investigating the validity of measurement of self-regulated learning based on trace data. https://app.cochat.ai/papers/10.1007/s11409-022-09291-1
The paper
Fan, Y., van der Graaf, J., Lim, L., Raković, M., Singh, S., Kilgour, J., Moore, J., Molenaar, I., Bannert, M., & Gašević, D. (2022). Towards investigating the validity of measurement of self-regulated learning based on trace data. Metacognition and Learning. https://doi.org/10.1007/s11409-022-09291-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.