Towards investigating the validity of measurement of self-regulated learning based on trace data
Open accessThe DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Abstract
Contemporary research that looks at self-regulated learning (SRL) as processes of learning events derived from trace data has attracted increasing interest over the past decade. However, limited research has been conducted that looks into the validity of trace-based measurement protocols. In order to fill this gap in the literature, we propose a novel validation approach that combines theory-driven and data-driven perspectives to increase the validity of interpretations of SRL processes extracted from trace-data. The main contribution of this approach consists of three alignments between trace data and think aloud data to improve measurement validity. In addition, we define the match rate between SRL processes extracted from trace data and think aloud as a quantitative indicator together with other three indicators (sensitivity, specificity and trace coverage), to evaluate the “degree” of validity. We tested this validation approach in a laboratory study that involved 44 learners who learned individually about the topic of artificial intelligence in education with the use of a technology-enhanced learning environment for 45 minutes. Following this new validation approach, we achieved an improved match rate between SRL processes extracted from trace-data and think aloud data (training set: 54.24%; testing set: 55.09%) compared to the match rate before applying the validation approach (training set: 38.97%; test set: 34.54%). By considering think aloud data as “reference point”, this improvement of the match rate quantified the extent to which validity can be improved by using our validation approach. In conclusion, the novel validation approach presented in this study used both empirical evidence from think aloud data and rationale from our theoretical framework of SRL, which now, allows testing and improvement of the validity of trace-based SRL measurements.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Coding was performed by experts using theory-driven guidelines and a tracing protocol adapted from validated methods for measuring SRL via trace data (Fan et al., 2022; Rakovic et al., 2022; Li et al., 2023; van der Graaf et al., 2023a; Lim et al., 2023).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Fan et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“We measured SRL processes via trace data with Fan et al.'s (2022) approach, using an SRL process library (see Appendix A) that parsed raw data into sequences of learning actions representing SRL activities.”
From How do students regulate their learning with a genAI chatbot? (2026) · cites this paper as Fan et al.'s (2022)
In the citing paper
“Specifically, AI can detect the SRL processes automatically in the separate data streams and at the same time determine the synchronicity between the different layers, such as the work in FLoRa (Fan et al., 2022).”
From Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data (2023) · cites this paper as Fan et al., 2022
In the citing paper
“The flora project validated the algorithms by first building a theory-driven process library and then training a multimodal data-driven process library on top of it (Fan et al., 2022).”
From Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data (2023) · cites this paper as Fan et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
The Role of Goal Orientation in Self-Regulated Learning
Pintrich, P. R. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50043-3 · Cited by 5,173
A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students
Pintrich, P. R. (2004) · Educational Psychology Review · doi:10.1007/s10648-004-0006-x · Cited by 3,269 · cited 1×
“To date, researchers have proposed several theoretical models to describe SRL processes (e.g., Winne and Hadwin (1998), Pintrich (2004), and Zimmerman (2000)).”
BackgroundMeasuring Self-Regulated Learning
Winne, P. H., & Perry, N. E. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50045-7 · Cited by 1,383
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017) · Computers & Education · doi:10.1016/j.compedu.2016.10.001 · Cited by 1,003 · cited 2×
“Building on the previous literature (Siadaty et al., 2016b, c; Saint et al., 2020a, b; Kizilcec et al., 2017), we conducted multiple rounds of in-depth discussion to construct the theorydriven process library that is based on our theoretical framework.”
BackgroundDoes adaptive scaffolding facilitate students’ ability to regulate their learning with hypermedia?
Azevedo, R., Cromley, J. G., & Seibert, D. (2004) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2003.09.002 · Cited by 473 · cited 1×
“…scheme operationalises five macro-level SRL processes such as planning, monitoring and strategy use, which are further subdivided into 35 micro-level SRL processes such as judgement of learning, self-questioning and content evaluation, as per (Azevedo et al., 2004, 2005; Greene et al., 2008).”
MethodToward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor
Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(2)02 · Cited by 406
Improving Measurements of Self-Regulated Learning
WINNE, P. H. (2010) · Educational Psychologist · doi:10.1080/00461520.2010.517150 · Cited by 405 · cited 12×
“Once the trace-based measurement protocols are validated, they could and should be combined with think aloud protocols to measure and describe “a fuller picture” of SRL (Winne, 2010).”
BackgroundMethodInfluentialMotivated Strategies for Learning Questionnaire
Pintrich, P. R., & De Groot, E. V. (2012) · PsycTESTS Dataset · doi:10.1037/t09161-000 · Cited by 399
The Experience of Agency: Feelings, Judgments, and Responsibility
Haggard, P., & Tsakiris, M. (2009) · Current Directions in Psychological Science · doi:10.1111/j.1467-8721.2009.01644.x · Cited by 367 · cited 1×
“In these models, learners are commonly considered as agents (Haggard & Tsakiris, 2009) who decide which learning processes they should engage in to accomplish their goals for learning (Dunlosky & Thiede, 2013).”
Background
Research works printed but not linked: 6
- Azevedo, R., & Witherspoon, A. (2009). Self-regulated learning with hypermedia. In Handbook of metacognition in education (pp. 331–351). Routledge.
- Biswas, G., Segedy, J. R., & Kinnebrew, J. S. (2014). ”A Combined Theory-and Data-Driven Approach for Interpreting Learners’ Metacognitive Behaviors in Open-Ended Tutoring Environments. Design recommendations for intelligent tutoring systems, 2, 135.
- Fan, Y., Lim, L., van der Graaf, J., Kilgour, J., Engelmann, K., Bannert, M., Molenaar, I., Moore, J., & Gasevic, D. (2020). Measuring Micro-Level Self-Regulated Learning Processes with Enhanced Log Data and Eye Tracking Data.pdf. In Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20) (pp. 433–436).
- Günther, C. W., & Rozinat, A. (2012). Disco: discover your processes. BPM (Demos), 940, 40–44. Publisher: Citeseer.
- Winne, P. H., & Hadwin, A. F. (2007). The weave of motivation and self-regulated learning. In Motivation and Self-Regulated Learning. Num Pages: 18. Routledge.
- Zimmerman, B. J. (2000). Chapter 2 - attaining self-regulation: a social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.) Handbook of Self-Regulation (pp. 13–39). San Diego: Academic Press.
Cited by
Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13544 · Cited by 805
Self-citationPromises and challenges of generative artificial intelligence for human learning
Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02004-5 · Cited by 499
“…reflection, and integrating these with processcentred methodologies (for example, modelling selfregulated learning from learners’ digital traces 55 ) from the field of learning analytics 56,57 , learning scientists and researchers can develop generative agents capable of autonomously…”
Self-citationTowards hybrid human‐AI learning technologies
Molenaar, I. (2022) · European Journal of Education · doi:10.1111/ejed.12527 · Cited by 348
Self-citationStadler, M., Bannert, M., & Sailer, M. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108386 · Cited by 262
Self-citationMolenaar, I., Mooij, S. D., Azevedo, R., Bannert, M., Järvelä, S., & Gašević, D. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107540 · Cited by 217
Self-citation
Metadata
- Authors
- Fan, Y., van der Graaf, J., Lim, L., Raković, M., Singh, S., Kilgour, J., Moore, J., Molenaar, I., Bannert, M., & Gašević, D.
- Affiliations
- University of Edinburgh (GB)
- Technical University of Munich (DE)
- Monash University (AU)
- Radboud University Nijmegen (NL)
- Corresponding author
- Fan, Yizhou
- ORCID
- Venue
- Metacognition and Learning
- Published
- 2022-05-04
- Volume
- 17
- Issue
- 3
- Pages
- 949-987
- ISSN
- 1556-1623 (ISSN-L)
- 1556-1631
- OpenAlex
- W4225374280
- Cited by
- 101
- References
- 54
- Citations by year
- 2026: 33 · 2025: 27 · 2024: 23 · 2023: 11 · 2022: 7
- FWCI
- 13.13
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Metacognition and Learning (publishedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- HAL (submittedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Funders
- Deutsche Forschungsgemeinschaft (BA20144/10-1)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO464.18.104)
- Economic and Social Research Council (ES/S015701/1)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- student self-regulation, trace data, measurement validity, validation methodology, match rate, sensitivity and specificity, technology-enhanced learning environments, artificial intelligence in education, learning analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards investigating the validity of measurement of self-regulated learning based on trace data. https://app.cochat.ai/papers/10.1007/s11409-022-09291-1
- The paper
- Fan, Y., van der Graaf, J., Lim, L., Raković, M., Singh, S., Kilgour, J., Moore, J., Molenaar, I., Bannert, M., & Gašević, D. (2022). Towards investigating the validity of measurement of self-regulated learning based on trace data. Metacognition and Learning. https://doi.org/10.1007/s11409-022-09291-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.