The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Subjective data, objective data and the role of bias in predictive modelling: Lessons from a dispositional learning analytics application
Open accessAbstract
For decades, self-report measures based on questionnaires have been widely used in educational research to study implicit and complex constructs such as motivation, emotion, cognitive and metacognitive learning strategies. However, the existence of potential biases in such self-report instruments might cast doubts on the validity of the measured constructs. The emergence of trace data from digital learning environments has sparked a controversial debate on how we measure learning. On the one hand, trace data might be perceived as "objective" measures that are independent of any biases. On the other hand, there is mixed evidence of how trace data are compatible with existing learning constructs, which have traditionally been measured with self-reports. This study investigates the strengths and weaknesses of different types of data when designing predictive models of academic performance based on computer-generated trace data and survey data. We investigate two types of bias in self-report surveys: response styles (i.e., a tendency to use the rating scale in a certain systematic way that is unrelated to the content of the items) and overconfidence (i.e., the differences in predicted performance based on surveys' responses and a prior knowledge test). We found that the response style bias accounts for a modest to a substantial amount of variation in the outcomes of the several self-report instruments, as well as in the course performance data. It is only the trace data, notably that of process type, that stand out in being independent of these response style patterns. The effect of overconfidence bias is limited. Given that empirical models in education typically aim to explain the outcomes of learning processes or the relationships between antecedents of these learning outcomes, our analyses suggest that the bias present in surveys adds predictive power in the explanation of performance data and other questionnaire data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Thus, LMS data are preferably combined with self-reports or other measures, as LMS data will only reflect a uni-dimensional aspect of the engagement in situ (e.g., Tempelaar et al., 2020).”
From Unpacking student engagement in higher education learning analytics: a systematic review (2024) · cites this paper as Tempelaar et al., 2020
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ryan, R. M., & Deci, E. L. (2000) · American Psychologist · doi:10.1037//0003-066x.55.1.68 · Cited by 54,599
Expectancy-value theory of achievement motivation
Wigfield, A., & Eccles, J. S. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1015 · Cited by 7,383
Moral Identity and Developmental Theory
Lapsley, D. K. (2015) · Human Development · doi:10.1159/000435926 · Cited by 3,843
The Academic Motivation Scale: A Measure of Intrinsic, Extrinsic, and Amotivation in Education
Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N., Senécal, C. B., & Vallières, É. F. (1992) · Educational and Psychological Measurement · doi:10.1177/0013164492052004025 · Cited by 2,890
Thanks coefficient alpha, we’ll take it from here.
McNeish, D. (2018) · Psychological Methods · doi:10.1037/met0000144 · Cited by 2,533
Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2010.10.002 · Cited by 2,248
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,512
Measuring Self-Regulated Learning
Winne, P. H., & Perry, N. E. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50045-7 · Cited by 1,383
On the measurement of achievement goals: Critique, illustration, and application.
Elliot, A. J., & Murayama, K. (2008) · Journal of Educational Psychology · doi:10.1037/0022-0663.100.3.613 · Cited by 1,281
Response Styles in Marketing Research: A Cross-National Investigation
Baumgartner, H., & Steenkamp, J.-B. E. M. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.2.143.18840 · Cited by 1,262
Research works printed but not linked: 6
- MVJ Veenman, 2013, International Handbook of Metacognition and Learning Technologies, Springer International Handbooks of Education 26, 157
- G Schraw, 2009, The educational psychology series. Handbook of metacognition in education, 415
- V Kovanović, 2017, SRI Education Analytics4Learning report series
- O Skrypnyk, 2015, Preparing for the digital university: a review of the history and current state of distance, blended, and online learning, 55
- A Williams, 2016, The Flipped College Classroom, Conceptualized and Re-Conceptualized, Part II, 149
- M Sangthong, 2020, The Effect of the Likert Point Scale and Sample Size on the Efficiency of Parametric and Nonparametric Tests, Thailand Statistician, 18, 55
Cited by
Sorted by citations
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
Future research recommendations for transforming higher education with generative AI
Chiu, T. K. F. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100197 · Cited by 511
Understanding mental fatigue and its detection: a comparative analysis of assessments and tools
Kunasegaran, K., Ismail, A. M. H., Ramasamy, S., Gnanou, J. V., Caszo, B. A., & Chen, P. L. (2023) · PeerJ · doi:10.7717/peerj.15744 · Cited by 188
Heikkinen, S., Saqr, M., Malmberg, J., & Tedre, M. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11281-4 · Cited by 124
Unpacking student engagement in higher education learning analytics: a systematic review
Bergdahl, N., Bond, M., Sjöberg, J., Dougherty, M., & Oxley, E. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00493-y · Cited by 138
Metadata
- Authors
- Tempelaar, D., Rienties, B., & Nguyen, Q.
- Affiliations
- Maastricht University (NL)
- The Open University (GB)
- University of Michigan (US)
- Corresponding author
- Tempelaar, Dirk
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-06-12
- Volume
- 15
- Issue
- 6
- Pages
- e0233977
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7292385
- OpenAlex
- W3035222942
- Cited by
- 137
- References
- 53
- Citations by year
- 2026: 18 · 2025: 30 · 2024: 38 · 2023: 23 · 2022: 18 · 2021: 9 · 2020: 1
- FWCI
- 11.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- Open Research Online (The Open University) (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- response style bias, overconfidence, self-report questionnaire, trace data, learning analytics, student performance prediction, process data, questionnaire validity, predictive modeling
- MeSH terms
- Metacognition, Academic Performance, Data Analysis, Attitude, Humans, Learning, Models, Theoretical, Motivation, Students, Cohort Studies, Bias, Databases, Factual, Self Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
