The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Subjective data, objective data and the role of bias in predictive modelling: Lessons from a dispositional learning analytics application
Open accessAbstract
For decades, self-report measures based on questionnaires have been widely used in educational research to study implicit and complex constructs such as motivation, emotion, cognitive and metacognitive learning strategies. However, the existence of potential biases in such self-report instruments might cast doubts on the validity of the measured constructs. The emergence of trace data from digital learning environments has sparked a controversial debate on how we measure learning. On the one hand, trace data might be perceived as "objective" measures that are independent of any biases. On the other hand, there is mixed evidence of how trace data are compatible with existing learning constructs, which have traditionally been measured with self-reports. This study investigates the strengths and weaknesses of different types of data when designing predictive models of academic performance based on computer-generated trace data and survey data. We investigate two types of bias in self-report surveys: response styles (i.e., a tendency to use the rating scale in a certain systematic way that is unrelated to the content of the items) and overconfidence (i.e., the differences in predicted performance based on surveys' responses and a prior knowledge test). We found that the response style bias accounts for a modest to a substantial amount of variation in the outcomes of the several self-report instruments, as well as in the course performance data. It is only the trace data, notably that of process type, that stand out in being independent of these response style patterns. The effect of overconfidence bias is limited. Given that empirical models in education typically aim to explain the outcomes of learning processes or the relationships between antecedents of these learning outcomes, our analyses suggest that the bias present in surveys adds predictive power in the explanation of performance data and other questionnaire data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Thus, LMS data are preferably combined with self-reports or other measures, as LMS data will only reflect a uni-dimensional aspect of the engagement in situ (e.g., Tempelaar et al., 2020).”
From Unpacking student engagement in higher education learning analytics: a systematic review (2024) · cites this paper as Tempelaar et al., 2020
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ryan, R. M., & Deci, E. L. (2000) · American Psychologist · doi:10.1037//0003-066x.55.1.68 · Cited by 54,599
Expectancy-value theory of achievement motivation
Wigfield, A., & Eccles, J. S. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1015 · Cited by 7,383
Moral Identity and Developmental Theory
Lapsley, D. K. (2015) · Human Development · doi:10.1159/000435926 · Cited by 3,843
The Academic Motivation Scale: A Measure of Intrinsic, Extrinsic, and Amotivation in Education
Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N., Senécal, C. B., & Vallières, É. F. (1992) · Educational and Psychological Measurement · doi:10.1177/0013164492052004025 · Cited by 2,890
Thanks coefficient alpha, we’ll take it from here.
McNeish, D. (2018) · Psychological Methods · doi:10.1037/met0000144 · Cited by 2,533
Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2010.10.002 · Cited by 2,248
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,512
Measuring Self-Regulated Learning
Winne, P. H., & Perry, N. E. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50045-7 · Cited by 1,383
On the measurement of achievement goals: Critique, illustration, and application.
Elliot, A. J., & Murayama, K. (2008) · Journal of Educational Psychology · doi:10.1037/0022-0663.100.3.613 · Cited by 1,281
Response Styles in Marketing Research: A Cross-National Investigation
Baumgartner, H., & Steenkamp, J.-B. E. M. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.2.143.18840 · Cited by 1,262
Academic buoyancy: Towards an understanding of students' everyday academic resilience
Martin, A. J., & Marsh, H. W. (2007) · Journal of School Psychology · doi:10.1016/j.jsp.2007.01.002 · Cited by 1,007
A 3 × 2 achievement goal model.
Elliot, A. J., Murayama, K., & Pekrun, R. H. (2011) · Journal of Educational Psychology · doi:10.1037/a0023952 · Cited by 970
Weijters, B., Cabooter, E., & Schillewaert, N. (2010) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2010.02.004 · Cited by 813
Martin, A. J. (2006) · British Journal of Educational Psychology · doi:10.1348/000709906x118036 · Cited by 739
Vermunt, J. D. H. M. (1996) · Higher Education · doi:10.1007/bf00129106 · Cited by 684
Ferguson, R., & Shum, S. B. (2012) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2330601.2330616 · Cited by 553
A Social-Cognitive, Control-Value Theory of Achievement Emotions
Pekrun, R. H. (2000) · Advances in psychology · doi:10.1016/s0166-4115(00)80010-2 · Cited by 541
Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2014.05.038 · Cited by 449
Self-citationMeasurement Theory and Applications for the Social Sciences
Gagnon, R. J. (2019) · Measurement Interdisciplinary Research and Perspectives · doi:10.1080/15366367.2019.1610343 · Cited by 400
Shum, S. B., & Ferguson, R. (2012) · Cited by 399
Measuring emotions during epistemic activities: the Epistemically-Related Emotion Scales
Pekrun, R., Vogl, E., Muis, K. R., & Sinatra, G. M. (2017) · Cognition and Emotion · doi:10.1080/02699931.2016.1204989 · Cited by 373
The Psychology of Academic Achievement
Winne, P. H., & Nesbit, J. C. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.093008.100348 · Cited by 320
Handbook of Research on Learning and Instruction
(2016) · Routledge eBooks · doi:10.4324/9781315736419 · Cited by 289
Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017) · Journal of Learning Analytics · doi:10.18608/jla.2017.42.10 · Cited by 225
Emotions as Drivers of Learning and Cognitive Development
Pekrun, R. H. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1_3 · Cited by 220
Personality attributes associated with extreme response style.
Hamilton, D. L. (1968) · Psychological Bulletin · doi:10.1037/h0025606 · Cited by 213
Measurement Theory and Applications for the Social Sciences
Bandalos, D. L. (2018) · Cited by 209
The stability of individual response styles.
Weijters, B., Geuens, M., & Schillewaert, N. (2010) · Psychological Methods · doi:10.1037/a0018721 · Cited by 207
Learning dispositions and transferable competencies
Shum, S. B., & Crick, R. D. (2012) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2330601.2330629 · Cited by 202
New Perspectives on Affect and Learning Technologies
Calvo, R. A., & D’Mello, S. K. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1 · Cited by 175
Modeling academic achievement by self-reported versus traced goal orientation
Zhou, M., & Winne, P. H. (2012) · Learning and Instruction · doi:10.1016/j.learninstruc.2012.03.004 · Cited by 172
Frenzel, A. C., Becker-Kurz, B., Pekrun, R. H., & Goetz, T. (2015) · PLoS ONE · doi:10.1371/journal.pone.0129630 · Cited by 160
Piecing the learning analytics puzzle: a consolidated model of a field of research and practice
Gašević, D., Kovanović, V., & Joksimovíc, S. (2017) · Learning Research and Practice · doi:10.1080/23735082.2017.1286142 · Cited by 146
Fincham, E., Gašević, D., Jovanović, J., & Pardo, A. (2018) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2018.2823317 · Cited by 145
LEARNING TO SELF-MONITOR AND SELF-REGULATE
Veenman, M. V. J. (2011) · Routledge eBooks · doi:10.4324/9780203839089-18 · Cited by 132
Tempelaar, D. T., Gijselaers, W., van der Loeff, S. S., & Nijhuis, J. F. H. (2006) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2006.10.004 · Cited by 124
Self-citationStudent profiling in a dispositional learning analytics application using formative assessment
Tempelaar, D., Rienties, B., Mittelmeier, J., & Nguyen, Q. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.010 · Cited by 122
Self-citationThe Stability of Individual Response StylesPreprint
Weijters, B., Geuens, M., & Schillewaert, N. (2008) · RePEc: Research Papers in Economics · Cited by 122
Measuring Metacognitive Judgments
Schraw, G. J. (2009) · doi:10.4324/9780203876428.ch21 · Cited by 90
Towards Actionable Learning Analytics Using Dispositions
Tempelaar, D. T., Rienties, B., & Nguyen, Q. (2017) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2017.2662679 · Cited by 85
Self-citationPotential‐based achievement goals
Elliot, A. J., Murayama, K., Kobeisy, A. N., & Lichtenfeld, S. (2014) · British Journal of Educational Psychology · doi:10.1111/bjep.12051 · Cited by 77
Formative assessment and learning analytics
Tempelaar, D. T., Heck, A., Cuypers, H., van der Kooij, H., & van de Vrie, E. (2013) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2460296.2460337 · Cited by 75
Self-citationUnpacking the intertemporal impact of self-regulation in a blended mathematics environment
Rienties, B., Tempelaar, D. T., Nguyen, Q., & Littlejohn, A. H. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.07.007 · Cited by 75
Self-citationLearning Strategies, Study Skills, and Self-Regulated Learning in Postsecondary Education
Winne, P. H. (2013) · Higher Education: Handbook of Theory and Research · doi:10.1007/978-94-007-5836-0_8 · Cited by 73
Assessing Metacognitive Skills in Computerized Learning Environments
Veenman, M. V. J. (2013) · Springer international handbooks of education · doi:10.1007/978-1-4419-5546-3_11 · Cited by 64
The Effect of Extreme Response and Non-extreme Response Styles on Testing Measurement Invariance
Liu, M., Harbaugh, A. G., Harring, J. R., & Hancock, G. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00726 · Cited by 56
RESPONSE STYLES AS PERSONALITY VARIABLES: A THEORETICAL INTEGRATION OF MULTIVARIATE RESEARCH1
Damarin, F. L., & Messick, S. (1965) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1965.tb00967.x · Cited by 42
Coertjens, L., Donche, V., Mæyer, S. D., Vanthournout, G., & Petegem, P. V. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067854 · Cited by 35
Effects of Scale Direction on Response Style of Ordinal Rating Scales
Liu, M., & Keusch, F. (2017) · Journal of Official Statistics · doi:10.1515/jos-2017-0008 · Cited by 32
Investigating learning strategies in a dispositional learning analytics context
Tempelaar, D. T., Rienties, B., & Nguyen, Q. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170385 · Cited by 19
Self-citationWilliams, A., Sun, Z., Xie, K., Garcia, E., Ashby, I., Exter, M., Largent, D., Lu, P., Szafron, D., Ahmed, S., et al. (2016) · Springer eBooks · doi:10.1007/978-3-319-41855-1_8 · Cited by 11
Self-citationA Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods
Kovanović, V., Gašević, D., Hatala, M., & Siemens, G. (2017) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · doi:10.13140/rg.2.2.10611.32804 · Cited by 9
Extreme response style bias in burn survivors
Ni, P., Marino, M., Dore, E. C., Sonis, L. A., Ryan, C. M., Schneider, J. C., Jette, A. M., & Kazis, L. E. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215898 · Cited by 5
Flagged first, then by citations
Research works printed but not linked: 6
- MVJ Veenman, 2013, International Handbook of Metacognition and Learning Technologies, Springer International Handbooks of Education 26, 157
- G Schraw, 2009, The educational psychology series. Handbook of metacognition in education, 415
- V Kovanović, 2017, SRI Education Analytics4Learning report series
- O Skrypnyk, 2015, Preparing for the digital university: a review of the history and current state of distance, blended, and online learning, 55
- A Williams, 2016, The Flipped College Classroom, Conceptualized and Re-Conceptualized, Part II, 149
- M Sangthong, 2020, The Effect of the Likert Point Scale and Sample Size on the Efficiency of Parametric and Nonparametric Tests, Thailand Statistician, 18, 55
Cited by
Sorted by citations
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
Future research recommendations for transforming higher education with generative AI
Chiu, T. K. F. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100197 · Cited by 511
Understanding mental fatigue and its detection: a comparative analysis of assessments and tools
Kunasegaran, K., Ismail, A. M. H., Ramasamy, S., Gnanou, J. V., Caszo, B. A., & Chen, P. L. (2023) · PeerJ · doi:10.7717/peerj.15744 · Cited by 188
Heikkinen, S., Saqr, M., Malmberg, J., & Tedre, M. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11281-4 · Cited by 124
Unpacking student engagement in higher education learning analytics: a systematic review
Bergdahl, N., Bond, M., Sjöberg, J., Dougherty, M., & Oxley, E. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00493-y · Cited by 138
Metadata
- Authors
- Tempelaar, D., Rienties, B., & Nguyen, Q.
- Affiliations
- Maastricht University (NL)
- The Open University (GB)
- University of Michigan (US)
- Corresponding author
- Tempelaar, Dirk
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-06-12
- Volume
- 15
- Issue
- 6
- Pages
- e0233977
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7292385
- OpenAlex
- W3035222942
- Cited by
- 137
- References
- 53
- Citations by year
- 2026: 18 · 2025: 30 · 2024: 38 · 2023: 23 · 2022: 18 · 2021: 9 · 2020: 1
- FWCI
- 11.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- Open Research Online (The Open University) (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- response style bias, overconfidence, self-report questionnaire, trace data, learning analytics, student performance prediction, process data, questionnaire validity, predictive modeling
- MeSH terms
- Metacognition, Academic Performance, Data Analysis, Attitude, Humans, Learning, Models, Theoretical, Motivation, Students, Cohort Studies, Bias, Databases, Factual, Self Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
