CoChat
Sign Up
Cited by 137Online Learning and AnalyticsIntelligent Tutoring Systems and Adaptive LearningPsychometric Methodologies and Testing

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.

Subjective data, objective data and the role of bias in predictive modelling: Lessons from a dispositional learning analytics application

Open access
Tempelaar, Dirk, Rienties, Bart, Nguyen, Quan·2020·PLOS ONE
DOI 10.1371/journal.pone.0233977PMC PMC7292385
Open this paper in CoChat — free accountOpen PDF

Abstract

For decades, self-report measures based on questionnaires have been widely used in educational research to study implicit and complex constructs such as motivation, emotion, cognitive and metacognitive learning strategies. However, the existence of potential biases in such self-report instruments might cast doubts on the validity of the measured constructs. The emergence of trace data from digital learning environments has sparked a controversial debate on how we measure learning. On the one hand, trace data might be perceived as "objective" measures that are independent of any biases. On the other hand, there is mixed evidence of how trace data are compatible with existing learning constructs, which have traditionally been measured with self-reports. This study investigates the strengths and weaknesses of different types of data when designing predictive models of academic performance based on computer-generated trace data and survey data. We investigate two types of bias in self-report surveys: response styles (i.e., a tendency to use the rating scale in a certain systematic way that is unrelated to the content of the items) and overconfidence (i.e., the differences in predicted performance based on surveys' responses and a prior knowledge test). We found that the response style bias accounts for a modest to a substantial amount of variation in the outcomes of the several self-report instruments, as well as in the course performance data. It is only the trace data, notably that of process type, that stand out in being independent of these response style patterns. The effect of overconfidence bias is limited. Given that empirical models in education typically aim to explain the outcomes of learning processes or the relationships between antecedents of these learning outcomes, our analyses suggest that the bias present in surveys adds predictive power in the explanation of performance data and other questionnaire data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0233977

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Scatterplot of ΔConfidence against ERSneg, dots are representing students.
Fig 1 Scatterplot of ΔConfidence against ERSneg, dots are representing students.CC BY · PubMed Central
Fig 2 Regression betas of ERSpos and ERSneg from regressions of educational variables on ERSpos and ERSneg.
Fig 2 Regression betas of ERSpos and ERSneg from regressions of educational variables on ERSpos and ERSneg.CC BY · PubMed Central
Fig 3 Regression betas of ΔConfidence and ERSneg from regressions of educational variables on ΔConfidence respectively ERSpos and ERSneg.
Fig 3 Regression betas of ΔConfidence and ERSneg from regressions of educational variables on ΔConfidence respectively ERSpos and ERSneg.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 53 references

Research works printed but not linked: 6

  1. MVJ Veenman, 2013, International Handbook of Metacognition and Learning Technologies, Springer International Handbooks of Education 26, 157
  2. G Schraw, 2009, The educational psychology series. Handbook of metacognition in education, 415
  3. V Kovanović, 2017, SRI Education Analytics4Learning report series
  4. O Skrypnyk, 2015, Preparing for the digital university: a review of the history and current state of distance, blended, and online learning, 55
  5. A Williams, 2016, The Flipped College Classroom, Conceptualized and Re-Conceptualized, Part II, 149
  6. M Sangthong, 2020, The Effect of the Likert Point Scale and Sample Size on the Efficiency of Parametric and Nonparametric Tests, Thailand Statistician, 18, 55

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Tempelaar, D., Rienties, B., & Nguyen, Q.
Affiliations
  • Maastricht University (NL)
  • The Open University (GB)
  • University of Michigan (US)
Corresponding author
Tempelaar, Dirk
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2020-06-12
Volume
15
Issue
6
Pages
e0233977
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0233977
PMC
PMC7292385
OpenAlex
W3035222942
Cited by
137
References
53
Citations by year
2026: 18 · 2025: 30 · 2024: 38 · 2023: 23 · 2022: 18 · 2021: 9 · 2020: 1
FWCI
11.71
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
  • Open Research Online (The Open University) (acceptedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords
response style bias, overconfidence, self-report questionnaire, trace data, learning analytics, student performance prediction, process data, questionnaire validity, predictive modeling
MeSH terms
Metacognition, Academic Performance, Data Analysis, Attitude, Humans, Learning, Models, Theoretical, Motivation, Students, Cohort Studies, Bias, Databases, Factual, Self Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review