The DOI matches its DataCite record.
ChatGPT in education: an effect in search of a cause
Abstract
Background: As researchers rush to investigate the potential of AI tools like ChatGPT to enhance learning, well‐documented pitfalls threaten the validity of this emerging research. Issues of media comparison research, where the confounding of instructional methods and technological affordances is unrecognised, may render effects uninterpretable. Objectives: Using a recent meta‐analysis by Deng et al. (Computers & Education, 227, 105224) as an example, we revisit key insights from the media/methods debate to highlight recurring conceptual challenges in ChatGPT efficacy studies. Methods: This conceptual article contrasts nascent ChatGPT research with the more established literature on Intelligent Tutoring Systems to identify three non‐negotiable considerations for interpretable effects: (1) descriptions of the precise nature of the experimental treatment and (2) the activities of the control group, as well as (3) outcome measures as valid indicators of learning. To provide some initial evidence, we audited a subset of primary experiments included in Deng et al.'s meta‐analysis, demonstrating that only a small minority of studies satisfied all three non‐negotiable considerations. Results and Conclusions: Loosely defined treatments, mismatched or opaque controls, and outcome measures with unclear links to durable learning obscure causal claims of this emerging literature. Observed gains cannot, at this time, be confidently attributed to ChatGPT, and meta‐analytics effect sizes may over‐ or understate its benefits. Progress, we argue, will require rigorous designs, transparent reporting, and a critical stance toward “fast science.”
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This finding aligns with the recent debate on performance versus learning confusion with GenAI (Weidlich and Gašević, 2025), where it was argued that some measures of performance are not sufficient to demonstrate true learning.”
From Beyond Self-Regulated Learning Processes: Unveiling Hidden Tactics in Generative AI-Assisted Writing (2025) · cites this paper as Weidlich and Gašević, 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Beyond Self-Regulated Learning Processes: Unveiling Hidden Tactics in Generative AI-Assisted Writing
Yang, K., Fan, Y., Tang, L., Raković, M., Li, X., Gašević, D., & Chen, G. (2025) · arXiv · doi:10.48550/arxiv.2508.10310 · Cited by 8
Self-citation
Metadata
- Authors
- Weidlich, J., Gašević, D., Drachsler, H., & Kirschner, P.
- Venue
- Wiley-Blackwell Publishing, Inc.
- Publisher
- Wiley-Blackwell Publishing, Inc.
- Published
- 2025-10
- References
- 0
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ChatGPT in education: an effect in search of a cause. https://app.cochat.ai/papers/10.5167/uzh-280621
- The paper
- Weidlich, J., Gašević, D., Drachsler, H., & Kirschner, P. (2025). ChatGPT in education: an effect in search of a cause. Wiley-Blackwell Publishing, Inc. https://doi.org/10.5167/uzh-280621
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.