CoChat

Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms

Open access
Kumar, Harsh, Xiao, Ruiwei, Lawson, Benjamin, Musabirov, Ilya, Shi, Jiakai, Wang, Xinyuan, Luo, Huayin, Williams, Joseph Jay, Rafferty, Anna N., Stamper, John, Liut, Michael · 2024 · Proceedings of the Eleventh ACM Conference on Learning @ Scale

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.

Abstract

Two randomized field experiments in undergraduate CS courses evaluated LLM-guided self-reflection. In Experiment 1 (N=145), post-lesson LLM reflection increased student self-confidence and showed a positive trend on later assessments. In Experiment 2 (N=112), LLM reflection and questionnaire reflection performed equally well, significantly outperforming passive review of lecture slides on a proctored exam two weeks later.

Overview

CoChat Summary

TL;DR

Two randomized field experiments in undergraduate computer science courses tested whether LLM-guided post-assignment self-reflection improves learning outcomes and self-efficacy. Study 1 (N=145) found higher self-confidence and greater willingness to interact with the LLM again, but no statistically significant exam-score gain; Study 2 (N=112) found questionnaire-based and LLM-based reflection groups performed comparably and apparently better than slide review, though the exam differences were not statistically significant.

Key findings

  • In Study 1, students assigned to LLM-guided reflection had a slightly higher mean exam score two weeks later (57.27) than students with no reflection (53.99), but the difference was not statistically significant (p=0.1008). Homework means were nearly identical (59.09 vs. 59.07).
  • Study 1 self-confidence increased from 4.18 to 4.56 in the reflection condition (paired t-test: t=-2.03, df=65, p=0.046), while the no-reflection condition showed no apparent change (4.04 to 4.15; t=-0.64, df=78, p=0.5269). The authors treat the confidence result as preliminary because it is close to the 0.05 threshold. Students in the reflection condition also reported higher willingness to interact with the LLM again (Wilcoxon rank sum: W=3097.00, p=0.046), while both groups found the LLM tutor equally helpful (W=2706.00, p=0.689).
  • In Study 2, exam means were 68.30 for questionnaire-based reflection, 66.91 for LLM-based reflection, and 62.25 for revision of important slides; the difference was not significant (F(2,105)=0.94, p=0.394; Eta2=0.02). No differences in self-confidence before and after reflection were found.
  • Qualitative analysis of Study 1 chat logs identified themes of positive feedback and affirmations, expanding on answers, offering guidance, and encouraging reflection; the LLM was sometimes inconsistent in prompting students to reflect.

Methods

  • Study 1 was a randomized field experiment in an upper-year “Introduction to Databases” course during Spring 2023. Of 218 enrolled students, 145 completed a take-home multiple-choice assignment with a GPT-3-based tutor; half were then randomly assigned to LLM-guided self-reflection and half to no reflection. Outcomes included a proctored exam two weeks later, self-confidence ratings, and subjective ratings; analysis used Mann-Whitney U tests, paired t-tests, Wilcoxon rank sum tests, and thematic analysis of chat logs.
  • Study 2 was a randomized field experiment in a first-year “Introduction to Computer Programming” course during Fall 2023. Of 1068 enrolled students, 112 clicked an optional reflection link and were roughly uniformly assigned to questionnaire-based reflection, LLM-based reflection (GPT-3.5-Turbo), or review of curated lecture slides. A proctored exam on the same subject matter was administered two weeks later, and exam scores were compared with ANOVA.
  • The LLM configurations are reported in Appendix A: Study 1 used text-davinci-003 with temperature 0, max tokens 300, top-p 1, frequency penalty 0, presence penalty 0.6; Study 2 used gpt-35-turbo-16k with temperature 0, max tokens 3925, top-p 0, frequency penalty 0.05, presence penalty 0.1.

Limitations

  • Stated in the paper: the studies did not evaluate LLM accuracy during reflection, and the authors note the risk of sycophantic incorrect validation of student responses. Study 2’s optional reflection activity had low engagement: only 10.48% of enrolled students clicked the link. The authors also state that small-to-medium classroom field studies limit broad statistical generalizations, that Study 2 may have had insufficient students per condition to detect confidence differences, and that the studies examined only a single-session reflection exercise using older GPT-3 and GPT-3.5 models.
  • Evident from the text: Study 1’s exam-score difference was not statistically significant (p=0.1008), and its self-confidence increase was close to the 0.05 threshold (p=0.046), which the authors themselves call preliminary. Study 2’s exam-score differences across conditions were also not statistically significant (p=0.394), so the apparent advantage of questionnaire and LLM reflection over slide review should be interpreted cautiously.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3657604.3662042

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Stimuli for Study-1. A) Half of the students got link to the reflection bot after completing their assignment. B) Example chat window for reflection.
Figure 1 Stimuli for Study-1. A) Half of the students got link to the reflection bot after completing their assignment. B) Example chat window for reflection.From the PDF
Figure 2 Comparative Analysis of Student Outcomes in Reflection vs. No-Reflection Conditions for Study 1. The left panel presents the mean final exam scores obtained two weeks post-assignment, indicating higher performance among students in the reflection group. The center panel assesses the perceived helpfulness of the LLM-tutor for other topics, as influenced by the assigned condition. Finally, the right panel evaluates the willingness of the students to interact again with the LLM tutor, highlighting a greater inclination among those in the reflection group to seek further interaction. Error bars represent standard errors.
Figure 2 Comparative Analysis of Student Outcomes in Reflection vs. No-Reflection Conditions for Study 1. The left panel presents the mean final exam scores obtained two weeks post-assignment, indicating higher performance among students in the reflection group. The center panel assesses the perceived helpfulness of the LLM-tutor for other topics, as influenced by the assigned condition. Finally, the right panel evaluates the willingness of the students to interact again with the LLM tutor, highlighting a greater inclination among those in the reflection group to seek further interaction. Error bars represent standard errors.From the PDF
Figure 3 Change in students' self-confidence from the beginning to the end of the assignment based on their engagement in reflection activities. Error bars represent standard error.
Figure 3 Change in students' self-confidence from the beginning to the end of the assignment based on their engagement in reflection activities. Error bars represent standard error.From the PDF
Image 4
From the PDF
Image 5
From the PDF
Figure 5 Average score of students on the assignment topic, in a proctored exam, conducted 2 weeks after the assignment. In total, 112 students participated in the optional reflection exercise and were uniformly distributed across conditions. We find that students who engaged in a questionnaire-based and LLM-based reflection exercise seemed to perform better than students in the revision of the important slides condition.
Figure 5 Average score of students on the assignment topic, in a proctored exam, conducted 2 weeks after the assignment. In total, 112 students participated in the optional reflection exercise and were uniformly distributed across conditions. We find that students who engaged in a questionnaire-based and LLM-based reflection exercise seemed to perform better than students in the revision of the important slides condition.From the PDF
Change in Self-Confidence based on Engagement in Reflection Activity in Study-2
Change in Self-Confidence based on Engagement in Reflection Activity in Study-2From the PDF

Verification

Claims the paper makes

1 Valid · 3 Not valid · 7 Not checked · 55 Own results and interpretations

23 more claims

Create a free account to see all 66 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 24
  • The quote only partly states this claim. 5

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • ConsistentF(2, 105) = 0.94reported p = .394 · recomputed p = .394

    “F(2, 105) = 0.94, p = .394”

1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 48 references

Research works printed but not linked: 7

  1. Roger Azevedo, Amy Witherspoon, Amber Chauncey, Candice Burkett, and Ashley Fike. 2009. MetaTutor: A MetaCognitive tool for enhancing self-regulated learning. In 2009 AAAI Fall symposium series.
  2. 10.1007/s11409-007--9009--6 · doi:10.1007/s11409-007--9009--6
  3. Virginia Braun, Victoria Clarke, Harris Cooper, Paul M Camic, Debra L Long, AT Panter, David Rindskopf, and Kenneth J Sher. 2012. APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. American Psychological Association , Vol. 2 (2012), 57--71.
  4. 10.1007/s11409-021-09289--1 · doi:10.1007/s11409-021-09289--1
  5. 10.1038/d41586-023-00288--7 · doi:10.1038/d41586-023-00288--7
  6. Leslie Owen Wilson. 2016. Anderson and Krathwohl--Bloom's taxonomy revised. Understanding the new version of Bloom's taxonomy (2016).
  7. 10.1007/s11409-007--9019--4 · doi:10.1007/s11409-007--9019--4

Cited by

See all 53

Metadata

Authors
Kumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A. N., Stamper, J., & Liut, M.
Affiliations
  • University of Toronto (CA)
  • Carnegie Mellon University (US)
  • University of Toronto, Toronto, ON, Canada
  • Carleton College (US)
Venue
Proceedings of the Eleventh ACM Conference on Learning @ Scale
Publisher
ACM
Published
2024-07-09
Pages
86-97
OpenAlex
W4400643163
Cited by
53
References
34
Citations by year
2026: 26 · 2025: 25 · 2024: 2
FWCI
20.18
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Learning @ Scale (L@S) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Natural Sciences and Engineering Research Council of Canada (RGPIN-2019-0696)
  • Office of Naval Research (N00014-21-1- 2576)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Social Sciences › Education › Reflective Practices in Education
Keywords
self-reflection, large language models, metacognition, post-lesson reflection, computer science education, undergraduate students, self-confidence, learning at scale

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms. https://app.cochat.ai/papers/10.1145/3657604.3662042
The paper
Kumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A. N., Stamper, J., & Liut, M. (2024). Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms. Proceedings of the Eleventh ACM Conference on Learning @ Scale. https://doi.org/10.1145/3657604.3662042

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.