Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.
Abstract
Two randomized field experiments in undergraduate CS courses evaluated LLM-guided self-reflection. In Experiment 1 (N=145), post-lesson LLM reflection increased student self-confidence and showed a positive trend on later assessments. In Experiment 2 (N=112), LLM reflection and questionnaire reflection performed equally well, significantly outperforming passive review of lecture slides on a proctored exam two weeks later.
Overview
CoChat Summary
TL;DR
Two randomized field experiments in undergraduate computer science courses tested whether LLM-guided post-assignment self-reflection improves learning outcomes and self-efficacy. Study 1 (N=145) found higher self-confidence and greater willingness to interact with the LLM again, but no statistically significant exam-score gain; Study 2 (N=112) found questionnaire-based and LLM-based reflection groups performed comparably and apparently better than slide review, though the exam differences were not statistically significant.
Key findings
- In Study 1, students assigned to LLM-guided reflection had a slightly higher mean exam score two weeks later (57.27) than students with no reflection (53.99), but the difference was not statistically significant (p=0.1008). Homework means were nearly identical (59.09 vs. 59.07).
- Study 1 self-confidence increased from 4.18 to 4.56 in the reflection condition (paired t-test: t=-2.03, df=65, p=0.046), while the no-reflection condition showed no apparent change (4.04 to 4.15; t=-0.64, df=78, p=0.5269). The authors treat the confidence result as preliminary because it is close to the 0.05 threshold. Students in the reflection condition also reported higher willingness to interact with the LLM again (Wilcoxon rank sum: W=3097.00, p=0.046), while both groups found the LLM tutor equally helpful (W=2706.00, p=0.689).
- In Study 2, exam means were 68.30 for questionnaire-based reflection, 66.91 for LLM-based reflection, and 62.25 for revision of important slides; the difference was not significant (F(2,105)=0.94, p=0.394; Eta2=0.02). No differences in self-confidence before and after reflection were found.
- Qualitative analysis of Study 1 chat logs identified themes of positive feedback and affirmations, expanding on answers, offering guidance, and encouraging reflection; the LLM was sometimes inconsistent in prompting students to reflect.
Methods
- Study 1 was a randomized field experiment in an upper-year “Introduction to Databases” course during Spring 2023. Of 218 enrolled students, 145 completed a take-home multiple-choice assignment with a GPT-3-based tutor; half were then randomly assigned to LLM-guided self-reflection and half to no reflection. Outcomes included a proctored exam two weeks later, self-confidence ratings, and subjective ratings; analysis used Mann-Whitney U tests, paired t-tests, Wilcoxon rank sum tests, and thematic analysis of chat logs.
- Study 2 was a randomized field experiment in a first-year “Introduction to Computer Programming” course during Fall 2023. Of 1068 enrolled students, 112 clicked an optional reflection link and were roughly uniformly assigned to questionnaire-based reflection, LLM-based reflection (GPT-3.5-Turbo), or review of curated lecture slides. A proctored exam on the same subject matter was administered two weeks later, and exam scores were compared with ANOVA.
- The LLM configurations are reported in Appendix A: Study 1 used text-davinci-003 with temperature 0, max tokens 300, top-p 1, frequency penalty 0, presence penalty 0.6; Study 2 used gpt-35-turbo-16k with temperature 0, max tokens 3925, top-p 0, frequency penalty 0.05, presence penalty 0.1.
Limitations
- Stated in the paper: the studies did not evaluate LLM accuracy during reflection, and the authors note the risk of sycophantic incorrect validation of student responses. Study 2’s optional reflection activity had low engagement: only 10.48% of enrolled students clicked the link. The authors also state that small-to-medium classroom field studies limit broad statistical generalizations, that Study 2 may have had insufficient students per condition to detect confidence differences, and that the studies examined only a single-session reflection exercise using older GPT-3 and GPT-3.5 models.
- Evident from the text: Study 1’s exam-score difference was not statistically significant (p=0.1008), and its self-confidence increase was close to the 0.05 threshold (p=0.046), which the authors themselves call preliminary. Study 2’s exam-score differences across conditions were also not statistically significant (p=0.394), so the apparent advantage of questionnaire and LLM reflection over slide review should be interpreted cautiously.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
1 Valid · 3 Not valid · 7 Not checked · 55 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Despite the considerable potential of LLMs to improve educational practices at scale, the novelty of this technology means that research on its efficacy, particularly in promoting specific educational strategies such as reflection techniques, remains sparse [35].”
What the sources say
- Could not verifycites [35] [35] San Murugesan and Aswani Kumar Cherukuri. 2023. The Rise of Generative Artificial Intelligence and Its Impact on Education: The Promises and Perils. Computer 56, 5 (2023), 116-121. - sourceExperimental — accuracy not yet measured
Our copy of Murugesan & Cherukuri (2023) is a perspective piece: the closest passages are CHATBOTS' LIMITATIONS AND CONCERNS ('LLM-based chatbots are still in their early stages'), the Abstract ('Opinions and viewpoints vary widely and are changing as we try to use them in unseen ways... is a moot point') and Research and development ('Several issues on the implications of GAI on research and development need further discussion'). None of these states that research on LLM efficacy is sparse, and no passage mentions reflection techniques or reflective learning at all (searched: efficacy/research sparse/limited empirical studies, reflection techniques/metacognition, novelty/early stages). The claim's assertion about the state of the literature is therefore not addressed by our copy. (LLM, conf=0.62)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, students who were also given the opportunity to engage in reflection with the tutor reported a higher willingness to interact with the LLM tutor again, compared to students who only used the tutor to solve problems (Wilcoxon rank sum test: 𝑊 = 3097 . 00 , 𝑝 = 0 . 046).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that both groups of students found the LLM tutor equally helpful (Wilcoxon rank sum test 𝑊 = 2706 . 00 , 𝑝 = 0 . 689).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not find any differences in self-confidence before and after the reflection exercise.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that students assigned to the LLM-based reflection condition performed on a level similar to that of students in the static questionnaire condition and apparently better than students who were assigned to revise the curated set of slides.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 66 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 24
- The quote only partly states this claim. 5
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(2, 105) = 0.94reported p = .394 · recomputed p = .394
“F(2, 105) = 0.94, p = .394”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Experimental Design · p. 8
“On clicking the link, students were randomly assigned to one of three ways to reflect.”
§ Performance in Final Exam in Study-2 · p. 9
“At the beginning of the assignment, they were randomly assigned to one of the three conditions described earlier.”
Citations
Cites
Reciprocal Teaching of Comprehension-Fostering and Comprehension-Monitoring Activities
Palinscar, A. S., & Brown, A. L. (1984) · Cognition and Instruction · doi:10.1207/s1532690xci0102_1 · Cited by 5,186 · cited 2×
“Researchers have documented the extensive benefits of reflection practices, facilitated through various prompts and methodologies [8, 17, 23, 29, 34, 35, 37, 40, 42, 44].”
BAİDOO-ANU, D., & OWUSU ANSAH, L. (2023) · Journal of AI · doi:10.61969/jai.1337500 · Cited by 3,939
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842
ChatGPT for Language Teaching and Learning
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · RELC Journal · doi:10.1177/00336882231162868 · Cited by 1,371
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
Dergaa, I., Chamari, K., Zmijewski, P., & Ben Saad, H. (2023) · Biology of Sport · doi:10.5114/biolsport.2023.125623 · Cited by 796 · cited 1×
“This in addition to much work that has demonstrated the capability of LLMs in aiding with academic writing and translation [15, 19, 26].”
BackgroundEliciting Self‐Explanations Improves Understanding
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994) · Cognitive Science · doi:10.1207/s15516709cog1803_3 · Cited by 757
Supporting Self-Regulated Learning in Online Learning Environments and MOOCs: A Systematic Review
Wong, J., Baars, M., Davis, D., Van Der Zee, T., Houben, G.-J., & Paas, F. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1543084 · Cited by 631
Scaffolding students' knowledge integration: prompts for reflection in KIE
Davis, E. A. (2000) · International Journal of Science Education · doi:10.1080/095006900412293 · Cited by 531 · cited 2×
“This is achieved by promoting connections between disparate concepts within academic material [6, 12, 13], encouraging the elaboration of learned ideas [12], and facilitating error detection and correction [11, 12].”
Research works printed but not linked: 7
- Roger Azevedo, Amy Witherspoon, Amber Chauncey, Candice Burkett, and Ashley Fike. 2009. MetaTutor: A MetaCognitive tool for enhancing self-regulated learning. In 2009 AAAI Fall symposium series.
- 10.1007/s11409-007--9009--6 · doi:10.1007/s11409-007--9009--6
- Virginia Braun, Victoria Clarke, Harris Cooper, Paul M Camic, Debra L Long, AT Panter, David Rindskopf, and Kenneth J Sher. 2012. APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. American Psychological Association , Vol. 2 (2012), 57--71.
- 10.1007/s11409-021-09289--1 · doi:10.1007/s11409-021-09289--1
- 10.1038/d41586-023-00288--7 · doi:10.1038/d41586-023-00288--7
- Leslie Owen Wilson. 2016. Anderson and Krathwohl--Bloom's taxonomy revised. Understanding the new version of Bloom's taxonomy (2016).
- 10.1007/s11409-007--9019--4 · doi:10.1007/s11409-007--9019--4
Cited by
Wu, F., Dang, Y., & Li, M. (2025) · Behavioral Sciences · doi:10.3390/bs15040467 · Cited by 79
“In Kumar et al. (2024)’s study, both groups of students had access to an LLM tutor interface to solve assignment problems.”
Metacognition meets AI : Empowering reflective writing with large language models
Neshaei, S. P., Mejia‐Domenzain, P., Davis, R. L., & Käser, T. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13601 · Cited by 39
Song, I., Park, S., Pendse, S. R., Schleider, J. L., Choudhury, M. D., & Kim, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713883 · Cited by 77
“Further, a growing body of research has investigated the LLMs’ potential for mental health support in the form of conventional chat interfaces ( e.g. , [47, 52, 105, 113]).”
PersonaFlow: Designing LLM-Simulated Expert Perspectives for Enhanced Research Ideation
Liu, Y., Sharma, P., Oswal, M., Xia, H., & Huang, Y. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735789 · Cited by 86
Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recovery
Choi, R., Kim, T., Park, S., Kim, J. G., & Lee, S.-J. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713485 · Cited by 19
Metadata
- Authors
- Kumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A. N., Stamper, J., & Liut, M.
- Affiliations
- University of Toronto (CA)
- Carnegie Mellon University (US)
- University of Toronto, Toronto, ON, Canada
- Carleton College (US)
- ORCID
- Venue
- Proceedings of the Eleventh ACM Conference on Learning @ Scale
- Publisher
- ACM
- Published
- 2024-07-09
- Pages
- 86-97
- OpenAlex
- W4400643163
- Cited by
- 53
- References
- 34
- Citations by year
- 2026: 26 · 2025: 25 · 2024: 2
- FWCI
- 20.18
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Learning @ Scale (L@S) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Natural Sciences and Engineering Research Council of Canada (RGPIN-2019-0696)
- Office of Naval Research (N00014-21-1- 2576)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Education › Reflective Practices in Education
- Keywords
- self-reflection, large language models, metacognition, post-lesson reflection, computer science education, undergraduate students, self-confidence, learning at scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms. https://app.cochat.ai/papers/10.1145/3657604.3662042
- The paper
- Kumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A. N., Stamper, J., & Liut, M. (2024). Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms. Proceedings of the Eleventh ACM Conference on Learning @ Scale. https://doi.org/10.1145/3657604.3662042
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.