The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.
Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms
Open accessAbstract
Two randomized field experiments in undergraduate CS courses evaluated LLM-guided self-reflection. In Experiment 1 (N=145), post-lesson LLM reflection increased student self-confidence and showed a positive trend on later assessments. In Experiment 2 (N=112), LLM reflection and questionnaire reflection performed equally well, significantly outperforming passive review of lecture slides on a proctored exam two weeks later.
Overview
CoChat Summary
TL;DR
Two randomized field experiments in undergraduate computer science courses tested whether LLM-guided post-assignment self-reflection improves learning outcomes and self-efficacy. Study 1 (N=145) found higher self-confidence and greater willingness to interact with the LLM again, but no statistically significant exam-score gain; Study 2 (N=112) found questionnaire-based and LLM-based reflection groups performed comparably and apparently better than slide review, though the exam differences were not statistically significant.
Key findings
- In Study 1, students assigned to LLM-guided reflection had a slightly higher mean exam score two weeks later (57.27) than students with no reflection (53.99), but the difference was not statistically significant (p=0.1008). Homework means were nearly identical (59.09 vs. 59.07).
- Study 1 self-confidence increased from 4.18 to 4.56 in the reflection condition (paired t-test: t=-2.03, df=65, p=0.046), while the no-reflection condition showed no apparent change (4.04 to 4.15; t=-0.64, df=78, p=0.5269). The authors treat the confidence result as preliminary because it is close to the 0.05 threshold. Students in the reflection condition also reported higher willingness to interact with the LLM again (Wilcoxon rank sum: W=3097.00, p=0.046), while both groups found the LLM tutor equally helpful (W=2706.00, p=0.689).
- In Study 2, exam means were 68.30 for questionnaire-based reflection, 66.91 for LLM-based reflection, and 62.25 for revision of important slides; the difference was not significant (F(2,105)=0.94, p=0.394; Eta2=0.02). No differences in self-confidence before and after reflection were found.
- Qualitative analysis of Study 1 chat logs identified themes of positive feedback and affirmations, expanding on answers, offering guidance, and encouraging reflection; the LLM was sometimes inconsistent in prompting students to reflect.
Methods
- Study 1 was a randomized field experiment in an upper-year “Introduction to Databases” course during Spring 2023. Of 218 enrolled students, 145 completed a take-home multiple-choice assignment with a GPT-3-based tutor; half were then randomly assigned to LLM-guided self-reflection and half to no reflection. Outcomes included a proctored exam two weeks later, self-confidence ratings, and subjective ratings; analysis used Mann-Whitney U tests, paired t-tests, Wilcoxon rank sum tests, and thematic analysis of chat logs.
- Study 2 was a randomized field experiment in a first-year “Introduction to Computer Programming” course during Fall 2023. Of 1068 enrolled students, 112 clicked an optional reflection link and were roughly uniformly assigned to questionnaire-based reflection, LLM-based reflection (GPT-3.5-Turbo), or review of curated lecture slides. A proctored exam on the same subject matter was administered two weeks later, and exam scores were compared with ANOVA.
- The LLM configurations are reported in Appendix A: Study 1 used text-davinci-003 with temperature 0, max tokens 300, top-p 1, frequency penalty 0, presence penalty 0.6; Study 2 used gpt-35-turbo-16k with temperature 0, max tokens 3925, top-p 0, frequency penalty 0.05, presence penalty 0.1.
Limitations
- Stated in the paper: the studies did not evaluate LLM accuracy during reflection, and the authors note the risk of sycophantic incorrect validation of student responses. Study 2’s optional reflection activity had low engagement: only 10.48% of enrolled students clicked the link. The authors also state that small-to-medium classroom field studies limit broad statistical generalizations, that Study 2 may have had insufficient students per condition to detect confidence differences, and that the studies examined only a single-session reflection exercise using older GPT-3 and GPT-3.5 models.
- Evident from the text: Study 1’s exam-score difference was not statistically significant (p=0.1008), and its self-confidence increase was close to the 0.05 threshold (p=0.046), which the authors themselves call preliminary. Study 2’s exam-score differences across conditions were also not statistically significant (p=0.394), so the apparent advantage of questionnaire and LLM reflection over slide review should be interpreted cautiously.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
1 Valid · 3 Not valid · 7 Not checked · 55 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Despite the considerable potential of LLMs to improve educational practices at scale, the novelty of this technology means that research on its efficacy, particularly in promoting specific educational strategies such as reflection techniques, remains sparse [35].”
What the sources say
- Could not verifycites [35] [35] San Murugesan and Aswani Kumar Cherukuri. 2023. The Rise of Generative Artificial Intelligence and Its Impact on Education: The Promises and Perils. Computer 56, 5 (2023), 116-121. - sourceExperimental — accuracy not yet measured
Our copy of Murugesan & Cherukuri (2023) is a perspective piece: the closest passages are CHATBOTS' LIMITATIONS AND CONCERNS ('LLM-based chatbots are still in their early stages'), the Abstract ('Opinions and viewpoints vary widely and are changing as we try to use them in unseen ways... is a moot point') and Research and development ('Several issues on the implications of GAI on research and development need further discussion'). None of these states that research on LLM efficacy is sparse, and no passage mentions reflection techniques or reflective learning at all (searched: efficacy/research sparse/limited empirical studies, reflection techniques/metacognition, novelty/early stages). The claim's assertion about the state of the literature is therefore not addressed by our copy. (LLM, conf=0.62)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, students who were also given the opportunity to engage in reflection with the tutor reported a higher willingness to interact with the LLM tutor again, compared to students who only used the tutor to solve problems (Wilcoxon rank sum test: 𝑊 = 3097 . 00 , 𝑝 = 0 . 046).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that both groups of students found the LLM tutor equally helpful (Wilcoxon rank sum test 𝑊 = 2706 . 00 , 𝑝 = 0 . 689).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not find any differences in self-confidence before and after the reflection exercise.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that students assigned to the LLM-based reflection condition performed on a level similar to that of students in the static questionnaire condition and apparently better than students who were assigned to revise the curated set of slides.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 66 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 24
- The quote only partly states this claim. 5
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(2, 105) = 0.94reported p = .394 · recomputed p = .394
“F(2, 105) = 0.94, p = .394”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Experimental Design · p. 8
“On clicking the link, students were randomly assigned to one of three ways to reflect.”
§ Performance in Final Exam in Study-2 · p. 9
“At the beginning of the assignment, they were randomly assigned to one of the three conditions described earlier.”
Citations
Cites
Reciprocal Teaching of Comprehension-Fostering and Comprehension-Monitoring Activities
Palinscar, A. S., & Brown, A. L. (1984) · Cognition and Instruction · doi:10.1207/s1532690xci0102_1 · Cited by 5,186 · cited 2×
“Researchers have documented the extensive benefits of reflection practices, facilitated through various prompts and methodologies [8, 17, 23, 29, 34, 35, 37, 40, 42, 44].”
BAİDOO-ANU, D., & OWUSU ANSAH, L. (2023) · Journal of AI · doi:10.61969/jai.1337500 · Cited by 3,939
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842
ChatGPT for Language Teaching and Learning
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · RELC Journal · doi:10.1177/00336882231162868 · Cited by 1,371
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
Dergaa, I., Chamari, K., Zmijewski, P., & Ben Saad, H. (2023) · Biology of Sport · doi:10.5114/biolsport.2023.125623 · Cited by 796 · cited 1×
“This in addition to much work that has demonstrated the capability of LLMs in aiding with academic writing and translation [15, 19, 26].”
BackgroundEliciting Self‐Explanations Improves Understanding
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994) · Cognitive Science · doi:10.1207/s15516709cog1803_3 · Cited by 757
Supporting Self-Regulated Learning in Online Learning Environments and MOOCs: A Systematic Review
Wong, J., Baars, M., Davis, D., Van Der Zee, T., Houben, G.-J., & Paas, F. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1543084 · Cited by 631
Scaffolding students' knowledge integration: prompts for reflection in KIE
Davis, E. A. (2000) · International Journal of Science Education · doi:10.1080/095006900412293 · Cited by 531 · cited 2×
“This is achieved by promoting connections between disparate concepts within academic material [6, 12, 13], encouraging the elaboration of learned ideas [12], and facilitating error detection and correction [11, 12].”
Role play with large language models
Shanahan, M., McDonell, K., & Reynolds, L. (2023) · Nature · doi:10.1038/s41586-023-06647-8 · Cited by 491
Azevedo, R., & Hadwin, A. F. (2005) · Instructional Science · doi:10.1007/s11251-005-1272-9 · Cited by 469
Bielaczyc, K., Pirolli, P. L., & Brown, A. L. (1995) · Cognition and Instruction · doi:10.1207/s1532690xci1302_3 · Cited by 422 · cited 3×
“This is achieved by promoting connections between disparate concepts within academic material [6, 12, 13], encouraging the elaboration of learned ideas [12], and facilitating error detection and correction [11, 12].”
GPTeach: Interactive TA Training with GPT-based Students
Markel, J. M., Opferman, S. G., Landay, J. A., & Piech, C. (2023) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3573051.3593393 · Cited by 276
Bannert, M., & Mengelkamp, C. (2008) · Metacognition and Learning · doi:10.1007/s11409-007-9009-6 · Cited by 198 · cited 2×
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundThe student activity meter for awareness and self-reflection
Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012) · CHI '12 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/2212776.2212860 · Cited by 193 · cited 2×
“Researchers have documented the extensive benefits of reflection practices, facilitated through various prompts and methodologies [8, 17, 23, 29, 34, 35, 37, 40, 42, 44].”
Streaming video to enhance students’ reflection in dance education
Leijen, Ä., Lam, I., Wildschut, L., Robert-Jan Simons, P., & Admiraal, W. (2009) · Computers & Education · doi:10.1016/j.compedu.2008.07.010 · Cited by 177 · cited 2×
“Researchers have documented the extensive benefits of reflection practices, facilitated through various prompts and methodologies [8, 17, 23, 29, 34, 35, 37, 40, 42, 44].”
Explicit teaching of meta-strategic knowledge in authentic classroom situations
Zohar, A., & David, A. B. (2008) · Metacognition and Learning · doi:10.1007/s11409-007-9019-4 · Cited by 175
The development of discourse strategies in adolescent argumentation
Felton, M. K. (2003) · Cognitive Development · doi:10.1016/j.cogdev.2003.09.001 · Cited by 144 · cited 1×
“The underlying mechanism for these benefits involves structured metacognition examinations, coupled with opportunities for feedback from instructors or peers [15, 20, 48].”
Revisiting Reflection in HCI: Four Design Resources for Technologies that Support Reflection
Bentvelzen, M., Woźniak, P. W., Herbes, P. S. F., Stefanidi, E., & Niess, J. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3517233 · Cited by 137
Examining transfer effects from dialogic discussions to new tasks and contexts
Reznitskaya, A., Glina, M. B., Carolan, B. V., Michaud, O., Rogers, J. G., & Sequeira, L. (2012) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2012.02.003 · Cited by 128 · cited 2×
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundChen, P., Chavez, O., Ong, D. C., & Gunderson, B. (2017) · Psychological Science · doi:10.1177/0956797617696456 · Cited by 122 · cited 1×
“Specifically, the literature demonstrates that encouraging students to engage in reflection activities after interacting with the course materials can lead to gains in knowledge and skill acquisition [9, 32].”
Promoting and scaffolding argumentation through reflective asynchronous discussions
Lin, H., Hong, Z., & Lawrenz, F. P. (2012) · Computers & Education · doi:10.1016/j.compedu.2012.01.019 · Cited by 118 · cited 2×
“Such reflective activities encourage students to revisit and consolidate what they have learned, fostering a deeper understanding and facilitating the application of knowledge in different contexts [12, 22, 32].”
BackgroundEvaluating Interpretive Research in HCI
Soden, R., Toombs, A., & Thomas, M. (2024) · Interactions · doi:10.1145/3633200 · Cited by 92
The Rise of Generative Artificial Intelligence and Its Impact on Education: The Promises and Perils
Murugesan, S., & Cherukuri, A. K. (2023) · Computer · doi:10.1109/mc.2023.3253292 · Cited by 92
An Interaction Design for Machine Teaching to Develop AI Tutors
Weitekamp, D. P., Harpstead, E., & Koedinger, K. R. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376226 · Cited by 78 · cited 1×
“These include aiding in grading [4], providing emotional or mental health support to students through text message based messaging systems, and functioning in capacities similar to teaching assistants capable of answering questions on course content [10, 31, 45].”
Pedagogical Agents: Back to the Future
Johnson, W. L., & Lester, J. C. (2018) · AI Magazine · doi:10.1609/aimag.v39i2.2793 · Cited by 77 · cited 1×
“Our work contributes to a wider body of work that has focused on how to support students in their learning strategies [22], and research that has looked at different levels of support needed and how that level of support impacts learners in both current and future interactions.”
Supporting strategic and meta-strategic development of argument skill: the role of reflection
Iordanou, K. (2022) · Metacognition and Learning · doi:10.1007/s11409-021-09289-1 · Cited by 64
Gerace, A., Day, A., Casey, S., & Mohr, P. (2017) · Journal of Relationships Research · doi:10.1017/jrr.2017.8 · Cited by 55 · cited 1×
“These were derived from established reflection and awareness questionnaires [16].”
Antonietti, A., Colombo, B., & Nuzzo, C. D. (2014) · Learning Media and Technology · doi:10.1080/17439884.2014.933112 · Cited by 54 · cited 1×
“Similar to reflection, researchers have explored how to support students in self-regulated learning and developing metacognitive skills [1–3].”
BackgroundMath Education with Large Language Models: Peril or Promise?
Kumar, H., Rothschild, D. M., Goldstein, D. G., & Hofman, J. (2023) · doi:10.2139/ssrn.4641653 · Cited by 54 · cited 2×
“Early evidence suggests that access to these personalized conversational AI tools could not only support knowledge acquisition [28] but also facilitate a growth mindset in learners [20], making them a valuable asset in the educational toolkit.”
BackgroundSelf-citationPrather, J. E., Denny, P. C., Leinonen, J., Becker, B. A., Al-Bluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587103.3594206 · Cited by 47 · cited 1×
“LLMs that are capable of generating computer code with relative ease have in particular believed to be highly applicable to aiding computer science students or students in other fields who lack a computer science background [38].”
MethodFrom Theory of Mind to Epistemic Cognition. A Lifespan perspective
Iordanou, K. (2016) · Frontline Learning Research · doi:10.14786/flr.v4i5.252 · Cited by 38
Webb, A. M., Linder, R., Kerne, A., Lupfer, N., Qu, Y., Poffenberger, B., & Revia, C. (2013) · Creativity and Cognition · doi:10.1145/2466627.2466636 · Cited by 35 · cited 2×
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundEnhancing evidence-based argumentation in a Mainland China middle school
Shi, Y. (2019) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2019.101809 · Cited by 35
GPTeach: Interactive TA Training with GPT-based StudentsPreprint
Markel, J. M., Opferman, S. G., Landay, J. A., et al. (2023) · doi:10.35542/osf.io/r23bu · Cited by 33
Menekse, M., Anwar, S., & Akdemir, Z. G. (2022) · The Journal of Experimental Education · doi:10.1080/00220973.2020.1786346 · Cited by 32 · cited 1×
“Specifically, the literature demonstrates that encouraging students to engage in reflection activities after interacting with the course materials can lead to gains in knowledge and skill acquisition [10, 34].”
BackgroundAmplifying reflective thinking in musical performance
Johnston, A., Amitani, S., & Edmonds, E. A. (2005) · ACM Conference on Creativity & Cognition (C&C) · doi:10.1145/1056224.1056248 · Cited by 22 · cited 2×
“Researchers have documented the extensive benefits of reflection practices, facilitated through various prompts and methodologies [8, 17, 23, 29, 34, 35, 37, 40, 42, 44].”
Camera talk: making the camera a partial participant
Lamberty, K. K., & Kolodner, J. L. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055090 · Cited by 12 · cited 1×
“Researchers such as Lamberty and Koloder [30] who had students speak out loud to a camera when describing their work noted the reflective nature of their comments as well as the benefits to their mathematical understanding.”
BackgroundKumar, H., Yu, K., Chung, A., Shi, J., & Williams, J. J. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545947.3576285 · Cited by 7
Self-citationTseng, T., & Bryant, C. (2013) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2468356.2468466 · Cited by 5 · cited 2×
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundVisualizing digital media interactions: providing feedback on jam2jam AV performances
Brown, A. R. (2010) · Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction · doi:10.1145/1952222.1952264 · Cited by 5 · cited 2×
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundHigh-Resolution Course Feedback: Timely Feedback Mechanism for Instructors
Kim, Y., & Piech, C. (2023) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3573051.3593391 · Cited by 5 · cited 1×
“LLMs and conversational AI systems have been acknowledged as game changers for science, and many have theorized about their potential applications in the field of education [25, 28, 45].”
BackgroundHanda, K., Clapper, M., Boyle, J., Wang, R. E., Yang, D., Yeager, D. S., & Demszky, D. (2023) · arXiv · doi:10.48550/arxiv.2310.10637 · Cited by 5
The effects of reflection on seventh grade social studies student learning and perception
Denton, D. W., & Ellis, A. K. (2023) · Educational Studies · doi:10.1080/03055698.2020.1835616 · Cited by 4 · cited 4×
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundInfluentialNelson, E. J., & Freier, N. G. (2008) · Proceedings of the 7th international conference on Interaction design and children · doi:10.1145/1463689.1463738 · Cited by 3 · cited 2×
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundRECIPE: How to Integrate ChatGPT into EFL Writing Education
Han, J., Yoo, H., Kim, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.11583
Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception
Kumar, H., Musabirov, I., Reza, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.13712
Self-citation
Research works printed but not linked: 7
- Roger Azevedo, Amy Witherspoon, Amber Chauncey, Candice Burkett, and Ashley Fike. 2009. MetaTutor: A MetaCognitive tool for enhancing self-regulated learning. In 2009 AAAI Fall symposium series.
- 10.1007/s11409-007--9009--6 · doi:10.1007/s11409-007--9009--6
- Virginia Braun, Victoria Clarke, Harris Cooper, Paul M Camic, Debra L Long, AT Panter, David Rindskopf, and Kenneth J Sher. 2012. APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. American Psychological Association , Vol. 2 (2012), 57--71.
- 10.1007/s11409-021-09289--1 · doi:10.1007/s11409-021-09289--1
- 10.1038/d41586-023-00288--7 · doi:10.1038/d41586-023-00288--7
- Leslie Owen Wilson. 2016. Anderson and Krathwohl--Bloom's taxonomy revised. Understanding the new version of Bloom's taxonomy (2016).
- 10.1007/s11409-007--9019--4 · doi:10.1007/s11409-007--9019--4
Cited by
Wu, F., Dang, Y., & Li, M. (2025) · Behavioral Sciences · doi:10.3390/bs15040467 · Cited by 176
“In Kumar et al. (2024)’s study, both groups of students had access to an LLM tutor interface to solve assignment problems.”
Metacognition meets AI : Empowering reflective writing with large language models
Neshaei, S. P., Mejia‐Domenzain, P., Davis, R. L., & Käser, T. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13601 · Cited by 39
Song, I., Park, S., Pendse, S. R., Schleider, J. L., Choudhury, M. D., & Kim, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713883 · Cited by 77
“Further, a growing body of research has investigated the LLMs’ potential for mental health support in the form of conventional chat interfaces ( e.g. , [47, 52, 105, 113]).”
PersonaFlow: Designing LLM-Simulated Expert Perspectives for Enhanced Research Ideation
Liu, Y., Sharma, P., Oswal, M., Xia, H., & Huang, Y. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735789 · Cited by 86
Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recovery
Choi, R., Kim, T., Park, S., Kim, J. G., & Lee, S.-J. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713485 · Cited by 19
Metadata
- Authors
- Kumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A. N., Stamper, J., & Liut, M.
- Affiliations
- University of Toronto (CA)
- Carnegie Mellon University (US)
- University of Toronto, Toronto, ON, Canada
- Carleton College (US)
- ORCID
- Venue
- Proceedings of the Eleventh ACM Conference on Learning @ Scale
- Publisher
- ACM
- Published
- 2024-07-09
- Pages
- 86-97
- OpenAlex
- W4400643163
- Cited by
- 53
- References
- 34
- Citations by year
- 2026: 26 · 2025: 25 · 2024: 2
- FWCI
- 20.18
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Learning @ Scale (L@S) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Natural Sciences and Engineering Research Council of Canada (RGPIN-2019-0696)
- Office of Naval Research (N00014-21-1- 2576)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Education › Reflective Practices in Education
- Keywords
- self-reflection, large language models, metacognition, post-lesson reflection, computer science education, undergraduate students, self-confidence, learning at scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.