Investigating the Efficacy of Large Language Models in Reflective Assessment Methods through Chain of Thought Prompting
Open accessAbstract
Large Language Models, such as Generative Pre-trained Transformer 3 (aka. GPT-3), have been developed to understand language through the analysis of extensive text data, allowing them to identify patterns and connections between words. While LLMs have demonstrated impressive performance across various text-related tasks, they encounter challenges in tasks associated with reasoning. To address this challenge, Chain of Thought (CoT) prompting method has been proposed as a means to enhance LLMs’ proficiency in complex reasoning tasks like solving math word problems and answering questions based on logical argumentative reasoning. The primary aim of this research is to assess how well four language models can grade reflective essays of third-year medical students. The assessment will specifically target the evaluation of critical thinking skills using CoT prompting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI was used to grade exams, particularly those involving short-answer, long-answer, and essay-based responses.
“AI was also used to grade exams, particularly those that involve short-answer, long-answer, and essay-based responses [137–139].”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [137]
Section 4.1 (Observations) says the authors used language models to grade reflective essays, supporting the essay-based part of the claim. The passages do not show that this study graded short-answer or long-answer exam responses. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 12 · Cited by 13
Research works printed but not linked: 4
- Samy A Azer. 2008. Use of portfolios by medical students: significance of critical thinking. The Kaohsiung journal of medical sciences 24, 7 (2008), 361-366.
- Beata Beigman Klebanov, Jill Burstein, Judith M Harackiewicz, Stacy J Priniski, and Matthew Mulholland. 2017. Reflective writing about the utility value of science as a tool for increasing STEM motivation and retention-can AI help scale up? International Journal of Artificial Intelligence in Education 27 (2017), 791-818.
- Afrizal Doewes, Nughthoh Kurdhi, and Akrati Saxena. 2023. Evaluating Qua- dratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring. In 16th International Conference on Educational Data Mining, EDM 2023. International Educational Data Mining Society (IEDMS), 103-113.
- Nikhil S Patil, Ryan S Huang, Christian B van der Pol, and Natasha Larocque. 2023. Comparative performance of ChatGPT and bard in a text-based radiology knowledge assessment. Canadian Association of Radiologists Journal (2023), 08465371231193716.
- Authors
- Masikisiki, B., Marivate, V., & Hlophe, Y.
- Affiliations
- University of Pretoria (ZA)
- Venue
- Proceedings of the 4th African Human Computer Interaction Conference
- Publisher
- ACM
- Published
- 2023-11-27
- Pages
- 44-49
- OpenAlex
- W4391367446
- Cited by
- 13
- References
- 12
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 2
- FWCI
- 3.34
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- African Human Computer Interaction Conference (AfriCHI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Education › Student Assessment and Feedback
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, chain-of-thought prompting, automated essay scoring, critical thinking skills, medical students, GPT-3, reasoning tasks, medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
