CoChat
Sign Up
Cited by 110Artificial Intelligence in EducationStudent Assessment and FeedbackIntelligent Tutoring Systems and Adaptive Learning

Exploring generative AI assisted feedback writing for students’ written responses to a physics conceptual question with prompt engineering and few-shot learning

Open access
Wan, Tong, Chen, Zhongzhou·2024·Physical Review Physics Education Research
DOI 10.1103/physrevphyseducres.20.010152
Open this paper in CoChat — free account

Abstract

Instructor’s feedback plays a critical role in students’ development of conceptual understanding and reasoning skills. However, grading student written responses and providing personalized feedback can take a substantial amount of time, especially in large enrollment courses. In this study, we explore using GPT-3.5 to write feedback on students’ written responses to conceptual questions with prompt engineering and few-shot learning techniques. In stage I, we used a small portion ( n = 2 0 ) of the student responses on one conceptual question to iteratively train GPT to generate feedback. Four of the responses paired with human-written feedback were included in the prompt as examples for GPT. We tasked GPT to generate feedback for another 16 responses and refined the prompt through several iterations. In stage II, we gave four student researchers (one graduate and three undergraduate researchers) the 16 responses as well as two versions of feedback, one written by the authors and the other by GPT. Students were asked to rate the correctness and usefulness of each feedback and to indicate which one was generated by GPT. The results showed that students tended to rate the feedback by human and GPT equally on correctness, but they all rated the feedback by GPT as more useful. Additionally, the success rates of identifying GPT’s feedback were low, ranging from 0.1 to 0.6. In stage III, we tasked GPT to generate feedback for the rest of the students’ responses ( n = 6 5 ). The feedback messages were rated by four instructors based on the extent of modification needed if they were to give the feedback to students. All four instructors rated approximately 70% (ranging from 68% to 78%) of the feedback statements needing only minor or no modification. This study demonstrated the feasibility of using generative artificial intelligence (AI) as an assistant to generate feedback for student written responses with only a relatively small number of examples in the prompt. An AI assistant can be one of the solutions to substantially reduce time spent on grading student written responses. Published by the American Physical Society 2024

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1103/physrevphyseducres.20.010152

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • One study reported a 34.6% reduction in grading time due to iterative prompts.

    “one study noted a 34.6% reduction in grading time due to iterative prompts.”

    From A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education (2025) · cites this paper as Wan and Chen, 2024

    The Abstract (p. 1) says the researchers refined their prompt over several iterations, and the Discussion (p. 14) suggests the approach could save grading time. The paper does not report a measured 34.6% reduction in grading time; its instructor ratings assessed how much editing the generated feedback would need (Stage 3, p. 8). (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • One study on grading support achieved a 34.6% reduction in grading time through iterative prompts.

    “One study on grading support achieved a 34.6% reduction in grading time through iterative prompts”

    From A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education (2025) · cites this paper as Wan and Chen, 2024

    The Study Design (p. 6) describes iteratively refining a prompt for grading assistance. The Discussion (p. 14) suggests this could save grading time, but reports no measured 34.6% reduction; instructor evaluations instead assessed how much modification the feedback needed (p. 8). (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. [1][1] R. Beichner, An Introduction to Physics Education Research , in Getting Started in PER , edited by C. Henderson and K. Harper, Vol. 2 (2009). -
  2. [10][10] C. Sung, T. Ma, T. I. Dhamecha, V. Reddy, S. Saha, and R. Arora, Pre-Training BERT on Domain Resources for Short Answer Grading , EMNLP-IJCNLP 2019 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference 6071 (2019). -
  3. [26][26] E. Radiya-Dixit and X. Wang, How Fine Can Fine-Tuning Be? Learning Efficient Language Models , in Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , edited by S. Chiappa and R. Calandra (PMLR, 2020), pp. 2435-2443. -
  4. [29][29] S. Wheeler and R. E. Scherr, ChatGPT Reflects Student Misconceptions in Physics , in 2023 Physics Education Research Conference Proceedings (American Association of Physics Teachers, 2023), pp. 386-390. -
  5. [36][36] M. Zhang, O. Press, W. Merrill, A. Liu, N. A. Smith, and P. G. Allen, How Language Model Hallucinations Can Snowball , (2023). -
  6. [37][37] D. Bilbao, A. Gelbukh, A. Rodrigo, and M. Lee, A Mathematical Investigation of Hallucination and Creativity in GPT Models , Mathematics 2023, Vol. 11, Page 2320 11 , 2320 (2023). -
  7. [39][39] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? , FAccT 2021 - Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 610 (2021).

Cited by

Sorted by citations

Related

Authors
Wan, T., & Chen, Z.
Affiliations
University of Central Florida (US)
ORCID
Venue
Physical Review Physics Education Research
Publisher
American Physical Society (APS)
Published
2024-06-13
Volume
20
Issue
1
ISSN
2469-9896 (ISSN-L)
DOI
10.1103/physrevphyseducres.20.010152
OpenAlex
W4399627833
Cited by
109
References
25
Citations by year
2026: 50 · 2025: 51 · 2024: 8
FWCI
41.11
Type
journal article
Language
en
Open access
Licence
Locations
  • Physical Review Physics Education Research (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • University of Central Florida
  • University of Central Florida Digital Curriculum Innovation Initiative
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Social Sciences › Education › Student Assessment and Feedback
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
generative AI, automated feedback, student written responses, prompt engineering, few-shot learning, GPT-3.5, personalized feedback, conceptual understanding, large-enrollment courses

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review