Cited by 59Artificial Intelligence in EducationStudent Assessment and FeedbackIntelligent Tutoring Systems and Adaptive Learning

The promises and pitfalls of large language models as feedback providers: A study of prompt engineering and the quality of AI-driven feedback

Open access
Jacobsen, Lucas Jasper, Weber, Kira Elena·2025·AI

Abstract

Background/Objectives: Artificial intelligence (AI) is transforming higher education (HE), reshaping teaching, learning, and feedback processes. Feedback generated by large language models (LLMs) has shown potential for enhancing student learning outcomes. However, few empirical studies have directly compared the quality of LLM feedback with feedback from novices and experts. This study investigates (1) the types of prompts needed to ensure high-quality LLM feedback in teacher education and (2) how feedback from novices, experts, and LLMs compares in terms of quality. Methods: To address these questions, we developed a theory-driven manual to evaluate prompt quality and designed three prompts of varying quality. Feedback generated by ChatGPT-4 was assessed alongside feedback from novices and experts, who were provided with the highest-quality prompt. Results: Our findings reveal that only the best prompt consistently produced high-quality feedback. Additionally, LLM feedback outperformed novice feedback and, in the categories explanation, questions, and specificity, even surpassed expert feedback in quality while being generated more quickly. Conclusions: These results suggest that LLMs, when guided by well-crafted prompts, can serve as high-quality and efficient alternatives to expert feedback. The findings underscore the importance of prompt quality and emphasize the need for prompt design guidelines to maximize the potential of LLMs in teacher education.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/ai6020035

Cites

48 of 51 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. ChatGPT; Ekin, S. Prompt engineering for ChatGPT: A quick guide to techniques, tips and best practice. Authorea Prepr. 2023. [CrossRef] · DOI
  2. Weber, K.E.; Prilop, C.N. Videobasiertes Training kollegialen Feedbacks in der Lehrkräftebildung. In Spezifische Aspekte von Trainings pädagogischer Kompetenzen in Abgrenzung zu anderen Lehr-Lern-Situationen in der Lehrkräftebildung: Tagungsband. Online- Tagung an der Uni Rostock am 4. und 5. März 2022 zum Thema "Ist das jetzt schon ein Training? Wie unterscheiden sich Trainings von anderen Lehr-Lern-Situationen in der Lehrkräftebildung?"; Carnein, O., Damnik, G., Krause, G., Vanier, D., Eds.; Publikationsserver RosDok: Rostock, Germany, 2023. [CrossRef] · DOI
  3. Narciss, S. Designing and evaluating tutoring feedback strategies for digital learning environments on the basis of the interactive feedback model. Digit. Educ. Rev. 2013, 23, 7-26.
  4. Spector, J.M., Merrill, M.D., van Merrienboer, J.J.G., Driscoll, M.P., Eds.; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 2008; pp. 125-144.
  5. Miranda, L.J.V.; Wang, Y.; Elazar, Y.; Kumar, S.; Pyatkin, V.; Brahman, F.; Smith, N.A.; Hajishirzi, H.; Dasigi, P. Hybrid preferences: Learning to route instances for human vs. AI feedback. arXiv 2025, arXiv:2410.19133. [CrossRef] · DOI
  6. CrossRef · DOI

Cited by

Sorted by citations

Authors
Jacobsen, L. J., & Weber, K. E.
Affiliations
Universität Hamburg (DE)
Corresponding author
Jacobsen, Lucas Jasper
Venue
AI
Publisher
MDPI AG
Published
2025-02-12
Volume
6
Issue
2
Pages
35
ISSN
2673-2688 (ISSN-L)
OpenAlex
W4407405478
Cited by
58
References
47
Citations by year
2026: 33 · 2025: 25
FWCI
60.81
Type
journal article
Language
en
Open access
Locations
  • AI (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Deutsche Forschungsgemeinschaft
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Social Sciences › Education › Student Assessment and Feedback
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
large language models, AI-generated feedback, prompt engineering, prompt quality, teacher professional development, ChatGPT-4, higher education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review