The Promises and Pitfalls of LLMs as Feedback Providers: A Study of Prompt Engineering and the Quality of AI-Driven Feedback
Open accessAbstract
Artificial intelligence (AI) in higher education (HE) is reshaping teaching and learning, and feedback provided by large language models (LLMs) seems to have an impact on student learning. However, few empirical studies have compared the quality of LLM feedback with the feedback quality of real persons. Therefore, this study addresses the following questions: What prompts are needed to ensure high-quality LLM feedback in HE? How does feedback from novices, experts, and LLMs differ in terms of quality and content accuracy? We developed a learning goal with three errors and a theory-based manual to evaluate prompt quality. Specifically, three prompts of varying quality were created and used to generate feedback from ChatGPT-4. We provided the highest-quality prompt to novices and experts. Our results showed that only the best prompt produced consistently high-quality feedback. Additionally, LLM and expert feedback were significantly better than novice feedback, with LLM feedback being both faster and better than expert feedback in the categories of explanation, questions, and specificity. This suggests that LLM feedback can be a high-quality and efficient alternative to expert feedback. However, we postulate that prompt quality is crucial, highlighting the need for prompting guidelines and human expertise.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
The role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · see citing works
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · see citing works
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242 · see citing works
Kluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Fleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354 · see citing works
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087 · see citing works
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079 · see citing works
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004 · see citing works
Artificial intelligence in higher education: the state of the field
Crompton, H., & Burke, D. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00392-8 · Cited by 2,010 · see citing works
Artificial Intelligence: A Modern Approach
Ligęza, A. (1995) · Neurocomputing · doi:10.1016/0925-2312(95)90020-9 · Cited by 1,857 · see citing works
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · see citing works
(2016) · Open Research Online (The Open University) · Cited by 1,410 · see citing works
Kraft, M. A., Blazar, D., & Hogan, D. (2018) · Review of Educational Research · doi:10.3102/0034654318759268 · Cited by 1,202 · see citing works
Feedback Strategies for Interactive Learning Tasks
Narciss, S. (2007) · doi:10.4324/9780203880869.ch11 · Cited by 388 · see citing works
Strijbos, J.-W., Narciss, S., & Dünnebier, K. (2009) · Learning and Instruction · doi:10.1016/j.learninstruc.2009.08.008 · Cited by 369 · see citing works
The CLEAR path: A framework for enhancing information literacy through prompt engineering
Lo, L. S. (2023) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2023.102720 · Cited by 347 · see citing works
A three-dimensional taxonomy of achievement emotions.
Pekrun, R. H., Marsh, H. W., Elliot, A. J., Stockinger, K., Perry, R. P., Vogl, E., Goetz, T., van Tilburg, W. A. P., Lüdtke, O., & Vispoel, W. P. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000448 · Cited by 344 · see citing works
Prompt Engineering for ChatGPT: A Quick Guide to Techniques, Tips, and Best Practices
Ekin, S. (2026) · ASEE Journal of Engineering Technology · doi:10.18260/jet.43.1.5 · Cited by 257 · see citing works
Zhu, M., Liu, O. L., & Lee, H.-S. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103668 · Cited by 229 · see citing works
Learning analytics in European higher education—Trends and barriers
Tsai, Y.-S., Rates, D., Moreno-Marcos, P. M., Muñoz-Merino, P. J., Jivet, I., Scheffel, M., Drachsler, H., Delgado Kloos, C., & Gašević, D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103933 · Cited by 192 · see citing works
Research on peer coaching in preservice teacher education – A review of literature
Lu, H. (2009) · Teaching and Teacher Education · doi:10.1016/j.tate.2009.10.015 · Cited by 174 · see citing works
Learning with ChatGPT 3.5 as a more knowledgeable other: an autoethnographic study
Stojanov, A. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00404-7 · Cited by 169 · see citing works
Weber, K. E., Gold, B., Prilop, C. N., & Kleinknecht, M. (2018) · Teaching and Teacher Education · doi:10.1016/j.tate.2018.08.008 · Cited by 165 · see citing works
Self-citationDemszky, D., Liu, J., Hill, H., Jurafsky, D., & Piech, C. (2023) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737231169270 · Cited by 141 · see citing works
The Impact of Feedback in Higher Education
Pitt, E. (2019) · Springer eBooks · doi:10.1007/978-3-030-25112-3 · Cited by 128 · see citing works
Gielen, M., & De Wever, B. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.06.019 · Cited by 122 · see citing works
Feedback for General Practitioners in Training: Quality, Styles, and Preferences
Prins, F. J., Sluijsmans, D. M. A., & Kirschner, P. A. (2006) · Advances in Health Sciences Education · doi:10.1007/s10459-005-3250-z · Cited by 120 · see citing works
A Survey of Unsupervised Generative Models for Exploratory Data Analysis and Representation Learning
Abukmeil, M., Ferrari, S., Genovese, A., Piuri, V., & Scotti, F. (2021) · ACM Computing Surveys · doi:10.1145/3450963 · Cited by 112 · see citing works
Sailer, M., Bauer, E., Hofmann, R., Kiesewetter, J., Glas, J., Gurevych, I., & Fischer, F. (2023) · Learning and Instruction · doi:10.1016/j.learninstruc.2022.101620 · Cited by 106 · see citing works
Effects of digital video-based feedback environments on pre-service teachers’ feedback competence
Prilop, C. N., Weber, K. E., & Kleinknecht, M. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.08.011 · Cited by 79 · see citing works
Self-citationMachine learning based feedback on textual student answers in large courses
Bernius, J. P., Krusche, S., & Bruegge, B. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100081 · Cited by 75 · see citing works
From plans to actions: A process model for why feedback features influence feedback implementation
Wu, Y., & Schunn, C. D. (2021) · Instructional Science · doi:10.1007/s11251-021-09546-5 · Cited by 56 · see citing works
Zottmann, J., Stegmann, K., Strijbos, J.-W., Vogel, F., Wecker, C., & Fischer, F. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.04.014 · Cited by 47 · see citing works
Umgang mit Feedback im Kontext Schule
Behnke, K. (2015) · Springer eBooks · doi:10.1007/978-3-658-10223-4 · Cited by 18 · see citing works
ChatGPT is a Remarkable Tool -- For ExpertsPreprint
Azaria, A., Azoulay, R., & Reches, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.03102 · Cited by 11 · see citing works
Kollegiales Feedback aus der Perspektive von Lehrpersonen
Funk, C. M. (2016) · Springer eBooks · doi:10.1007/978-3-658-13062-6 · Cited by 5 · see citing works
Wie gut erkennen Lehrkräfte typische Fehler in der Formulierung von Lernzielen?
Wittwer, J., Kratschmayr, L., & Voss, T. (2019) · Unterrichtswissenschaft · doi:10.1007/s42010-019-00056-5 · Cited by 5 · see citing works
Video-, Text- oder Live-Coaching? Konzeption und Erprobung neuer Formate der Praktikumsbegleitung
Weber, K. E., Prilop, C. N., Glimm, K., et al. (2018) · Herausforderung Lehrer*innenbildung - Zeitschrift zur Konzeption, Gestaltung und Diskussion · doi:10.4119/hlz-2384 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 6
- Hammerness, K. M., Darling-Hammond, L., Bransford, J., Berliner, D. C., Cochran-Smith, M., McDonald, M., & Zeichner, K. M. (2005). How teachers learn and develop. In Darling-Hammond, L., Bransford, J., LePage, P., Hammerness, K., and Duffy, H. (Eds.), Preparing Teachers for a Changing World: What teachers should learn and be able to do (pp. 358-389). San Francisco: Jossey-Bass.
- Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81- 112.
- Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., (...), Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education, Learning and Individual Differences, 103, https://doi.org/10.1016/j.lindif.2023.102274 · DOI · Paper page
- Uhde, G. & Thies, B. (Eds). Kompetenzentwicklung im Lehramtsstudium durch professionelles Training (pp.83-108). https://doi.org/10.24355/dbbs.084-201901231126-0 · DOI
- Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital learning environments on the basis of the interactive feedback model. Digital Education Review, 23.
- Prilop, C. N., Weber, K., & Kleinknecht, M. (2019). Entwicklung eines video-und textbasierten Instruments zur Messung kollegialer Feedbackkompetenz von Lehrkräften [Development of a video- and text-based instrument for the assessment of teachers' peer feedback competence]. In T. Ehmke, P. Kuhl, & M. Pietsch (Eds.), Lehrer. Bildung. Gestalten: Beiträge zur empirischen Forschung in der Lehrerbildung (pp. 153-163). Weinheim Basel: Beltz Juventa Verlag.
Cited by
Sorted by citations
Chan, S. T. S., Lo, N. P. K., & Wong, A. M. H. (2024) · Contemporary Educational Technology · doi:10.30935/cedtech/15607 · Cited by 56 · see citing works
Chan, S., Lo, N., & Wong, A. (2024) · rEFLections · doi:10.61508/refl.v31i3.277514 · Cited by 29 · see citing works
Jones, B., Desu, A., & Honig, C. D. F. (2026) · European Journal of Dental Education · doi:10.1111/eje.13135 · Cited by 21 · see citing works
Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S. (2026) · JMIR Medical Education · doi:10.2196/90064 · Cited by 1 · see citing works
Untitled work
Chan, C. K. Y., Lo, N., & Wong, G. (2024)
- Authors
- Jacobsen, L. J., & Weber, K. E.
- Affiliations
- Leuphana University of Lüneburg (DE)
- Universität Hamburg (DE)
- Publisher
- Center for Open Science
- Published
- 2023-09-29
- OpenAlex
- W4387221424
- Cited by
- 57
- References
- 40
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 16
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- large language models, AI-generated feedback, prompt engineering, prompt quality, feedback quality, ChatGPT-4, higher education, automated feedback, student learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
