The Death of the Short-Form Physics Essay in the Coming AI Revolution
Open accessAbstract
The latest AI language modules can produce original, high quality full short-form ($300$-word) Physics essays within seconds. These technologies such as ChatGPT and davinci-003 are freely available to anyone with an internet connection. In this work, we present evidence of AI generated short-form essays achieving first-class grades on an essay writing assessment from an accredited, current university Physics module. The assessment requires students answer five open-ended questions with a short, $300$-word essay each. Fifty AI answers were generated to create ten submissions that were independently marked by five separate markers. The AI generated submissions achieved an average mark of $71 \pm 2 \%$, in strong agreement with the current module average of $71 \pm 5 %$. A typical AI submission would therefore most-likely be awarded a First Class, the highest classification available at UK universities. Plagiarism detection software returned a plagiarism score between $2 \pm 1$% (Grammarly) and $7 \pm 2$% (TurnitIn). We argue that these results indicate that current AI MLPs represent a significant threat to the fidelity of short-form essays as an assessment method in Physics courses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many are predicting the end of essays as a form of assessment (e.g. Stokel-Walker, 2022; Yeadon et al., 2022).
“Looking at the very swift increase in use of the ChatGPT site, it's not hard to see why many are predicting the end of essays as a form of assessment (e.g.Stokel-Walker, 2022;Yeadon et al., 2022).”
From Chatting and cheating: Ensuring academic integrity in the era of ChatGPT (2024) · cites this paper as Yeadon et al., 2022
The Background (section 1.1) argues that AI effectively renders the short-form essay obsolete as an assessment tool. The Concluding thoughts (section 4.5) narrow this to non-invigilated short-form essay questions, while section 4.1 suggests invigilation as a possible solution. The source supports a prediction about certain essay assessments, not the end of essays as an assessment form generally. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 13
- Clark E, August T, Serrano S, Haduong N, Gururangan S and Smith N A 2021 All that's 'human' is not gold: evaluating human evaluation of generated text Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int. Joint Conf. on Natural Language Processing (Long Papers) vol 1 (Association for Computational Linguistics) pp 7282-96
- Daniel J E, Brink W, Eloff R and Copley C 2019 Towards automating healthcare question answering in a noisy multilingual low-resource setting Proc. 57th Annual Meeting of the Association for Computational Linguistics (Florence: Association for Computational Linguistics) pp 948-53
- Queudot M, Charton É and Meurs M-J 2020 Improving access to justice with legal chatbots Stats 3 356-75 · DOI
- Malik A, Wu M, Vasavada V, Song J, Coots M, Mitchell J, Goodman N and Piech C 2019 Generative grading: near human-level accuracy for automated feedback on richly structured problems (arXiv:1905.09916) · arXiv
- Pillutla K, Swayamdipta S, Zellers R, Thickstun J, Welleck S, Choi Y and Harchaoui Z 2021 MAUVE: measuring the gap between neural text and human text using divergence frontiers Advances in Neural Information Processing Systems vol 34, ed M Ranzato, A Beygelzimer, Y Dauphin, P Liang and J W Vaughan (Curran Associates, Inc.) pp 4816-28
- Blin F and Munro M 2008 Why hasn't technology disrupted academics' teaching practices? understanding resistance to change through the lens of activity theory Comput. Educ. 50 475-90 · DOI
- Fyfe P 2022 How to cheat on your final paper: assigning AI for student writing AI Soc. 1-11 · DOI · Paper page
- Sharples M 2022 Automated essay writing: an AIED opinion Int. J. Artif. Intell. Educ. 32 1119-26 · DOI
- Brown T 2020 Language models are few-shot learners Advances in Neural Information Processing Systems vol 33, ed H Larochelle, M Ranzato, R Hadsell, M Balcan and H Lin (Curran Associates, Inc.) pp 1877-901
- Köbis N and Mossink L D 2021 Artificial intelligence versus maya angelou: experimental evidence that people cannot differentiate ai-generated from human-written poetry Comput. Hum. Behav. 114 106553 · DOI · Paper page
- Jordan K 2015 Massive open online course completion rates revisited: assessment, length and attrition Int. Rev. Res. Open Distrib. Learn. 16 341-58 · DOI
- Shermis M D 2014 State-of-the-art automated essay scoring: competition, results and future directions from a United States demonstration Assess. Writ. 20 53-76 · DOI
- Ramesh D and Sanampudi S K 2021 An automated essay scoring systems: a systematic literature review Artif. Intell. Rev. 55 2495-527 · DOI
Cited by
Sorted by citations
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087 · see citing works
A large-scale comparison of human-written versus ChatGPT-generated essays
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-45644-9 · Cited by 338 · see citing works
Tossell, C. C., Tenhundfeld, N. L., Momen, A., Cooley, K., & de Visser, E. J. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3355015 · Cited by 218 · see citing works
From Tool to Method: Developing AI Literacy in an Upper-Level History Course
Dumančić, M., & Stickle, T. (2026) · Journal of Teaching and Learning with Technology · doi:10.14434/jotlt.v14i1.41669
- Authors
- Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A., & Testrow, C.
- Affiliations
- Durham University (GB)
- Corresponding author
- Yeadon, Will
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Volume
- 58
- Issue
- 3
- Pages
- 035027-035027
- ISSN
- 0031-9120 (ISSN-L)
- arXiv
- 2212.11661
- OpenAlex
- W4362551436
- Cited by
- 199
- References
- 0
- Citations by year
- 2026: 23 · 2025: 54 · 2024: 68 · 2023: 54
- FWCI
- 46.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Physics Education (publishedVersion, cc-by)
- Durham Research Online (Durham University) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- Durham Research Online (Durham University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- AI-generated text, ChatGPT, physics assessment, plagiarism detection, academic integrity, natural language processing, university physics education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
