Evaluating AI and human authorship quality in academic writing through physics essays
Open accessAbstract
This study aims to compare the academic writing quality and detectability of authorship between human and AI-generated texts by evaluating n = 300 short-form physics essay submissions, equally divided between student work submitted before the introduction of ChatGPT and those generated by OpenAI’s GPT-4. In blinded evaluations conducted by five independent markers who were unaware of the origin of the essays, we observed no statistically significant differences in scores between essays authored by humans and those produced by AI ( p -value = 0.107, α = 0.05). Additionally, when the markers subsequently attempted to identify the authorship of the essays on a 4-point Likert scale—from ‘Definitely AI’ to ‘Definitely Human’—their performance was only marginally better than random chance. This outcome not only underscores the convergence of AI and human authorship quality but also highlights the difficulty of discerning AI-generated content solely through human judgment. Furthermore, the effectiveness of five commercially available software tools for identifying essay authorship was evaluated. Among these, ZeroGPT was the most accurate, achieving a 98% accuracy rate and a precision score of 1.0 when its classifications were reduced to binary outcomes. This result is a source of potential optimism for maintaining assessment integrity. Finally, we propose that texts with ≤50% AI-generated content should be considered the upper limit for classification as human-authored, a boundary inclusive of a future with ubiquitous AI assistance whilst also respecting human-authorship.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lexical density is an indicator of information richness and compactness.
“Lexical density is an indicator of information richness and compactness. 57,58,59”
From Style, sentiment, and quality of undergraduate writing in the AI era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports (2025) · cites this paper as [57]
Section 3.2 (Lexical Analysis; page unavailable) discusses unique-word counts, word length, and lexical richness, but does not define or assess lexical density. Searches for lexical density and information conveyed per word found no passage describing it as an indicator of information richness or compactness. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794 · cited 1× · see citing works
“These results echo those of Ghassemi et al.[11], who found that below-average performers gained more from using AI than above-average performers.”
ResultGPT detectors are biased against non-native English writers
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023) · Patterns · doi:10.1016/j.patter.2023.100779 · Cited by 673 · cited 1× · see citing works
“Some initial detection tools were criticized for their bias against non-native English speakers [17].”
BackgroundHow Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507 · see citing works
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151 · see citing works
ChatGPT and the frustrated Socrates
Gregorcic, B., & Pendrill, A.-M. (2023) · Physics Education · doi:10.1088/1361-6552/acc299 · Cited by 125 · see citing works
RADAR: Robust AI-Text Detection via Adversarial Learning
Hu, X., Chen, P.-Y., & Ho, T.-Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0662 · Cited by 62 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- Gianluca Grimaldi and Bruno Ehrler. "AI et al.: machines are about to change scientific publishing forever". In: ACS Energy Letters 8.1 (2023), pp. 878-880.
- Michael Dowling and Brian Lucey. "ChatGPT for (finance) research: The Bananarama conjecture". In: Finance Research Letters 53 (2023), p. 103662.
- Tom Revell, Will Yeadon, Glenn Cahilly-Bretzin, et al. ChatGPT versus Human Essayists: An Exploration of the Impact of Artificial Intelligence for Authorship and Academic Integrity in the Humanities. PREPRINT (Version 1) available at Research Square. Nov. 2023. url: https://doi.org/10.21203/rs.3.rs-3483059/v1. · DOI
- Md Tahmid Rahman Laskar et al. "A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets". In: arXiv preprint arXiv:2305.18486 (2023). · arXiv
- Josh Achiam et al. "Gpt-4 technical report". In: arXiv preprint arXiv:2303.08774 (2023). · arXiv
- Alex Wang et al. "Superglue: A stickier benchmark for general-purpose language understanding systems". In: Advances in neural information processing systems 32 (2019).
- Manmeet Singh, Vaisakh SB, Neetiraj Malviya, et al. "Mind meets machine: Unravelling GPT-4's cognitive psychology". In: arXiv preprint arXiv:2303.11436 (2023). · arXiv
- Will Yeadon and Douglas P Halliday. "Exploring durham university physics exams with large language models". In: arXiv preprint arXiv:2306.15609 (2023). · arXiv
- Jiameng Pu et al. "Deepfake text detection: Limitations and opportunities". In: 2023 IEEE Symposium on Security and Privacy (SP). IEEE. 2023, pp. 1613-1630.
- Vinu Sankar Sadasivan et al. "Can ai-generated text be reliably detected?" In: arXiv preprint arXiv:2303.11156 (2023). · arXiv
- Giulia Polverini and Bor Gregorcic. "Performance of ChatGPT on the test of understanding graphs in kinematics". In: Physical Review Physics Education Research 20.1 (2024), p. 010109.
- Bor Gregorcic and Giulia Polverini. "ChatGPT-4 and the satisfied Socrates". In: arXiv preprint arXiv:2401.11987 (2024). · arXiv
- Giulia Polverini and Bor Gregorcic. "How understanding large language models can inform the use of ChatGPT in physics education". In: European Journal of Physics 45.2 (2024), p. 025701.
- Ozgur Dedehayir and Martin Steinert. "The hype cycle model: A review and future directions". In: Technological Forecasting and Social Change 108 (2016), pp. 28-41.
- Jackson P Davis and Watt A Price. "Deep learning for teaching university physics to computers". In: American Journal of Physics 85.4 (2017), pp. 311-312.
Cited by
Sorted by citations
Revell, T., Yeadon, W., Cahilly-Bretzin, G., Clarke, I., Manning, G., Jones, J., Mulley, C., Pascual, R. J., Bradley, N., Thomas, D., et al. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00161-8 · Cited by 51 · see citing works
Self-citationMak, M. H. C., & Walasek, L. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100507 · Cited by 1 · see citing works
- Authors
- Yeadon, W., Agra, E., Inyang, O.-O., Mackay, P., & Mizouri, A.
- Affiliations
- Durham University (GB)
- Corresponding author
- Yeadon, Will
- Venue
- European Journal of Physics
- Publisher
- IOP Publishing
- Published
- 2024-09-01
- Volume
- 45
- Issue
- 5
- Pages
- 055703
- ISSN
- 0143-0807 (ISSN-L)
- 1361-6404
- OpenAlex
- W4400915448
- Cited by
- 26
- References
- 6
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 6
- FWCI
- 10.78
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- European Journal of Physics (publishedVersion, cc-by)
- Durham Research Online (Durham University) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- AI-generated text, authorship detection, academic writing quality, GPT-4, ChatGPT, AI detection software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
