ChatGPT versus human essayists: an exploration of the impact of artificial intelligence for authorship and academic integrity in the humanities
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Generative AI has prompted educators to reevaluate traditional teaching and assessment methods. This study examines AI’s ability to write essays analysing Old English poetry; human markers assessed and attempted to distinguish them from authentic analyses of poetry by first-year undergraduate students in English at the University of Oxford. Using the standard UK University grading system, AI-written essays averaged a score of 60.46, whilst human essays achieved 63.57, a margin of difference not statistically significant ( p = 0.10). Notably, student submissions applied a nuanced understanding of cultural context and secondary criticism to their close reading, while AI essays often described rather than analysed, lacking depth in the evaluation of poetic features, and sometimes failing to properly recognise key aspects of passages. Distinguishing features of human essays included detailed and sustained analysis of poetic style, as well as spelling errors and lack of structural cohesion. AI essays, on the other hand, exhibited a more formal structure and tone but sometimes fell short in incisive critique of poetic form and effect. Human markers correctly identified the origin of essays 79.41% of the time. Additionally, we compare three purported AI detectors, finding that the best, ‘Quillbot’, correctly identified the origin of essays 95.59% of the time. However, given the high threshold for academic misconduct, conclusively determining origin remains challenging. The research also highlights the potential benefits of generative AI’s ability to advise on structuring essays and suggesting avenues for research. We advocate for transparency regarding AI’s capabilities and limitations, and this study underscores the importance of human critical engagement in teaching and learning in Higher Education. As AI’s proficiency grows, educators must reevaluate what authentic assessment is, and consider implementing dynamic, holistic methods to ensure academic integrity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The findings revealed that AI essays achieved an average score of 60.46, closely trailing human submissions, which averaged 63.57 a difference that is not statistically significant (p = 0.10) [6].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many institutions are moving toward more authentic and formative assessment practices.These include in-class written tasks, oral presentations, staged writing processes with feedback loops, and portfolio-based evaluations [6].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI-generated essays in humanities disciplines often meet acceptable grading criteria, mimicking the surface-level features of human writing [6].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While lacking critical analysis, these essays are well-organized, grammatically polished, and difficult to differentiate from human-written content without detailed forensic examination.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, the study raised concerns about the fairness of detection algorithms, highlighting the risk that non-native English speakers could be disproportionately penalized due to linguistic patterns that diverge from native norms [6].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [6]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1×
“There is promising potential in using ChatGPT as a research and learning assistant (Kas-neci et al. 2023).”
BackgroundZawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242 · cited 3×
“The urgency for comprehensive insights into AI’s ramifications for both teaching and assessment has only intensified in an era where powerful generative AI tools are publicly-accessible (Zawacki-Richter et al. 2019).”
BackgroundArtificial Intelligence in Education: A Review
Chen, L., Chen, P., & Lin, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2988510 · Cited by 4,133 · cited 1×
“Numerous review articles portrayed artificial intelligence as a significant asset to academic administration (Chen et al. 2020) and a pathway to more dynamic learning experiences.”
BackgroundArtificial intelligence in education : challenges and opportunities for sustainable development
Pedró, F., Subosa, M., Rivas, A., & Valverde, P. (2019) · MINISTERIO DE EDUCACIÓN · Cited by 2,426
Artificial intelligence in higher education: the state of the field
Crompton, H., & Burke, D. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00392-8 · Cited by 2,010 · cited 2×
“Like many STEM subjects, language acquisition can be assessed more objectively, and a relatively linear progression of skills and complexity can be arranged.”
BackgroundThe promises and challenges of artificial intelligence for teachers: A systematic review of research
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022) · TechTrends · doi:10.1007/s11528-022-00715-y · Cited by 984 · cited 1×
“Intriguingly, challenges highlighted in AI-focused literature as recent as 2022 included the ‘[l]imited technical capacity of AI’ and the ‘[i]napplicability of the AI system to multiple settings’ (Celik et al. 2022), rather than the evident risks to assessment authenticity observed today.”
BackgroundRecipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784
Recipes for building an open-domain chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Shuster, K., Smith, E. M., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.13637 · Cited by 784 · cited 1×
“While recommending relevant journals, an essay by J.R.R. Tolkien, and Seamus Heaney’s adaptation of “Beowulf”, the other references were either mis-attributed to authors who have produced similar works, or entirely “hallucinated” (Roller et al. 2021), in each case with an inaccurate summary.”
MethodGPT detectors are biased against non-native English writers
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023) · Patterns · doi:10.1016/j.patter.2023.100779 · Cited by 673 · cited 2×
“Beyond these challenges, other studies have highlighted textual nuances, such as the presence or absence of specific phrases and symbols, as potential markers for distinguishing between human and AI-generated content (Desaire et al. 2023).”
BackgroundArtificial Intelligence for Student Assessment: A Systematic Review
González-Calatayud, V., Prendes-Espinosa, P., & Roig-Vila, R. (2021) · Applied Sciences · doi:10.3390/app11125467 · Cited by 503 · cited 1×
“In contrast, the humanities, a field rich in nuance and interpretation, remains underrepresented in AI assessment explorations (González-Calatayud et al. 2021).”
BackgroundKöbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406 · cited 1×
“Moreover, in a comprehensive study with a large sample size ( N = 830 ), researchers found that outputs from models like GPT-2, particularly in specialised domains like poetry, were hard to distinguish from human-created content (Köbis and Mossink 2021).”
BackgroundHuman heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342 · cited 1×
“Compounding the issue, some AI systems are being fine-tuned to produce content intentionally designed to evade human detection, making the distinction even more challenging (Jakesch et al. 2023).”
BackgroundTowards artificial intelligence-based assessment systems
Luckin, R. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0028 · Cited by 303 · cited 1×
“These advantages were often designed to reduce educator workload (and associated stress), ranging from smart classrooms (Kim et al. 2018) to tailored assessment technologies (Luckin 2017).”
BackgroundUnpacking teachers’ intentions to integrate technology: A meta-analysis
Scherer, R., & Teo, T. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.03.001 · Cited by 302 · cited 2×
“While past technological innovations were adopted gradually within education (Scherer and Teo 2019), the emergence of ChatGPT suggests a significant shift in the realm of essay writing as a mode of assessment (Yeadon et al. 2024).”
BackgroundEducator and Student Perspectives on the Impact of Generative AI on Assessments in Higher Education
Smolansky, A., Cram, A., Raduescu, C., Zeivots, S., Huber, E., & Kizilcec, R. F. (2023) · Proceedings of the Tenth ACM Conference on Learning @ Scale · doi:10.1145/3573051.3596191 · Cited by 275 · cited 2×
“The urgency for comprehensive insights into AI’s ramifications for both teaching and assessment has only intensified in an era where powerful generative AI tools are publicly-accessible (Zawacki-Richter et al. 2019).”
BackgroundShermis, M. D. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.04.001 · Cited by 224 · cited 1×
“Over the past two decades, technology-driven assessments such as automated grading and essay scoring have sought to augment traditional teaching and learning practices (Shermis 2014; Vajjala 2018).”
BackgroundKim, Y., Soyata, T., & Behnagh, R. F. (2018) · IEEE Access · doi:10.1109/access.2018.2791861 · Cited by 213 · cited 1×
“These advantages were often designed to reduce educator workload (and associated stress), ranging from smart classrooms (Kim et al. 2018) to tailored assessment technologies (Luckin 2017).”
BackgroundRethinking authentic assessment: work, well-being, and society
McArthur, J. (2023) · Higher Education · doi:10.1007/s10734-022-00822-y · Cited by 196 · cited 1×
“Authentic assessment also plays a pivotal role in nurturing student self-esteem and motivation (McArthur 2023).”
BackgroundChatGPT: The End of Online Exam Integrity?
Susnjak, T., & McIntosh, T. (2024) · Education Sciences · doi:10.3390/educsci14060656 · Cited by 189 · cited 1×
“In the short term, alterations to assessments could prevent the exploitation of some of the biggest vulnerabilities (Susnjak and McIntosh 2024).”
Automated Essay Writing: An AIED Opinion
Sharples, M. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00300-7 · Cited by 147 · cited 2×
“In a small study comparing US College student essays to GPT-3 outputs, AI responses achieved a similar grade to human essays (passing, but with lower variance) in Law, Research Methods, and US History, but failed Creative Writing (Sharples 2022).”
BackgroundGunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11644 · Cited by 138
Automated Assessment of Non-Native Learner Essays: Investigating the Role of Linguistic Features
Vajjala, S. (2018) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0142-3 · Cited by 121 · cited 1×
“Over the past two decades, technology-driven assessments such as automated grading and essay scoring have sought to augment traditional teaching and learning practices (Shermis 2014; Vajjala 2018).”
BackgroundDesaire, H., Chua, A. E., Isom, M., Jarosova, R., & Hua, D. (2023) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2023.101426 · Cited by 110 · cited 3×
“This aligns with the findings of Desaire et al. (2023), who noted discrepancies in the use of punctuation and the use of ambiguous, vague, or generalising language between human-and AI-writing.”
BackgroundMethodResultGenerative AI detection in higher education assessments
Ardito, C. G. (2025) · New Directions for Teaching and Learning · doi:10.1002/tl.20624 · Cited by 86
The impact of AI in physics education: a comprehensive review from GCSE to university levels
Yeadon, W., & Hardy, T. (2024) · Physics Education · doi:10.1088/1361-6552/ad1fa2 · Cited by 78 · cited 3×
“While past technological innovations were adopted gradually within education (Scherer and Teo 2019), the emergence of ChatGPT suggests a significant shift in the realm of essay writing as a mode of assessment (Yeadon et al. 2024).”
BackgroundResultSelf-citationAssisting academics to identify computer generated writing
Abd-Elaal, E.-S., Gamage, S. H. P. W., & Mills, J. E. (2022) · European Journal of Engineering Education · doi:10.1080/03043797.2022.2046709 · Cited by 62 · cited 1×
“The lack of consistency between markers might be a result of unequal levels of experience with generative AI: other studies have shown that markers with more awareness perform better at detection of authentic writing (Abd-Elaal et al. 2022).”
Lavidas, K., Achriani, A., Athanassopoulos, S., Messinis, I., & Kotsiantis, S. B. (2019) · Education and Information Technologies · doi:10.1007/s10639-019-10071-9 · Cited by 32 · cited 11×
“However, most studies of the impact of AI in Higher Education are in STEM (Crompton and Burke 2023), while many of those in the humanities are concerned with language acquisition (Zawacki-Richter et al. 2019).”
BackgroundInfluentialEvaluating AI and human authorship quality in academic writing through physics essays
Yeadon, W., Agra, E., Inyang, O.-O., Mackay, P., & Mizouri, A. (2024) · European Journal of Physics · doi:10.1088/1361-6404/ad669d · Cited by 26 · cited 2×
“While past technological innovations were adopted gradually within education (Scherer and Teo 2019), the emergence of ChatGPT suggests a significant shift in the realm of essay writing as a mode of assessment (Yeadon et al. 2024).”
BackgroundResultSelf-citationWhat's in a Name? The Past and Present Racism in 'Anglo-Saxon' Studies
Rambaran-Olm, M., & Wade, E. (2022) · The Yearbook of English Studies · doi:10.1353/yes.2022.0010 · Cited by 23 · cited 1×
“…an awareness of relevant, evolving issues (dependent on the cut-off date of its training data); when asked about the use of the term ‘Anglo-Saxon’, GPT-4 provides information on both sides of the current scholarly debate 3 surrounding this now-contentious terminology (Rambaran-Olm and Wade 2022).”
BackgroundWashington and Lee Law Review OnlinePreprint
Frazier, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5148069 · Cited by 9
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 6
- Baird JA, Greatorex J, Bell JF (2004) What makes marking reliable? experiments with uk examinations. Assess Educ Princ Policy Pract 11(3):331–348
- Gupta S, Chen Y (2022) Supporting inclusive learning using chatbots? a chatbot-led interview study. J Inf Syst Educ 33(1):98–108
- Hu X, Chen PY, Ho TY (2023) Radar: Robust ai-text detection via adversarial learning. Adv Neural Inf Process Syst 36:15077–15095
- Jannai D, Meron A, Lenz B, Levine Y, Shoham Y (2023) Human or not? a gamified approach to the turing test. arXiv preprint arXiv:2305.20010 · arXiv
- Liu Z, Yao Z, Li F, Luo B (2023) Check me if you can: Detecting chatgpt-generated academic writing using checkgpt. arXiv preprint arXiv:2306.05524 · arXiv
- Sadasivan VS, Kumar A, Balasubramanian S, Wang W, Feizi S (2023) Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156 · arXiv
Cited by
Tejeda, A. S., Oller, J. C. D., Escandell, J. M. B., Gómez-Díaz, R., & Rodríguez, A. G. (2026) · Frontiers in Education · doi:10.3389/feduc.2025.1711718 · Cited by 131
“Some of these tools have been criticized for “hallucinating” content or providing incorrect information, a phenomenon that refers to the generation of inaccurate responses that nevertheless appear realistic (Kim et al., 2025; Revell et al., 2024).”
InfluentialKrupp, L., Steinert, S., Kiefer-Emmanouilidis, M., Avila, K. E., Lukowicz, P., Kuhn, J., Küchemann, S., & Karolus, J. (2024) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia240195 · Cited by 82
Does ChatGPT Write Like a Student? Engagement Markers in Argumentative Essays
Jiang, F. (., & Hyland, K. (2025) · Written Communication · doi:10.1177/07410883251328311 · Cited by 52
“Others express concern that “the eerily humanlike chatbot” (Satariano & Kang, 2023) might make it difficult to distinguish GPT-generated and human-authored texts in assessing students’ writing (Revell et al., 2024).”
Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education
Deep, P. D., Edgington, W. D., Ghosh, N., & Rahaman, M. S. (2025) · Information · doi:10.3390/info16100905 · Cited by 26
Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education
Deep, P. D., Edgington, W. D., Ghosh, N., & Rahaman, M. S. (2025) · doi:10.20944/preprints202507.2233.v1 · Cited by 26
Metadata
- Authors
- Revell, T., Yeadon, W., Cahilly-Bretzin, G., Clarke, I., Manning, G., Jones, J., Mulley, C., Pascual, R. J., Bradley, N., Thomas, D., & Leneghan, F.
- Affiliations
- University of Oxford (GB)
- Durham University (GB)
- University of Liverpool (GB)
- Corresponding author
- Revell, T.
- Venue
- International Journal for Educational Integrity
- Published
- 2024-10-21
- Volume
- 20
- Issue
- 1
- ISSN
- 1833-2595 (ISSN-L)
- OpenAlex
- W4403579585
- Cited by
- 51
- References
- 29
- Citations by year
- 2026: 33 · 2025: 17 · 2024: 1
- FWCI
- 21.14
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- International Journal for Educational Integrity (publishedVersion, cc-by)
- Durham Research Online (Durham University) (publishedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- UK Research and Innovation
- Arts and Humanities Research Council
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Keywords
- generative AI, ChatGPT, academic integrity, essay scoring, AI detection, Old English poetry, close reading, essay marking, QuillBot, artificial intelligence in higher education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.