Evaluating the accuracy and reliability of AI content detectors in academic contexts
Open accessPublished version of a preprint
Abstract
Abstract Generative Artificial Intelligence (GenAI) tools capable of producing human-like text have raised considerable concerns regarding academic integrity. In response, AI content detectors such as Turnitin and Originality are increasingly employed in higher education. However, empirical evidence regarding their accuracy, reliability, and fairness, particularly in the context of English as a Foreign Language (EFL) writing remains limited. This study evaluates the performance of both detectors across variations in text length, genre, and authorship type. A balanced dataset of 192 texts was constructed, comprising authentic EFL student writing, professionally authored human texts, AI-generated outputs, and hybrid compositions. Based on the percentage of AI content identified by each detector, texts were categorized as Human, Hybrid, or AI. Detector performance was assessed against ground truth labels using precision, recall, specificity, F1 score, and accuracy. Statistical significance was tested using Pearson’s chi-square and Fisher’s Exact Test.Originality outperformed Turnitin in overall accuracy (0.69 vs. 0.61) and macro-average recall (0.60 vs. 0.51). However, both detectors performed poorly on Hybrid texts, with recall scores of 0.31 for Turnitin and 0.02 for Originality. Performance declined significantly with longer texts (p < 0.015 for Turnitin; p < 0.002 for Originality) and varied across genres, with higher accuracy observed in humanities than in science (p < 0.0001 for both detectors). Originality also exhibited a borderline statistically significant bias favoring professionally authored texts over EFL texts (p = 0.058). These findings suggest that neither detector is sufficiently reliable to serve as the sole basis for academic misconduct decisions. Institutions are advised to supplement AI detection tools with human judgment, incorporate AI literacy into academic curricula, and encourage detector developers to pursue further research into bias mitigation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The disparity in false positive rates between non-native and native English writers has been replicated across different detection platforms, linguistic populations, and testing conditions.
“This disparity has been replicated across different detection platforms, linguistic populations, and testing conditions”
From ARE AI DETECTORS FAIR TO EFL LEARNERS? A LIBRARY RESEARCH ON DETECTION BIAS AND ACADEMIC INTEGRITY (2026) · cites this paper as Hadra et al., 2026
The ‘Effect of authorship type’ section reports more misclassifications of EFL student writing than professional writing by Originality, but calls the difference a borderline, non-significant trend. It reports no measurable EFL bias for Turnitin. The paper therefore does not establish that a false-positive disparity was replicated across detection platforms, linguistic populations, and testing conditions. (LLM, conf=0.89)
Experimental — accuracy not yet measuredAI content detectors produce inconsistent results and are readily circumvented by minor edits to AI-generated text, limiting the weight detection-based enforcement alone can carry.
“AI content detectors have been shown to produce inconsistent results and are readily circumvented by minor edits to AI-generated text, which limits how much weight detection-based enforcement alone can carry (Hadra et al., 2026;Van Vlasselaer et al., 2026).”
From Convenience Amid Crisis: Explaining Generative AI Misuse and Compliance Intention Among Myanmar University Students (2026) · cites this paper as Hadra et al., 2026
The ‘Reconsidering the role of detection in academic integrity’ section reports inconsistent performance and says detectors should not be primary or determinative evidence in misconduct investigations. The Introduction reports that paraphrasing sharply reduced GPT-Zero’s accuracy, but the paper does not establish that AI detectors generally are readily circumvented by minor edits. Its ‘Limitations and future research’ section says the effects of editing AI-generated text to varying degrees were beyond this study’s scope. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCurrent AI detection tools are unsuitable as authoritative indicators of authorship in EFL assessment contexts.
“one recent evaluation concluded that current tools are unsuitable as authoritative indicators of authorship”
From ARE AI DETECTORS FAIR TO EFL LEARNERS? A LIBRARY RESEARCH ON DETECTION BIAS AND ACADEMIC INTEGRITY (2026) · cites this paper as Hadra et al., 2026
Hadra et al. evaluated Turnitin and Originality across 192 texts and found overall accuracy of 0.61 and 0.69, respectively.
“Hadra et al. [34] evaluated Turnitin and Originality across 192 texts - including authentic writing by English as a Foreign Language (EFL) students produced before the generative AI storm - and found overall accuracy of just 0.61 and 0.69, respectively”
From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [34]
Hadra et al. emphasize that for EFL-focused institutions, overreliance on machine-based identification risks misclassifying legitimate student work.
“emphasizing that 'for [EFL]-focused institutions in particular, overreliance on machine-based identification risks misclassifying legitimate student work'”
From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [34]
Hadra et al. conclude that AI detection tools' limitations make them unsuitable as the sole basis for decisions regarding academic misconduct.
“Hadra et al. [34] conclude that 'their limitations make them unsuitable as the sole basis for decisions regarding academic misconduct,'”
From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [34]
Both Turnitin and Originality performed especially poorly on hybrid human-AI texts.
“with both tools performing especially poorly on hybrid human-AI texts”
From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [34]
Prior work has documented the unreliability of AI detection tools and called for caution in their deployment in educational settings.
“Prior work has documented the unreliability of AI detection tools and called for caution in their deployment in educational settings [34, 43, 77].”
From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 33 · Cited by 27
Research works printed but not linked: 17
- Akram A (2023) An empirical study of AI-generated text detection tools. Adv Mach Learn Artif Intell 4(2). h t t p s : / / d o i . o r / 1 0 . 3 3 1 4 0 / a m l a i . 0 4 . 0 2 . 0 · DOI
- Alnsour MM, Qouzah L, Aljamani S, Alamoush RA, AL-Omiri MK (2025) AI in education: enhancing learning potential and addressing ethical considerations among academic staff-a cross-sectional study at the university of Jordan. Int J Educa- tional Integr 21(1):16-. https://doi.org/10.1007/s40979-025-00189-4 · DOI
- Andersen N, Mang J, Goldhammer F, Zehner F (2025) Algorithmic fairness in automatic short answer scoring. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-025-00495-5 · DOI
- Avello D, Aranguren Zurita S (2025) Exploring the nexus of academic integrity and artificial intelligence in higher education: a bibliometric analysis. In International journal for educational integrity. BioMed Cent 21(1):24. h t t p s : / / d o i . o r g / 1 0 . 1 0 0 7 / s 4 0 9 7 9 -0 2 5 -0 0 1 9 9 -2 · DOI
- Bewick V, Cheek L, Ball J (2003) Statistics review 8: qualitative data -tests of association. Crit Care 8(1):46. h t t p s : / / d o i . o r g / 1 0 . 1 1 8 6 / c c 2 4 · DOI
- Buchert J (2024) How do AI detectors work? Do they work? Intellectualead. h / / , Bedi A, Zhu S, An B, Manocha D, Huang F (2024) Position: On the possibilities of AI-generated text detection. In: Salakhutdinov R, Kolter Z, Heller K, Weller A, Oliver N, Scarlett J, Berkenkamp F, editors. Proceedings of the 41st Interna- tional Conference on Machine Learning. Vol. 235. Proceedings of Machine Learning Research. p. 6093-6115. h t t p s : / / p r Chan CKY 2025) Students' perceptions of 'AI-giarism': investigating changes in Understandings of academic misconduct. Educ Inform Technol 30(6):8087-8108. https://doi.org/10.1007/s10639-024-13151-7 · DOI · Paper page
- De Angeli K, Gao S, Danciu I, Durbin EB, Wu X-C, Stroup A, Doherty J, Schwartz S, Wiggins C, Damesyn M, Coyle L, Penberthy L, Tourassi GD, Yoon H-J (2022) Class imbalance in out-of-distribution datasets: improving the robustness of the TextCNN for the classification of rare cancer types. J Biomed Inform 125:103957. https://doi.org/10.1016/j.jbi.2021.103957 · DOI
- Doru B, Maier C, S Busse J, Lücke T, Schönhoff J, Enax-Krumova E, Hessler S, Berger M, Tokic M (2025) Detecting artificial intel- ligence-generated versus human-written medical student essays: semirandomized controlled study. JMIR Med Educ 11. https://doi.org/10.2196/62779 · DOI
- Fariello S, Fenza G, Forte F, Gallo M, Marotta M (2025) Distinguishing human from machine: A review of advances and chal- lenges in AI-Generated text detection. Int J Interact Multimedia Artif Intell 9(3):6-18. h t t p s : / / d o i . o r g / 1 0 . 9 7 8 1 / i j i m a i . 2 0 2 4 . 1 2 . 0 0 · DOI
- Fisher MJ, Marshall AP, Mitchell M (2011) Testing differences in proportions. Australian Crit Care 24(2):133-138. h t t p s : / / d o i . o r / 1 0 . 1 0 1 6 / j . a u c c . 2 0 1 1 . 0 1 . 0 0 · DOI
- Helm P, Bella G, Koch G, Giunchiglia F (2024) Diversity and Language technology: how Language modeling bias causes epis- temic injustice. Ethics Inf Technol 26(1):1-15. https://doi.org/10.1007/s10676-023-09742-6 · DOI · Paper page
- 7 · DOI · Paper page
- Kumar N, Singh AK (2025) Artificial intelligence content detection techniques using watermarking: a survey. Image Vis Comput. 163:105728. https://doi.org/10.1016/j.imavis.2025.105728 · DOI
- Saheb T, Sidaoui M, Schmarzo B (2024) Convergence of artificial intelligence with social media: A bibliometric & qualitative analysis. Telematics Inf Rep 14:100146. https://doi.org/10.1016/j.teler.2024.100146 · DOI
- Skaik Y (2008) Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol 56(4):341. h t t p s : / / d o i . o r g / 1 0 . 4 1 0 3 / 0 3 0 1 -4 7 3 8 . 4 1 4 · DOI
- Waltzer T, Pilegard C, Heyman GD (2024) Can you spot the bot? Identifying AI-generated writing in college essays. Int J Educa- tional Integr 20(1):11. https://doi.org/10.1007/s40979-024-00158-3 · DOI · Paper page
- Zhao Y, Zou C, Sridhar R, Cui C, Starner T (2024) Leveraging past assignments to determine if students are using ChatGPT for their essays. In: Proceedings of the 11th ACM Conference on Learning @ Scale (L@S 2024). p. 320-324. h t t p s : / / d o i . o r g / 1 0 . 1 1 4 5 / 3 6 5 7 6 0 4 . 3 6 6 4 7 · DOI
- Authors
- Hadra, M., Cambridge, K., & Mesbah, M.
- Affiliations
- Sultan Qaboos University (OM)
- Corresponding author
- Hadra, Mohammad
- Venue
- International Journal for Educational Integrity
- Published
- 2026-02-02
- Volume
- 22
- Issue
- 1
- ISSN
- 1833-2595 (ISSN-L)
- OpenAlex
- W7127044430
- Cited by
- 27
- References
- 33
- Citations by year
- 2026: 27
- FWCI
- 153.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- International Journal for Educational Integrity (publishedVersion, cc-by-nc-nd)
- Research Square (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Keywords
- generative AI, academic integrity, Turnitin, originality, AI-generated text detection, hybrid authorship, detection accuracy, text length effects, genre variation, scientific writing, humanities writing, AI use policies, higher education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
