CoChat
Sign Up
Cited by 5Academic integrity and plagiarismArtificial Intelligence in EducationAuthorship Attribution and Profiling

The Robustness of AI-Classifiers in the Face of AI-Assisted Plagiarism: The Case of Turnitin AI Content Detector

Open access
Ibrahim, Karim Hesham Shaker, Al Otaibi, Dhari, Sibai, Fadi N.·2025·International Journal of Computer-Assisted Language Learning and Teaching
DOI 10.4018/ijcallt.375428
Open this paper in CoChat — free accountOpen PDF

Abstract

The release of ChatGPT marked the beginning of a new era of artificial intelligence (AI)-assisted plagiarism that disrupted traditional assessment practices. In response to this existential threat, Turnitin added an AI content detector to its platform, which is the most used plagiarism detection software in most academic institutions. However, several early studies suggest that AI detectors have questionable accuracy rates, especially when dealing with English as a second language writings or modified AI texts. To explore this uncharted territory in second language writing research, the present study evaluated the performance of Turnitin AI Detector with different text types. Using a comparative descriptive research paradigm, the researchers measured and compared Turnitin's detection accuracy in four conditions: a) English as a foreign language texts, (b) AI-generated texts, (c) paraphrased AI-generated texts, and (d) humanized AI-generated texts. The results demonstrated that Turnitin had an above-average accuracy in detecting AI texts, but its accuracy dropped slightly for paraphrased text and significantly for humanized texts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.4018/ijcallt.375428

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Detection models trained on balanced datasets including both native and non-native writing samples have demonstrated substantially reduced bias without major loss in overall accuracy.

    “Detection models trained on balanced datasets that include both native and non-native writing samples have demonstrated substantially reduced bias without major loss in overall accuracy”

    From ARE AI DETECTORS FAIR TO EFL LEARNERS? A LIBRARY RESEARCH ON DETECTION BIAS AND ACADEMIC INTEGRITY (2026) · cites this paper as Ibrahim et al., 2025

    The Literature Review (page unavailable) says Turnitin reports training on non-native writing, and the RQ#1 discussion (page unavailable) reports a 1% false-positive rate on non-native essays alongside identification of 85% of AI texts. The paper does not establish that its training dataset was balanced between native and non-native writing, or compare models to show reduced bias without a major loss in overall accuracy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Ali, W. Z. W., Ismail, H., & Cheat, T. T. (2012). Plagiarism: To what extent it is understood? Procedia: Social and Behavioral Sciences, 59, 604-611. DOI: 10.1016/j.sbspro.2012.09.320 · DOI
  2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
  3. Deng, X., Liu, Q., Deng, Y., & Mahadevan, S. (2016). An improved method to construct basic probability assignment based on the confusion matrix for classification problem. Information Sciences, 340, 250-261. DOI: 10.1016/j.ins.2016.01.033 · DOI
  4. Dornyei, Z. (2007). Research methods in applied linguistics. Oxford University Press.
  5. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861-874. DOI: 10.1016/j.patrec.2005.10.010 · DOI · Paper page
  6. Fireman Kramer, R. (1985). An overview of descriptive research. Journal of the Association of Pediatric Oncology Nurses, 2(2), 41-45. DOI: 10.1177/104345428500200208 PMID: 3848484 · DOI
  7. Francke, E., & Bennett, A. (2019). The potential influence of artificial intelligence on plagiarism: A higher education perspective. In Proceedings of the European Conference on the Impact of Artificial Intelligence and Robotics (ECIAIR 2019). Publisher.
  8. Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023). How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. DOI: 10.48550/arXiv.2301.07597 · DOI · Paper page
  9. Han, T., & Sari, E. (2022). An investigation on the use of automated feedback in Turkish EFL students' writing classes. Computer Assisted Language Learning, 37(4), 961-985. DOI: 10.1080/09588221.2022.2067179 · DOI
  10. Ibrahim, K. (2023). Using AI-based classifiers to detect AI-generated texts in ESL composition. Language Testing in Asia, 13, 46. DOI: 10.1186/s40468-023-00260-2 · DOI · Paper page
  11. Jawahar, G., Abdul-Mageed, M., & Lakshmanan, L. V. (2020). Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics. DOI: 10.18653/v1/2020.coling-main.208 · DOI · Paper page
  12. King, M. R., & Chat, G. P. T. (2023). A conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cellular and Molecular Bioengineering, 16(1), 1-2. DOI: 10.1007/s12195-022-00754-8 PMID: 36660590 · DOI
  13. Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023). A watermark for large language models. In Proceedings of the 40th International Conference on Machine Learning. https:// proceedings .mlr .press/ v202/ kirchenbauer23a .html
  14. Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns (New York, N.Y.), 4(7), 100779. DOI: 10.1016/j.patter.2023.100779 PMID: 37521038 · DOI · Paper page
  15. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. DOI: 10.48550/arXiv.1907.11692 · DOI · Paper page
  16. Lowie, W., & Seton, B. (2013). Essential statistics for applied linguistics. Bloomsbury Publishing. DOI: 10.1007/978-1-137-28490-7 · DOI
  17. Lu, N., Liu, S., He, R., & Tang, K. (2023). Large language models can be guided to evade AI-generated text detection. arXiv preprint arXiv:2305.10847. DOI: 10.48550/arXiv.2305.10847 · DOI · Paper page
  18. Mahdi, H. S., & Alkhateeb, A. (2025). Revolutionising essay evaluation: A cutting-edge rubric for AI-assisted writing. International Journal of Computer-Assisted Language Learning and Teaching, 15(1), 1-19. DOI: 10.4018/IJCALLT.368226 · DOI
  19. Nhu, V. H., Mohammadi, A., Shahabi, H., Ahmad, B. B., Al-Ansari, N., Shirzadi, A., Clague, J., Jaafari, A., Chen, W., & Nguyen, H. (2020). Landslide susceptibility mapping using machine learning algorithms and remote sensing data in a tropical environment. International Journal of Environmental Research and Public Health, 17(14), 4933. DOI: 10.3390/ijerph17144933 PMID: 32650595 · DOI
  20. Pecorari, D., & Petrić, B. (2014). Plagiarism in second-language writing. Language Teaching, 47(3), 269-302. DOI: 10.1017/S0261444814000056 · DOI
  21. Pegoraro, A., Kumari, K., Fereidooni, H., & Sadeghi, A. R. (2023). To ChatGPT, or not to ChatGPT: That is the question! The Pediatric Infectious Disease Journal, 42(4), 275. DOI: 10.1097/INF.0000000000003852 PMID: 36757192 · DOI
  22. Perkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2023). Game of tones: Faculty detection of GPT-4 generated content in university assessments. arXiv preprint arXiv:2305.18081. DOI: 10.48550/ arXiv.2305.18081 · DOI · Paper page
  23. Powers, D. M. (2011). Evaluation: From precision, recall and F-factor to ROC, informedness, markedness & correlation. Journal of Machine Learning Technologies, 2(1), 37-63. DOI: 10.9735/2229-3981 · DOI
  24. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9.
  25. Shi, H., & Aryadoust, V. (2024). A systematic review of AI-based automated written feedback research. ReCALL, 36(2), 187-209. DOI: 10.1017/S0958344023000265 · DOI · Paper page
  26. Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J., Kreps, S., McCain, M., Newhouse, A., Blazakis, J., McGuffie, K., & Wang, J. (2019). Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203. DOI: 10.48550/arXiv.1908.09203 · DOI · Paper page
  27. Susnjak, T., & McIntosh, T. (2024). ChatGPT: The end of online exam integrity? Education Sciences, 14(6), 656. DOI: 10.3390/educsci14060656 · DOI
  28. Teng, M. F., & Huang, J. (2025). Incorporating ChatGPT for EFL writing and its effects on writing engagement. International Journal of Computer-Assisted Language Learning and Teaching, 15(1), 1-21. DOI: 10.4018/ IJCALLT.367874 Turnitin (2023, Nov 30). One year of ChatGPT: Where are we now? Turnitin Blog. https:// www .turnitin .com/ blog/ one -year -of -chatgpt -where -are -we -now Turnitin (n.d.). AI writing detection capabilities. Turnitin Blog. https:// www .turnitin .com/ products/ features/ ai -writing -detection/ · DOI
  29. Waltzer, T., & Dahl, A. (2023). Why do students cheat? Perceptions, evaluations, and motivations. Ethics & Behavior, 33(2), 130-150. DOI: 10.1080/10508422.2022.2026775 · DOI
  30. Woo, D. J., Susanto, H., Yeung, C. H., Guo, K., & Fung, A. K. Y. (2024). Exploring AI generated text in student writing: How does AI help? Language Learning & Technology, 28(2), 183-209. https:// hdl .handle .net/ 10125/ 73577
  31. Yan, D. (2024). Comparing individual vs. collaborative processing of ChatGPT-generated feedback: Effects on L2 writing task improvement and learning. Language Learning & Technology, 28(1), 1-19. https:// hdl .handle .net/ 10125/ 73597
  32. Yang, X., Cheng, W., Petzold, L., Wang, W. Y., & Chen, H. (2023). DNA-GPT: Divergent N-gram analysis for training-free detection of GPT-generated text. arXiv preprint arXiv:2305.17359. DOI: 10.48550/arXiv.2305.17359 · DOI · Paper page

Cited by

Sorted by citations

Authors
Ibrahim, K. H. S., Al Otaibi, D., & Sibai, F. N.
Affiliations
  • Abdullah Al Salem University, Kuwait
  • Gulf University for Science and Technology, Kuwait
  • Gulf University for Science & Technology (KW)
ORCID
Venue
International Journal of Computer-Assisted Language Learning and Teaching
Publisher
IGI Global Scientific Publishing
Published
2025-06-20
Volume
15
Issue
1
Pages
1-27
ISSN
  • 2155-7098 (ISSN-L)
  • 2155-7101
DOI
10.4018/ijcallt.375428
OpenAlex
W4411500917
Cited by
5
References
14
Citations by year
2026: 4 · 2025: 1
FWCI
5.17
Type
journal article
Language
ng
Open access
Licence
Locations
International Journal of Computer-Assisted Language Learning and Teaching (publishedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
Keywords
AI-generated text detection, detection accuracy, ESL writing, academic integrity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review