CoChat
Sign Up
Cited by 28Academic integrity and plagiarismAuthorship Attribution and ProfilingArtificial Intelligence in Education

Can we trust academic AI detective? Accuracy and limitations of AI-output detectors

Open access
Erol, Gökberk, Ergen, Anıl, Gülşen Erol, Büşra, Kaya Ergen, Şebnem, Bora, Tevfik Serhan, Çölgeçen, Ali Deniz, Araz, Büşra, Şahin, Cansel +5 more·2025·Acta Neurochirurgica
DOI 10.1007/s00701-025-06622-4PMC PMC12331776
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVE: This study evaluates the reliability and accuracy of AI-generated text detection tools in distinguishing human-authored academic content from AI-generated texts, highlighting potential challenges and ethical considerations in their application within the scientific community. METHODS: This study analyzed the detectability of AI-generated academic content using abstracts and introductions created by ChatGPT versions 3.5, 4, and 4o, alongside human-written originals from the pre-ChatGPT era. Articles were sourced from four high impact neurosurgery journals and categorized into four categories: originals and generated by ChatGPT 3.5, ChatGPT 4, and ChatGPT 4o. AI-output detectors (GPTZero, ZeroGPT, Corrector App) were employed to classify 1,000 texts as human- or AI-generated. Additionally, plagiarism checks were performed on AI-generated content to evaluate uniqueness. RESULTS: A total of 250 human-authored articles and 750 ChatGPT-generated texts were analyzed using three AI-output detectors (Corrector, ZeroGPT, GPTZero). Human-authored texts consistently had the lowest AI likelihood scores, while AI-generated texts exhibited significantly higher scores across all versions of ChatGPT (p < 0.01). Plagiarism detection revealed high originality for ChatGPT-generated content, with no significant differences among versions (p > 0.05). ROC analysis demonstrated that AI-output detectors effectively distinguished AI-generated content from human-written texts, with areas under the curve (AUC) ranging from 0.75 to 1.00 for all models. However, none of the detectors achieved 100% reliability in distinguishing AI-generated content. CONCLUSIONS: While models like ChatGPT enhance content creation and efficiency, they raise ethical concerns, particularly in fields demanding trust and precision. AI-output detectors exhibit moderate to high success in distinguishing AI-generated texts, but false positives pose risks to researchers. Improving detector reliability and establishing clear policies on AI usage are critical to mitigate misuse while fully leveraging AI's benefits.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00701-025-06622-4

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Liu et al. (2024) and Erol et al. (2025) found that AI detection tools and language models show significant cultural and linguistic bias.

    “These insights parallel Liu et al. (2024) and Erol et al. (2025), who found that AI detection tools and language models show significant cultural and linguistic bias.”

    From Ethical Concerns and Institutional Policy Responses to Artificial Intelligence (AI) in Higher Education Across Sub-Saharan Africa (2025) · cites this paper as Erol et al. (2025)

    The Introduction mentions that language-based AI can amplify existing biases in data and algorithms, and the Discussion finds that AI-output detectors are not always reliable. Neither passage reports a finding of significant cultural or linguistic bias in the detectors or language models, so the claim is more specific than this paper supports. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Erol, Aydemir, and Koc (2025) found that AI-text detection systems are often unreliable and biased.

    “Erol, Aydemir, and Koc (2025) found that AI-text detection systems are often unreliable and biased, reinforcing the need for ethical literacy rather than punitive surveillance.”

    From Ethical Concerns and Institutional Policy Responses to Artificial Intelligence (AI) in Higher Education Across Sub-Saharan Africa (2025) · cites this paper as Erol, Aydemir, and Koc (2025)

    The Discussion (page unavailable) says AI detection tools alone are not always reliable and reports false positives for human-written articles. It does not establish that the systems are generally biased; its mention of bias concerns AI systems more broadly. The recommendation for ethical literacy is the citing authors’ framing, not a finding attributed to this source. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [1]Alkaissi H, McFarlane SI (2023) Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus 15(2):e35179 36811129 PMC9939079
  2. [2]Anderson N, Belavy DL, Perle SM, Hendricks S, Hespanhol L, Verhagen E, Memon AR (2023) AI did not write this manuscript, or did it? Can we trick the AI text detector into generated texts? The potential future of ChatGPT and AI in sports & exercise medicine manuscript generation. BMJ Open Sport Exerc Med 9(1):e001568. 10.1136/bmjsem-2023-001568 PMC9936276 36816423 · DOI · Paper page
  3. [3]Atici MA, Sagiroglu S, Celtikci P, Ucar M, Borcek AO, Emmez H et al (2020) A novel deep learning algorithm for the automatic detection of high-grade gliomas on T2-weighted magnetic resonance images: a preliminary machine learning study. Turk Neurosurg 30(2):199–205 31608975
  4. [4]Bassiri-Tehrani B, Cress PE (2023) Unleashing the power of ChatGPT: revolutionizing plastic surgery and beyond. Aesthet Surg J 43(11):1395–1399 37154803
  5. [5]Bhatia P (2023) ChatGPT for academic writing: a game changer or a disruptive tool? J Anaesthesiol Clin Pharmacol 39:1–2 37250265 PMC10220188
  6. [6]Boaro A, Mezzalira E, Siddi F, Bagattini C, Gabrovsky N, Marchesini N et al (2025) Knowledge, interest and perspectives on artificial intelligence in neurosurgery. A global survey. Brain Spine 5:104156 39802868 PMC11721513
  7. [8]Cingillioglu I (2023) Detecting AI-generated essays: the ChatGPT challenge. Int J Inf Learn Technol 40(3):259–268
  8. [9]Duman EA, Sagiroglu S, Celtikci P, Demirezen MU, Borcek AO, Emmez H et al (2022) Utilizing deep convolutional generative adversarial networks for automatic segmentation of gliomas: an artificial intelligence study. Turk Neurosurg 32(1):16–21 34542897
  9. [10]Dutton JJ (2023) Artificial intelligence and the future of computer-assisted medical research and writing. Ophthalmic Plast Reconstr Surg. Wolters Kluwer Health 39:203–205 37166288
  10. [12]Hua HU, Kaakour AH, Rachitskaya A, Srivastava S, Sharma S, Mammo DA (2023) Evaluation and comparison of ophthalmic scientific abstracts and references by current artificial intelligence chatbots. JAMA Ophthalmol 141(9):819–824 37498609 PMC10375387
  11. [13]Huang KT, Mehta NH, Gupta S, See AP, Arnaout O (2024) Evaluation of the safety, accuracy, and helpfulness of the GPT-4.0 large language model in neurosurgery. J Clin Neurosci 1(123):151–156 38574687
  12. [15]Keskinbora KH (2019) Medical ethics considerations on artificial intelligence. J Clin Neurosci 64:277–82 30878282
  13. [17]Kozel G, Gurses ME, Gecici NN, Gökalp E, Bahadir S, Merenzon MA et al (2024) Chat-GPT on brain tumors: an examination of artificial intelligence/machine learning’s ability to provide diagnoses and treatment plans for example neuro-oncology cases. Clin Neurol Neurosurg 1:239 38507989
  14. [18]Makiev KG, Asimakidou M, Vasios IS, Keskinis A, Petkidis G, Tilkeridis K et al (2023) A Study on Distinguishing ChatGPT-Generated and Human-Written Orthopaedic Abstracts by Reviewers: Decoding the Discrepancies. Cureus 15(11):e49166 PMC10733892 38130535
  15. [19]Nernekli K, Persad AR, Hori YS, Yener U, Celtikci E, Sahin MC, Sozer A, Sozer B, Park DJ, Chang SD (2024) Automatic segmentation of vestibular schwannomas: a systematic review. World Neurosurg 188:35–44. 10.1016/j.wneu.2024.04.145 38685346 · DOI
  16. [20]O’Connor S, ChatGPT. Open artificial intelligence platforms in nursing education: tools for academic progress or abuse? Nurse Educ Pract. 2023;66:103537. Elsevier Ltd. 36549229
  17. [21]Pressman SM, Borna S, Gomez-Cabello CA, Haider SA, Haider C, Forte AJ (2024) AI and ethics: a systematic review of the ethical considerations of large language model use in surgery research. Healthcare (Basel) 12(8):825. 10.3390/healthcare12080825 38667587 PMC11050155 · DOI
  18. [22]Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, Radiological Society of North America Inc et al (2023) ChatGPT and other large language models are double-edged swords. Radiology 307(2):e230163 36700838
  19. [23]Rashidi HH, Fennell BD, Albahra S, Hu B, Gorbett T (2023) The ChatGPT conundrum: human-generated scientific manuscripts misidentified as AI creations by AI text detection tool. J Pathol Inform 14:100342. 10.1101/2022.12.23.521610 38116171 PMC10727991 · DOI · Paper page
  20. [24]Rashidi HH, Tran NK, Betts EV, Howell LP, Green R (2019) Artificial intelligence and machine learning in pathology: the present landscape of supervised methods. Acad Pathol 6:2374289519873088. 10.1177/2374289519873088 31523704 PMC6727099 · DOI
  21. [25]Reddy S (2022) Explainability and artificial intelligence in medicine. Lancet Digit Health 4(4):e214–e215. 10.1016/S2589-7500(22)00029-2 35337639 · DOI
  22. [26]Sahin MC, Sozer A, Kuzucu P, Turkmen T, Sahin MB, Sozer E et al (2024) Beyond human in neurosurgical exams: ChatGPT’s success in the Turkish neurosurgical society proficiency board exams. Comput Biol Med 1:169 38091727
  23. [27]Sevgi UT, Erol G, Doğruel Y, Sönmez OF, Tubbs RS, Güngor A (2023) The role of an open artificial intelligence platform in modern neurosurgical education: a preliminary study. Neurosurg Rev 46(1):86 37059815
  24. [28]Sozer A, Borcek AO, Sagiroglu S, Poshtkouh A, Demirtas Z, Karaaslan MM et al (2023) The first case of glioma detected by an artificial intelligence algorithm running on real-time data in neurosurgery: illustrative case. J Neurosurg: Case Less 5(19):CASE22536 PMC10550689 37158388
  25. [29]Stengel FC, Stienen MN, Ivanov M, Gandía-González ML, Raffa G, Ganau M et al (2024) Can AI pass the written European board examination in neurological surgery? Brain and Spine 4:102765 38510593 PMC10951784
  26. [30]Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW (2023) Large language models in medicine. Nat Med 29(8):1930–1940 37460753
  27. [33]Taguchi K, Gu Y, Sakurai K (2024)The Impact of Prompts on Zero-Shot Detection of AI-Generated Text. Available from: http://arxiv.org/abs/2403.20127. Accessed 29 Mar 2025 · arXiv
  28. [35]Mathiesen T, Broekman M (2022) Machine Learning and Ethics, pp 251–256. Available from: https://link.springer.com/10.1007/978-3-030-85292-4_28 34862548 · DOI

Cited by

Sorted by citations

Authors
Erol, G., Ergen, A., Gülşen Erol, B., Kaya Ergen, Ş., Bora, T. S., Çölgeçen, A. D., Araz, B., Şahin, C., Bostancı, G., Kılıç, İ., Macit, Z. B., Sevgi, U. T., & Güngör, A.
Affiliations
  • Adıyaman Eğitim ve Araştırma Hastanesi (TR)
  • Health Sciences University Derince Training and Research Hospital (TR)
  • University of Pavia (IT)
  • Zonguldak Bülent Ecevit University (TR)
  • Trakya University (TR)
  • Ankara University (TR)
  • Bahçeşehir University (TR)
  • İzmir Şehir Hastanesi (TR)
  • Istinye University (TR)
Corresponding author
Güngör, Abuzer
ORCID
Venue
Acta Neurochirurgica
Publisher
Springer Science and Business Media LLC
Published
2025-08-07
Volume
167
Issue
1
Pages
214-214
ISSN
  • 0001-6268 (ISSN-L)
  • 0942-0940
DOI
10.1007/s00701-025-06622-4
PMC
PMC12331776
OpenAlex
W4413033417
Cited by
28
References
6
Citations by year
2026: 28
FWCI
28.97
Type
journal article
Language
en
Open access
Licence
Locations
  • Acta Neurochirurgica (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Kocaeli Üniversitesi - AVESİS (submittedVersion)
  • Zonguldak Bülent Ecevit University Institutional Repository (submittedVersion)
  • Trakya University's Institutional Open Access System (Trakya University) (submittedVersion, other-oa)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
AI-generated text detection, ChatGPT, GPTZero, academic integrity, AI-generated research, plagiarism detection, neurosurgery journals, AI ethics, false positives
MeSH terms
Artificial Intelligence, Authorship, Humans, Neurosurgery, Periodicals as Topic, Reproducibility of Results, Plagiarism

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review