CoChat
Sign Up
Cited by 58Academic integrity and plagiarismAuthorship Attribution and ProfilingArtificial Intelligence in Healthcare and Education

The great detectives: humans versus AI detectors in catching large language model-generated medical writing

Open access
Liu, Jae Q. J., Hui, Kelvin T. K., Al Zoubi, Fadi, Zhou, Zing Z. X., Samartzis, Dino, Yu, Curtis C. H., Chang, Jeremy R., Wong, Arnold Y. L.·2024·International Journal for Educational Integrity
DOI 10.1007/s40979-024-00155-6
Open this paper in CoChat — free account

Abstract

Background: The application of artificial intelligence (AI) in academic writing has raised concerns regarding accuracy, ethics, and scientific rigour. Some AI content detectors may not accurately identify AI-generated texts, especially those that have undergone paraphrasing. Therefore, there is a pressing need for efficacious approaches or guidelines to govern AI usage in specific disciplines. Objective: Our study aims to compare the accuracy of mainstream AI content detectors and human reviewers in detecting AI-generated rehabilitation-related articles with or without paraphrasing. Study design: This cross-sectional study purposively chose 50 rehabilitation-related articles from four peer-reviewed journals, and then fabricated another 50 articles using ChatGPT. Specifically, ChatGPT was used to generate the introduction, discussion, and conclusion sections based on the original titles, methods, and results. Wordtune was then used to rephrase the ChatGPT-generated articles. Six common AI content detectors (Originality.ai, Turnitin, ZeroGPT, GPTZero, Content at Scale, and GPT-2 Output Detector) were employed to identify AI content for the original, ChatGPT-generated and AI-rephrased articles. Four human reviewers (two student reviewers and two professorial reviewers) were recruited to differentiate between the original articles and AI-rephrased articles, which were expected to be more difficult to detect. They were instructed to give reasons for their judgements. Results: Originality.ai correctly detected 100% of ChatGPT-generated and AI-rephrased texts. ZeroGPT accurately detected 96% of ChatGPT-generated and 88% of AI-rephrased articles. The areas under the receiver operating characteristic curve (AUROC) of ZeroGPT were 0.98 for identifying human-written and AI articles. Turnitin showed a 0% misclassification rate for human-written articles, although it only identified 30% of AI-rephrased articles. Professorial reviewers accurately discriminated at least 96% of AI-rephrased articles, but they misclassified 12% of human-written articles as AI-generated. On average, students only identified 76% of AI-rephrased articles. Reviewers identified AI-rephrased articles based on ‘incoherent content’ (34.36%), followed by ‘grammatical errors’ (20.26%), and ‘insufficient evidence’ (16.15%). Conclusions and relevance: This study directly compared the accuracy of advanced AI detectors and human reviewers in detecting AI-generated medical writing after paraphrasing. Our findings demonstrate that specific detectors and experienced reviewers can accurately identify articles generated by Large Language Models, even after paraphrasing. The rationale employed by our reviewers in their assessments can inform future evaluation strategies for monitoring AI usage in medical education or publications. AI content detectors may be incorporated as an additional screening tool in the peer-review process of academic journals.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s40979-024-00155-6

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Liu et al. (2024) and Erol et al. (2025) found that AI detection tools and language models show significant cultural and linguistic bias.

    “These insights parallel Liu et al. (2024) and Erol et al. (2025), who found that AI detection tools and language models show significant cultural and linguistic bias.”

    From Ethical Concerns and Institutional Policy Responses to Artificial Intelligence (AI) in Higher Education Across Sub-Saharan Africa (2025) · cites this paper as Liu et al. (2024)

    The Discussion (page not provided) reports prior research suggesting that AI detectors may penalize non-native English writers, which supports a concern about linguistic bias. It does not report that Liu et al. found significant cultural bias or bias in language models, and it presents the detector concern as a possibility rather than a finding of this study. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Such high detection rates have not been widely observed across other disciplines or with newer large language model versions.

    “It is notable that such high detection rates have not been widely observed across other disciplines or with newer large language model versions.”

    From Applications, Challenges, and Prospects of Generative Artificial Intelligence Empowering Medical Education: Scoping Review (2025) · cites this paper as [27]

    The Introduction reports inconsistent detector accuracy across studies in different domains, which supports caution about generalizing the high detection rates. But it also reports excellent accuracy for some detectors across several disciplines, so it does not establish that high rates have not been widely observed there. The Discussion (Limitations) says the study used ChatGPT-3.5 and calls for research on upgraded LLMs; it provides no detection rates for newer versions. No page numbers are available in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The detection study focused specifically on rehabilitation medicine with analyzed content limited to partial article sections rather than full texts.

    “The study focused specifically on rehabilitation medicine with analyzed content limited to partial article sections rather than full texts.”

    From Applications, Challenges, and Prospects of Generative Artificial Intelligence Empowering Medical Education: Scoping Review (2025) · cites this paper as [27]

    The Methods confirm that the study selected rehabilitation-related articles and that authentic methods and results sections were not entered into the AI content detectors. But the fabricated articles included those original sections alongside generated sections, and human reviewers assessed the articles. The claim therefore overstates the partial-section limitation as applying to all analyzed content. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The specialized nature of medical writing may influence detection outcomes in ways not seen in broader academic contexts.

    “The specialized nature of medical writing, including technical terminology use, may also influence these outcomes in ways not seen in broader academic contexts”

    From Applications, Challenges, and Prospects of Generative Artificial Intelligence Empowering Medical Education: Scoping Review (2025) · cites this paper as [27]

    The Introduction (page unavailable) says AI-detector accuracy varies across domains, and the Conclusions (page unavailable) note that ChatGPT may misuse medical abbreviations. These passages support concern about detecting medical writing, but the paper does not establish that technical terminology changes detection outcomes or compare that effect with broader academic contexts. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Other work has used these features, or language models trained on sample texts, to classify LLM-written texts with varying degrees of success.

    “Other work has used these features, or language models trained on sample texts, to classify LLM-written texts with varying degrees of success (15–17).”

    From Do LLMs write like humans? Variation in grammatical and rhetorical styles (2025) · cites this paper as [15]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Sadasivan V, Kumar A, Balasubramanian S, Wang W, Feizi S (2023) Can AI-generated text be reliably detected? arXiv e-prints: 2303.11156 · arXiv
  2. Sallam M (2023) ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare MDPI 887:1
  3. Sinha RK, Deb Roy A, Kumar N, Mondal H (2023) Applicability of ChatGPT in assisting to solve higher order problems in pathology. Cureus 15(2):e35237

Cited by

Sorted by citations

Related

Authors
Liu, J. Q. J., Hui, K. T. K., Al Zoubi, F., Zhou, Z. Z. X., Samartzis, D., Yu, C. C. H., Chang, J. R., & Wong, A. Y. L.
Affiliations
  • Hong Kong Polytechnic University (HK)
  • Rush University Medical Center (US)
Corresponding author
Wong, Arnold Y. L.
ORCID
Venue
International Journal for Educational Integrity
Publisher
Springer Science and Business Media LLC
Published
2024-05-20
Volume
20
Issue
1
ISSN
1833-2595 (ISSN-L)
DOI
10.1007/s40979-024-00155-6
OpenAlex
W4397292525
Cited by
56
References
19
Citations by year
2026: 22 · 2025: 25 · 2024: 9
FWCI
23.32
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal for Educational Integrity (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (acceptedVersion)
  • PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
AI-generated text detection, large language models, ChatGPT, paraphrase, medical writing, Turnitin, academic integrity, AI in academic writing, peer review, detection accuracy comparison, GPTZero

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review