CoChat

Between human and AI: assessing the reliability of AI text detection tools

Bellini, Valentina, Semeraro, Federico, Montomoli, Jonathan, Cascella, Marco, Bignami, Elena · 2024 · Current Medical Research and Opinion
Cited by 65

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Abstract

OBJECTIVE: Large language models (LLMs) such as ChatGPT-4 have raised critical questions regarding their distinguishability from human-generated content. In this research, we evaluated the effectiveness of online detection tools in identifying ChatGPT-4 vs human-written text. METHODS: A two texts produced by ChatGPT-4 using differing prompts and one text created by a human author were analytically assessed using the following online detection tools: GPTZero, ZeroGPT, Writer ACD, and Originality. RESULTS: The findings revealed a notable variance in the detection capabilities of the employed detection tools. GPTZero and ZeroGPT exhibited inconsistent assessments regarding the AI-origin of the texts. Writer ACD predominantly identified texts as human-written, whereas Originality consistently recognized the AI-generated content in both samples from ChatGPT-4. This highlights Originality's enhanced sensitivity to patterns characteristic of AI-generated text. CONCLUSION: The study demonstrates that while automatic detection tools may discern texts generated by ChatGPT-4 significant variability exists in their accuracy. Undoubtedly, there is an urgent need for advanced detection tools to ensure the authenticity and integrity of content, especially in scientific and academic research. However, our findings underscore an urgent need for more refined detection methodologies to prevent the misdetection of human-written content as AI-generated and vice versa.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/03007995.2024.2310086

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 6 references

Research works printed but not linked: 12

  1. 10.24875/GMM.M23000795 · DOI
  2. 10.1177/25152459211061337 · DOI
  3. 10.1056/AIe2300197 · DOI
  4. 10.3389/frai.2023.1166014 · DOI · Paper page
  5. 10.21037/jmai-23-13 · doi:10.21037/jmai-23-13
  6. 10.17085/apm.23056 · DOI · Paper page
  7. 10.1016/j.cris.2023.100057 · DOI
  8. 10.2196/50591 · DOI
  9. 10.1186/s44158-023-00118-2 · DOI
  10. 10.1007/s12630-023-02565-5 · DOI
  11. 10.1186/s13054-023-04598-0 · DOI
  12. 10.1007/s10916-023-01925-4 · DOI · Paper page

Cited by

See all 65

Metadata

Authors
Bellini, V., Semeraro, F., Montomoli, J., Cascella, M., & Bignami, E.
Venue
Current Medical Research and Opinion
Published
2024-02-02
Volume
40
Issue
3
Pages
353-358
ISSN
  • 0300-7995 (ISSN-L)
  • 1473-4877
OpenAlex
W4391169883
Cited by
65
References
6
FWCI
9.52
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Between human and AI: assessing the reliability of AI text detection tools. https://app.cochat.ai/papers/10.1080/03007995.2024.2310086
The paper
Bellini, V., Semeraro, F., Montomoli, J., Cascella, M., & Bignami, E. (2024). Between human and AI: assessing the reliability of AI text detection tools. Current Medical Research and Opinion. https://doi.org/10.1080/03007995.2024.2310086

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.