Between human and AI: assessing the reliability of AI text detection tools
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: Large language models (LLMs) such as ChatGPT-4 have raised critical questions regarding their distinguishability from human-generated content. In this research, we evaluated the effectiveness of online detection tools in identifying ChatGPT-4 vs human-written text. METHODS: A two texts produced by ChatGPT-4 using differing prompts and one text created by a human author were analytically assessed using the following online detection tools: GPTZero, ZeroGPT, Writer ACD, and Originality. RESULTS: The findings revealed a notable variance in the detection capabilities of the employed detection tools. GPTZero and ZeroGPT exhibited inconsistent assessments regarding the AI-origin of the texts. Writer ACD predominantly identified texts as human-written, whereas Originality consistently recognized the AI-generated content in both samples from ChatGPT-4. This highlights Originality's enhanced sensitivity to patterns characteristic of AI-generated text. CONCLUSION: The study demonstrates that while automatic detection tools may discern texts generated by ChatGPT-4 significant variability exists in their accuracy. Undoubtedly, there is an urgent need for advanced detection tools to ensure the authenticity and integrity of content, especially in scientific and academic research. However, our findings underscore an urgent need for more refined detection methodologies to prevent the misdetection of human-written content as AI-generated and vice versa.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The use of artificial intelligence to improve the scientific writing of non-native english speakers
Giglio, A. D., & Costa, M. U. P. D. (2023) · Revista da Associação Médica Brasileira · doi:10.1590/1806-9282.20230560 · Cited by 158 · cited 1×
“Therefore, Giglio et al. 14 suggest that these tools could lead to a reduction in inequalities between native and non-native English-speaking researchers.”
BackgroundIbrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154 · cited 1×
“On the other hand, the extraordinary speed with which tools like ChatGPT-4 are improving their generative capacity has both intriguing and concerning aspects 13 .”
BackgroundArtificial intelligence in academic writing: a paradigm-shifting technological advance
Golan, R., Reddy, R., Muthigi, A., & Ramasamy, R. (2023) · Nature Reviews Urology · doi:10.1038/s41585-023-00746-x · Cited by 136
Sharma, M., & Sharma, S. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105972 · Cited by 76
Chat GPT and artificial intelligence in medical writing: concerns and ethical considerations
Doyal, A. S., Sender, D., Nanda, M., & Serrano, R. A. (2023) · Cureus · doi:10.7759/cureus.43292 · Cited by 65
Scientists used ChatGPT to generate an entire paper from scratch — but is it any good?
Conroy, G. (2023) · Nature · doi:10.1038/d41586-023-02218-z · Cited by 65
Research works printed but not linked: 12
- 10.24875/GMM.M23000795 · DOI
- 10.1177/25152459211061337 · DOI
- 10.1056/AIe2300197 · DOI
- 10.3389/frai.2023.1166014 · DOI · Paper page
- 10.21037/jmai-23-13 · doi:10.21037/jmai-23-13
- 10.17085/apm.23056 · DOI · Paper page
- 10.1016/j.cris.2023.100057 · DOI
- 10.2196/50591 · DOI
- 10.1186/s44158-023-00118-2 · DOI
- 10.1007/s12630-023-02565-5 · DOI
- 10.1186/s13054-023-04598-0 · DOI
- 10.1007/s10916-023-01925-4 · DOI · Paper page
Cited by
AI and academic integrity: Exploring student perceptions and implications for higher education
Lund, B. D., Lee, T. H., Mannuru, N. R., & Arutla, N. (2025) · Journal of Academic Ethics · doi:10.1007/s10805-025-09613-3 · Cited by 96
“There are detectors that can be used to identify potential usage of AI tools – though they are imperfect (Bellini et al, 2024).”
Kim, S.-J. (2024) · Science Editing · doi:10.6087/kcse.343 · Cited by 42
Hua, H., & Yao, C.-J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1469197 · Cited by 27
Kangwa, D., Msafiri, M. M., & Fute, A. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70109 · Cited by 22
AI detecting AI in academic writing: Why most AI detector findings are false
Tsigaris, P., & Teixeira da Silva, J. A. (2026) · Next Research · doi:10.1016/j.nexres.2026.101396 · Cited by 11
Metadata
- Authors
- Bellini, V., Semeraro, F., Montomoli, J., Cascella, M., & Bignami, E.
- Venue
- Current Medical Research and Opinion
- Publisher
- Informa UK Limited
- Published
- 2024-02-02
- Volume
- 40
- Issue
- 3
- Pages
- 353-358
- ISSN
- 0300-7995 (ISSN-L)
- 1473-4877
- OpenAlex
- W4391169883
- Cited by
- 65
- References
- 6
- FWCI
- 9.52
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Between human and AI: assessing the reliability of AI text detection tools. https://app.cochat.ai/papers/10.1080/03007995.2024.2310086
- The paper
- Bellini, V., Semeraro, F., Montomoli, J., Cascella, M., & Bignami, E. (2024). Between human and AI: assessing the reliability of AI text detection tools. Current Medical Research and Opinion. https://doi.org/10.1080/03007995.2024.2310086
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.