The DOI matches its DataCite record; 28 of the 29 references it cites were found in a registry or an index, and 1 are not checked yet.
Differentiate ChatGPT-generated and Human-written Medical Texts
Open accessAbstract
Background: Large language models such as ChatGPT are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the Internet. However, medical texts such as clinical notes and diagnoses require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to healthcare and the general public. Objective: This research is among the first studies on responsible and ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus on analyzing the differences between medical texts written by human experts and generated by ChatGPT, and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods: We first construct a suite of datasets containing medical texts written by human experts and generated by ChatGPT. In the next step, we analyze the linguistic features of these two types of content and uncover differences in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally, we design and implement machine learning methods to detect medical text generated by ChatGPT. Results: Medical texts written by humans are more concrete, more diverse, and typically contain more useful information, while medical texts generated by ChatGPT pay more attention to fluency and logic, and usually express general terminologies rather than effective information specific to the context of the problem. A BERT-based model can effectively detect medical texts generated by ChatGPT, and the F1 exceeds 95%.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In contrast, LLM-generated papers exhibit fewer grammatical and other types of errors and demonstrate a broader variety of expression (Yan et al. 2023; Liao et al. 2023a).”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Liao et al. 2023a
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine-generated text has a more complex syntactic structure and uses more normalization, whereas human essays tend to be more lexically complex [81,83].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [83]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent research (Giorgi et al. 2023; Liao et al. 2023b) has elucidated distinct disparities between LLM-generated text and human-written text, underscoring a tangible and discernible gap between the two.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Liao et al. 2023b
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In an experiment, AI models were able to generate more high-quality argumentative essays than German high school students in an online forum [83].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [83]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
Self-citationChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
Self-citation
Research works printed but not linked: 22
- Devlin J, Chang MW, Lee K, Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding. 2019 Presented at: North American Chapter of the Association for Computational Linguistics: Human Language Technologies; June 2-7, 2019; Minneapolis p. 4171-4186
- Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:1-11
- Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877-1901
- Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al. Training language models to follow instructions with human feedback. arXiv. Preprint posted online on March 4, 2022. [FREE Full text]
- Christiano PF, Leike J, Brown T, Martic M, Legg S, Amodei D. Deep reinforcement learning from human preferences. Adv Neural Inf Process Syst. 2017;30:1-9
- Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv. Preprint posted online on July 20, 2017. [FREE Full text]
- Guan Z, Wu Z, Liu Z, Wu D, Ren H, Li Q, et al. Cohortgpt: an enhanced gpt for participant recruitment in clinical study. arXiv. Preprint posted online on July 21, 2023. [FREE Full text]
- Dai H, Liu Z, Liao W, Huang X, Wu Z, Zhao L, et al. AugGPT: leveraging ChatGPT for text data augmentation. arXiv. Preprint posted online on February 25, 2023. [FREE Full text]
- Ma C, Wu Z, Wang J, Xu S, Wei Y, Liu Z, et al. ImpressionGPT: an iterative optimizing framework for radiology report summarization with ChatGPT. arXiv. Preprint posted online on April 17, 2023. [FREE Full text]
- Liu Z, Yu X, Zhang L, Wu Z, Cao C, Dai H, et al. Deid-GPT: zero-shot medical text de-identification by GPT-4. arXiv. Preprint posted online on March 20, 2023. [FREE Full text]
- Shi Y, Xu S, Liu Z, Liu T, Li X, Liu N. MedEdit: model editing for medical question answering with external knowledge bases. arXiv. Preprint posted online on September 27, 2023. [FREE Full text]
- Mitrović S, Andreoletti D, Ayoub O. ChatGPT or human? Detect and explain. Explaining decisions of machine learning model for detecting short ChatGPT-generated text. arXiv. Preprint posted online on January 30, 2023. [FREE Full text]
- Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. RoBERTa: a robustly pptimized BERT pretraining approach. arXiv. Preprint posted online on July 26, 2019. [FREE Full text]
- Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv. Preprint posted online on Septemeber 26, 2019. [FREE Full text]
- Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. Semantic Scholar. 2019. URL: https://www.semanticscholar.org/paper/ Language-Models-are-Unsupervised-Multitask-Learners-Radford-Wu/9405cc0d6169988371b2755e573cc28650d14dfe [accessed 2023-10-18]
- Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res. 2020;21(1):1-67
- Liao W, Liu Z, Dai H, Wu Z, Zhang Y, Huang X, et al. Mask-guided BERT for few shot text classification. arXiv. Preprint posted online on February 21, 2023. [FREE Full text]
- Cai H, Liao W, Liu Z, Huang X, Zhang Y, Ding S, et al. Coarse-to-fine knowledge graph domain adaptation based on distantly-supervised iterative training. arXiv. Preprint posted online on November 5, 2022. [FREE Full text]
- Liu Z, Li Y, Shu P, Zhong A, Yang L, Ju C, et al. Radiology-Llama2: best-in-class large language model for radiology. arXiv. Preprint posted online on August 29, 2023. [FREE Full text] [doi: 10.1007/978-3-031-45673-2_46] · DOI
- Guo B, Zhang X, Wang Z, Jiang M, Nie J, Ding Y, et al. How close is chatgpt to human experts? Comparison corpus, evaluation, and detection. arXiv. Preprint posted online on January 18, 2023. [FREE Full text]
- Bird S, Klein E, Loper E. Natural Language Processing with Python -Analyzing Text with the Natural Language Toolkit. Sebastopol. O'Reilly Media; 2009.
- Carlini N, Tramer F, Wallace E, Jagielski M, Herbert-Voss A, Lee K, et al. Extracting training data from large language models. In: Proceedings of the 30th USENIX Security Symposium. 2021 Presented at: 30th USENIX Security Symposium; August 11-13, 2021; Vancouver
Cited by
Sorted by citations
Items 101–120 of 120
Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts
Baştürk, B., & Onan, A. (2025) · Scientific journal of Mehmet Akif Ersoy University. · doi:10.70030/sjmakeu.1730246
Ashour, S. (2025) · The political economy of the Middle East/The political economy of the Middle East · doi:10.1007/978-981-96-7963-8_4
Amplifying signal-to-noise: Responsible use of large language models in radiology publishing
Song, A. S., Masutani, E., Tejani, A. S., & Retson, T. (2025) · Clinical Imaging · doi:10.1016/j.clinimag.2025.110679 · Cited by 1
How Do LLM-Generated Texts Impact Term-Based Retrieval Models?
Huang, W., Bi, K., Cai, Y., Chen, W., Guo, J., & Cheng, X. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777988 · Cited by 3
Zhang, H., Zhang, Z., Liu, S., & Li, C. (2026) · Journal of Hospitality and Tourism Technology · doi:10.1108/jhtt-05-2025-0438 · Cited by 3
Chaleendra, U. R., Nitin, V., Mahesh, B., Lahari, Y. L., Shashank, K., & Anil, B. (2026) · International Conference on Expert Clouds and Applications (ICOECA) · doi:10.1109/icoeca68095.2026.11485495
Mourão, C. F., Juliasse, L. E. R., & Pereira, R. D. S. (2026) · International Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.ijom.2026.04.017
Grillo, R., Neto, L. S. C., Bottura, L., Llanos, A. H., Samieirad, S., & Melhem-Elias, F. (2026) · International Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.ijom.2026.04.020
Out of Order: On the Importance of Word Positions in Explaining Text Classification
Stockdiek, J. L., Britta, G., Griesbach, M., & Grimme, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21477-5_29
Distinguishing Human and AI-Generated Text: A BiLSTM-Based Deep Learning Approach
Sumaya, S. T., Arnob, H. I., Ahmed, A., Sultana, A., & Rifat, A. A. S. (2026) · 2026 IEEE 2nd International Conference on Quantum Photonics, Artificial Intelligence & Networking (QPAIN) · doi:10.1109/qpain69676.2026.11546461
Responsible Use of GenAI Technologies in Health Communication: Translating the Covid-19 Pandemic
Çetin, Ö., Eser, O., & Durán, A. (2026) · Öğretim Teknolojisi ve Hayat Boyu Öğrenme Dergisi - Instructional Technology and Lifelong Learning · doi:10.52911/itall.1870352
Qiu, W., & Pan, F. (2026) · Scientometrics · doi:10.1007/s11192-026-05701-2 · Cited by 1
AI-Generated Fake Review Detection in E-Commerce: A Unified Taxonomy and Multi-Stakeholder Roadmap
Singla, S., & Handa, D. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100359
Dual-View Learning for Robust Detection of AI-Generated Text
Zita, K., Bellaouar, S., Dahma, M., & Bensania, A. (2026) · International Conference on Pattern Analysis and Intelligent Systems (PAIS) · doi:10.1109/pais69842.2026.11605378
Who is mimicking whom? The mutual shaping effect between AI and college students’ English writing
Qin, X. (2026) · Computer Assisted Language Learning · doi:10.1080/09588221.2026.2723281
Inooka, T., Yuki, K., & Nishiguchi, K. M. (2026) · Ophthalmology Science · doi:10.1016/j.xops.2026.101418
Sivakumar, W., Singh, R. A., Garras, Y., & Hennessey, D. B. (2026) · JMIR Formative Research · doi:10.2196/104063
Multidimensional linguistic evaluation of LLM-generated and resident-rewritten discharge summaries
Trindade, R. P. D. A., Senra, L. P. C., Feitosa, G. F., & dos Santos, E. R. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106770
Robust Essay Detection: Small Language Models Adversarial Parameter-Efficient Fine-Tuning
Herman, A., Cristin, P., Aryati, P., Purwitasari, D., Anggraini, R., & Faizin, M. (2026) · 2026 International Seminar on Application for Technology of Information and Communication (iSemantic) · doi:10.1109/isemantic71495.2026.11742641
Metadata
- Authors
- Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Liu, T., & Li, X.
- Affiliations
- South China University of Technology (CN)
- University of Georgia (US)
- The University of Texas at Arlington (US)
- Massachusetts General Hospital (US)
- Corresponding author
- Li, Xiang
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 9
- Pages
- e48904-e48904
- ISSN
- 2369-3762 (ISSN-L)
- arXiv
- 2304.11567
- PMC
- PMC10784984
- OpenAlex
- W4390347092
- Cited by
- 119
- References
- 28
- Citations by year
- 2026: 26 · 2025: 36 · 2024: 42 · 2023: 15
- FWCI
- 11.22
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U21A20520, 62172112)
- National Key Research and Development Program of China (2022YFE0112200)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- ChatGPT, large language models, AI-generated text detection, medical texts, clinical notes, disinformation, linguistic features, perplexity, BERT, responsible AI
- MeSH terms
- Disinformation, Algorithms, Artificial Intelligence, Health Facilities, Humans, Electric Power Supplies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
