CoChat
Sign Up
PreprintCited by 97Artificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)Machine Learning in Healthcare

ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports

Open access
Jeblick, Katharina, Schachtner, Balthasar, Dexl, Jakob, Mittermeier, Andreas, Stüber, Anna Theresa, Topalis, Johanna, Weber, Tobias, Wesp, Philipp +3 more·2022·arXiv
DOI 10.48550/arxiv.2212.14882arXiv 2212.14882
Open this paper in CoChat — free accountOpen PDF

Abstract

The release of ChatGPT, a language model capable of generating text that appears human-like and authentic, has gained significant attention beyond the research community. We expect that the convincing performance of ChatGPT incentivizes users to apply it to a variety of downstream tasks, including prompting the model to simplify their own medical reports. To investigate this phenomenon, we conducted an exploratory case study. In a questionnaire, we asked 15 radiologists to assess the quality of radiology reports simplified by ChatGPT. Most radiologists agreed that the simplified reports were factually correct, complete, and not potentially harmful to the patient. Nevertheless, instances of incorrect statements, missed key medical findings, and potentially harmful passages were reported. While further studies are needed, the initial insights of this study indicate a great potential in using large language models like ChatGPT to improve patient-centered care in radiology and other medical domains.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2212.14882

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ChatGPT offers numerous benefits in health care, including optimizing radiology reporting, generating patient discharge summaries, improving patient care, providing antimicrobial advice, and improving clinical decision support.

    “In health care, ChatGPT offers numerous benefits [17], including optimizing radiology reporting [18], generating patient discharge summaries [19], improving patient care [20], providing antimicrobial advice [21], and improving clinical decision support [22].”

    From Ethical Considerations of Using ChatGPT in Health Care (2023) · cites this paper as [18]

  • Cases of apparent error have been identified in the provision of discharge summaries and radiology reports by ChatGPT.

    “In particular, cases of apparent error have been identified in the provision of discharge summaries and radiology reports by ChatGPT [18,19].”

    From Ethical Considerations of Using ChatGPT in Health Care (2023) · cites this paper as [18]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [7][7] Teresa Martin-Carreras, Tessa S. Cook, and Charles E. Kahn. 'Readability of radiology reports: implications for patient-centered care'. eng. In: Clinical Imaging 54 (2019), pp. 116-120. -
  2. [8][8] Seong Cheol Oh, Tessa S Cook, and Charles E Kahn. 'PORTER: a prototype system for patient-oriented radiology reporting'. In: Journal of digital imaging 29.4 (2016), pp. 450-454. -
  3. [9][9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 'Attention is all you need'. In: Advances in neural information processing systems 30 (2017). -
  4. [11][11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding'. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT . 2019, pp. 4171-4186. -
  5. [13][13] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 'Exploring the limits of transfer learning with a unified text-to-text transformer.' In: J. Mach. Learn. Res. 21.140 (2020), pp. 1-67. -
  6. [19][19] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 'Language models are few-shot learners'. In: Advances in neural information processing systems 33 (2020), pp. 1877-1901. -
  7. [21][21] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 'Deep reinforcement learning from human preferences'. In: Advances in neural information processing systems 30 (2017). -
  8. [23][23] Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D. Manning, and Curtis P. Langlotz. 'Learning to Summarize Radiology Findings'. In: Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis, Louhi@EMNLP . Association for Computational Linguistics, 2018, pp. 204-213. -
  9. [24][24] Xiaoyan Cai, Sen Liu, Junwei Han, Libin Yang, Zhenguo Liu, and Tianming Liu. 'ChestXRayBERT: A Pretrained Language Model for Chest Radiology Report Summarization'. In: IEEE Transactions on Multimedia (2021). -
  10. [25][25] Siting Liang, Klaus Kades, Matthias Fink, Peter Full, Tim Weber, Jens Kleesiek, Michael Strube, and Klaus Maier-Hein. 'Fine-tuning BERT Models for Summarizing German Radiology Findings'. In: Proceedings of the 4th Clinical Natural Language Processing Workshop . 2022, pp. 30-40. -
  11. [26][26] Sean MacAvaney, Sajad Sotudeh, Arman Cohan, Nazli Goharian, Ish Talati, and Ross W Filice. 'Ontology-aware clinical abstractive summarization'. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval . 2019, pp. 1013-1016. -
  12. [27][27] Paul Gigioli, Nikhita Sagar, Anand Rao, and Joseph Voyles. 'Domain-aware abstractive text summarization for medical documents'. In: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . IEEE. 2018, pp. 2338-2343. -
  13. [29][29] Eva K Lee and Karan Uppal. 'CERC: an interactive content extraction, recognition, and construction tool for clinical and biomedical text'. In: BMC Medical Informatics and Decision Making 20.14 (2020), pp. 1-14. -
  14. [30][30] Andrea Chaves, Cyrille Kesiku, and Begonya Garcia-Zapirain. 'Automatic Text Summarization of Biomedical Text Data: A Systematic Review'. In: Information 13.8 (2022), p. 393. -
  15. [31][31] Matthew Shardlow. 'A survey of automated text simplification'. In: International Journal of Advanced Computer Science and Applications 4.1 (2014), pp. 58-70. -
  16. [32][32] Suha S Al-Thanyyan and Aqil M Azmi. 'Automated text simplification: a survey'. In: ACM Computing Surveys (CSUR) 54.2 (2021), pp. 1-36. -
  17. [33][33] Maria Kvist and Sumithra Velupillai. 'Professional language in swedish radiology reportscharacterization for patient-adapted text simplification'. In: Scandinavian Conference on Health Informatics . Linköping University Electronic Press. 2013, pp. 55-59. -
  18. [34][34] Emil Abrahamsson, Timothy Forni, Maria Skeppstedt, and Maria Kvist. 'Medical text simplification using synonym replacement: Adapting assessment of word difficulty to a compounding language'. In: Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR) . 2014, pp. 57-65. -
  19. [35][35] Lionel Ramadier and Mathieu Lafourcade. 'Radiological Text Simplification Using a General Knowledge Base'. In: International Conference on Computational Linguistics and Intelligent Text Processing . Springer. 2017, pp. 617-627. -
  20. [36][36] Basel Qenam, Tae Youn Kim, Mark J Carroll, Michael Hogarth, et al. 'Text simplification using consumer health vocabulary to generate patient-centered radiology reporting: translation and evaluation'. In: Journal of medical Internet research 19.12 (2017). -
  21. [37][37] Paul Hyunsoo Yi, Sean Kenney Golden, John B. Harringa, and Mark A. Kliewer. 'Readability of Lumbar Spine MRI Reports: Will Patients Understand?' In: American Journal of Roentgenology 212.3 (2019). Publisher: American Roentgen Ray Society, pp. 602-606. -
  22. [38][38] Spencer K. Barrett, James Patrie, Andrea B. Kitts, Michael Hanley, Christina M. Swanson, Hans Vitzthum von Eckstaedt, and Arun Krishnaraj. 'Patient-centered Reporting in Radiology: A Single-site Survey Study of Lung Cancer Screening Results'. In: Journal of Thoracic Imaging 36.6 (2021), p. 367. -
  23. [40][40] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 'On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?' In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency . 2021, pp. 610-623. -
  24. [41][41] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 'Extracting training data from large language models'. In: 30th USENIX Security Symposium (USENIX Security 21) . 2021, pp. 2633-2650. -
  25. [42][42] Stephanie Lin, Jacob Hilton, and Owain Evans. 'TruthfulQA: Measuring How Models Mimic Human Falsehoods'. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL . Association for Computational Linguistics, 2022, pp. 32143252. -
  26. [44][44] Chunting Zhou, Graham Neubig, Jiatao Gu, Mona T. Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 'Detecting Hallucinated Content in Conditional Neural Sequence Generation'. In: Findings of the Association for Computational Linguistics: ACL/IJCNLP . Association for Computational Linguistics, 2021, pp. 1393-1404. -
  27. [45][45] Tom L. Beauchamp and James F. Childress. Principles of biomedical ethics . Eighth edition. New York: Oxford University Press, 2019. -
  28. [46][46] Thilo Hagendorff. 'The Ethics of AI Ethics: An Evaluation of Guidelines'. In: Minds and Machines 30.1 (2020), pp. 99-120.

Cited by

Sorted by citations

Related

Authors
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B., Ricke, J., & Ingrisch, M.
Affiliations
LMU Klinikum (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2212.14882
arXiv
2212.14882
OpenAlex
W4313447794
Cited by
97
References
9
Citations by year
2025: 9 · 2024: 35 · 2023: 52
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Deutsche Forschungsgemeinschaft (460037581)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
ChatGPT, large language models, radiology reports, medical report summarization, radiologist evaluation, factual accuracy, patient-centered care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review