CoChat
Sign Up
Cited by 46Artificial Intelligence in Healthcare and EducationThyroid Cancer Diagnosis and TreatmentAI in cancer detection

Assessing the role of GPT-4 in thyroid ultrasound diagnosis and treatment recommendations: enhancing interpretability with a chain of thought approach

Open access
Wang, Zhixiang, Zhang, Zhen, Traverso, Alberto, Dekker, Andre, Qian, Linxue, Sun, Pengfei·2024·Quantitative Imaging in Medicine and Surgery
DOI 10.21037/qims-23-1180PMC PMC10895085
Open this paper in CoChat — free account

Abstract

Background: As artificial intelligence (AI) becomes increasingly prevalent in the medical field, the effectiveness of AI-generated medical reports in disease diagnosis remains to be evaluated. ChatGPT is a large language model developed by open AI with a notable capacity for text abstraction and comprehension. This study aimed to explore the capabilities, limitations, and potential of Generative Pre-trained Transformer (GPT)-4 in analyzing thyroid cancer ultrasound reports, providing diagnoses, and recommending treatment plans. Methods: Using 109 diverse thyroid cancer cases, we evaluated GPT-4's performance by comparing its generated reports to those from doctors with various levels of experience. We also conducted a Turing Test and a consistency analysis. To enhance the interpretability of the model, we applied the Chain of Thought (CoT) method to deconstruct the decision-making chain of the GPT model. Results: GPT-4 demonstrated proficiency in report structuring, professional terminology, and clarity of expression, but showed limitations in diagnostic accuracy. In addition, our consistency analysis highlighted certain discrepancies in the AI's performance. The CoT method effectively enhanced the interpretability of the AI's decision-making process. Conclusions: GPT-4 exhibits potential as a supplementary tool in healthcare, especially for generating thyroid gland diagnostic reports. Our proposed online platform, "ThyroAIGuide", alongside the CoT method, underscores the potential of AI to augment diagnostic processes, elevate healthcare accessibility, and advance patient education. However, the journey towards fully integrating AI into healthcare is ongoing, requiring continuous research, development, and careful monitoring by medical professionals to ensure patient safety and quality of care.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21037/qims-23-1180

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The majority of reviewed studies leverage the inherent text generation capabilities of LLMs to produce explanations as a natural by-product of inference.

    “the majority of reviewed studies leverage the inherent text generation capabilities of LLMs to produce explanations as a natural by-product of inference [62,71,72,78,97,112].”

    From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [97]

    The Methods section (“GPT Chain of Thought (CoT) visualization”; page unavailable) describes GPT-4 illustrating its reasoning during thyroid diagnosis, so this paper is an example of an LLM producing an explanation. It does not establish what the majority of reviewed studies do, and its explanation is elicited through a deliberate chain-of-thought method rather than described as a natural by-product of inference. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [1]Durante C Grani G Lamartina L Filetti S Mandel SJ Cooper DS. The Diagnosis and Management of Thyroid Nodules: A Review. JAMA 2018; 319: 914- 24. 10.1001/jama.2018.0898 29509871 · DOI
  2. [2]Guth S Theune U Aberle J Galach A Bamberger CM. Very high prevalence of thyroid nodules detected by high frequency (13 MHz) ultrasound examination. Eur J Clin Invest 2009; 39: 699- 706. 10.1111/j.1365-2362.2009.02162.x 19601965 · DOI
  3. [3]Haugen BR Alexander EK Bible KC Doherty GM Mandel SJ Nikiforov YE Pacini F Randolph GW Sawka AM Schlumberger M Schuff KG Sherman SI Sosa JA Steward DL Tuttle RM Wartofsky L. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 2016; 26: 1- 133. 10.1089/thy.2015.0020 26462967 PMC4739132 · DOI · Paper page
  4. [4]Moon WJ Baek JH Jung SL Kim DW Kim EK Kim JY Kwak JY Lee JH Lee JH Lee YH Na DG Park JS Park SW Korean Society of Thyroid Radiology (KSThR); Korean Society of Radiology. Ultrasonography and the ultrasound-based management of thyroid nodules: consensus statement and recommendations. Korean J Radiol 2011; 12: 1- 14. 10.3348/kjr.2011.12.1.1 21228935 PMC3017873 · DOI
  5. [6]Silver D Huang A Maddison CJ Guez A Sifre L van den Driessche G Schrittwieser J Antonoglou I Panneershelvam V Lanctot M Dieleman S Grewe D Nham J Kalchbrenner N Sutskever I Lillicrap T Leach M Kavukcuoglu K Graepel T Hassabis D. Mastering the game of Go with deep neural networks and tree search. Nature 2016; 529: 484- 9. 10.1038/nature16961 26819042 · DOI · Paper page
  6. [7]Brown T Mann B Ryder N Subbiah M Kaplan JD Dhariwal P Neelakantan A Shyam P Sastry G Askell A Agarwal S. Language models are few-shot learners. Advances in Neural Information Processing Systems 2020; 33: 1877- 901.
  7. [8]Ferrucci D Brown E Chu-Carroll J Fan J Gondek D Kalyanpur AA Lally A Murdock JW Nyberg E Prager J Schlaefer N Welty C. Building Watson: An overview of the DeepQA project. AI Magazine 2010; 31: 59- 79.
  8. [9]Schrittwieser J Antonoglou I Hubert T Simonyan K Sifre L Schmitt S Guez A Lockhart E Hassabis D Graepel T Lillicrap T Silver D. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 2020; 588: 604- 9. 10.1038/s41586-020-03051-4 33361790 · DOI · Paper page
  9. [10]Rajkomar A Dean J Kohane I. Machine Learning in Medicine. N Engl J Med 2019; 380: 1347- 58. 10.1056/NEJMra1814259 30943338 · DOI · Paper page
  10. [11]Koubaa A. GPT-4 vs. GPT-3.5: A Concise Showdown. Preprints 2023, 2023030422. Available online: https://doi.org/ 10.20944/preprints202303.0422.v1 · DOI · Paper page
  11. [12]Shickel B Tighe PJ Bihorac A Rashidi P. Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis. IEEE J Biomed Health Inform 2018; 22: 1589- 604. 10.1109/JBHI.2017.2767063 29989977 PMC6043423 · DOI · Paper page
  12. [14]Jiang F Jiang Y Zhi H Dong Y Li H Ma S Wang Y Dong Q Shen H Wang Y. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol 2017; 2: 230- 43. 10.1136/svn-2017-000101 29507784 PMC5829945 · DOI · Paper page
  13. [15]Grinberg M. Flask web development: developing web applications with python[M]. O'Reilly Media, Inc.; 2018.
  14. [17]Murdoch TB Detsky AS. The inevitable application of big data to health care. JAMA 2013; 309: 1351- 2. 10.1001/jama.2013.393 23549579 · DOI
  15. [18]He W Zha S Li L. Social media competitive analysis and text mining: A case study in the pizza industry. Int J Inf Manage 2013; 33: 464- 72.
  16. [19]Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi E, Le Q, Zhou D. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Part of Advances in Neural Information Processing Systems 35 (NeurIPS 2022); 2022.
  17. [20]McKnight PE, Najab JJTCeop. Mann‐Whitney U Test. 2010:1-1. Available online: https://doi.org/ 10.4135/9781412961288.n228 · DOI
  18. [21]Virtanen P Gommers R Oliphant TE Haberland M Reddy T Cournapeau D SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods 2020; 17: 261- 72. 10.1038/s41592-019-0686-2 32015543 PMC7056644 · DOI · Paper page
  19. [22]Gebrael G, Sahu KK, Chigarira B, Tripathi N, Mathew Thomas V, Sayegh N, Maughan BL, Agarwal N, Swami U, Li H. Enhancing Triage Efficiency and Accuracy in Emergency Rooms for Patients with Metastatic Prostate Cancer: A Retrospective Analysis of Artificial Intelligence-Assisted Triage Using ChatGPT 4.0. Cancers (Basel) 2023. PMC10378202 37509379
  20. [23]Secinaro S Calandra D Secinaro A Muthurangu V Biancone P. The role of artificial intelligence in healthcare: a structured literature review. BMC Med Inform Decis Mak 2021; 21: 125. 10.1186/s12911-021-01488-9 33836752 PMC8035061 · DOI
  21. [24]Petch J Di S Nelson W. Opening the Black Box: The Promise and Limitations of Explainable Machine Learning in Cardiology. Can J Cardiol 2022; 38: 204- 13. 10.1016/j.cjca.2021.09.004 34534619 · DOI · Paper page
  22. [25]Coskun B Ocakoglu G Yetemen M Kaygisiz O. Can ChatGPT, an Artificial Intelligence Language Model, Provide Accurate and High-quality Patient Information on Prostate Cancer? Urology 2023; 180: 35- 58. 10.1016/j.urology.2023.05.040 37406864 · DOI
  23. [26]Vorisek CN Lehne M Klopfenstein SAI Mayer PJ Bartschke A Haese T Thun S. Fast Healthcare Interoperability Resources (FHIR) for Interoperability in Health Research: Systematic Review. JMIR Med Inform 2022; 10: e35724. 10.2196/35724 35852842 PMC9346559 · DOI
  24. [27]Myong Y Yoon D Kim BS Kim YG Sim Y Lee S Yoon J Cho M Kim S. Evaluating diagnostic content of AI-generated chest radiography: A multi-center visual Turing test. PLoS One 2023; 18: e0279349. 10.1371/journal.pone.0279349 37043456 PMC10096231 · DOI
  25. [28]Sohn E. The reproducibility issues that haunt health-care AI. Nature 2023; 613: 402- 3. 10.1038/d41586-023-00023-2 36624237 · DOI
  26. [29]Liu S Wright AP Patterson BL Wanderer JP Turer RW Nelson SD McCoy AB Sittig DF Wright A. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc 2023; 30: 1237- 45. 10.1093/jamia/ocad072 37087108 PMC10280357 · DOI · Paper page
  27. [30]Joyce DW Kormilitzin A Smith KA Cipriani A. Explainable artificial intelligence for mental health through transparency and interpretability for understandability. NPJ Digit Med 2023; 6: 6. 10.1038/s41746-023-00751-9 36653524 PMC9849399 · DOI
  28. [31]Frasca F Piticchio T Le Moli R Tumino D Cannavò S Ruggeri RM Campennì A Giovanella L. Early detection of suspicious lymph nodes in differentiated thyroid cancer. Expert Rev Endocrinol Metab 2022; 17: 447- 54. 10.1080/17446651.2022.2112176 35993330 · DOI

Cited by

Sorted by citations

Authors
Wang, Z., Zhang, Z., Traverso, A., Dekker, A., Qian, L., & Sun, P.
Affiliations
  • Capital Medical University (CN)
  • Maastricht University Medical Centre (NL)
Corresponding author
  • Qian, Linxue
  • Sun, Pengfei
ORCID
Venue
Quantitative Imaging in Medicine and Surgery
Publisher
AME Publishing Company
Published
2024-02
Volume
14
Issue
2
Pages
1602-1615
ISSN
  • 2223-4292 (ISSN-L)
  • 2223-4306
DOI
10.21037/qims-23-1180
PMC
PMC10895085
OpenAlex
W4391258615
Cited by
46
References
3
Citations by year
2026: 15 · 2025: 23 · 2024: 8
FWCI
4.30
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Quantitative Imaging in Medicine and Surgery (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by-nc-nd)
  • Archivio Istituzionale della Ricerca - Università Vita-Salute San Raffaele (submittedVersion, cc-by-nc-nd)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Endocrinology, Diabetes and Metabolism › Thyroid Cancer Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
Keywords
GPT-4, thyroid cancer, thyroid ultrasound, chain-of-thought, interpretability, large language models, treatment guidelines, diagnostic accuracy, Turing test, consistency analysis, patient education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review