Evaluating the diagnostic performance of a large language model‐powered chatbot for providing immunohistochemistry recommendations in dermatopathology
Open accessAbstract
Background: Large language model (LLM)‐powered chatbots such as ChatGPT have numerous applications. However, their effectiveness in dermatopathology has not been formally evaluated. Dermatopathological cases often require immunohistochemical workup. Here, we evaluate the performance of a chatbot in providing diagnostically useful information on immunohistochemistry relating to dermatological diseases. Methods: We queried a commonly used chatbot for the immunophenotypes of 51 cutaneous diseases, including a diverse variety of epidermal, adnexal, hematolymphoid, and soft tissue entities. We requested it to provide references for each diagnosis. All tests were repeated, compiled, quantified, and then compared with established literature standards. Results: Clustering analysis demonstrated that recommendations correlated with tumor type, suggesting chatbots can supply appropriate panels. However, a significant portion of recommendations were factually incorrect (13.9%). Citations were rarely clinically useful (24.5%). Many were confabulated (27.2%). Prompt responses for cutaneous adnexal lesions tended to be less accurate while literature references were less useful. Reference retrieval performance was associated with the number of PubMed entries per entity. Conclusions: This foundational study suggests that LLM‐powered chatbots may be useful for generating immunohistochemical panels for dermatologic diagnoses. However, specific performance capabilities and biases must be considered. In addition, extreme caution is advised regarding the tendencies to fabricate material. Future models intentionally fine‐tuned to augment diagnostic medicine may prove to be valuable.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models (LLMs) can be integrated to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks.
“Moreover, large language models (LLMs) can be integrated into this principle to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks [23, 48–52].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1× · see citing works
“18 These fabricated outputs have been termed “ hallucinations ” and raise concerns over the general usability and safety of outputs in settings where specific factual details are necessary.”
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · see citing works
Abstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611 · see citing works
Performance of ChatGPT on Specialty Certificate Examination in Dermatology multiple-choice questions
Passby, L., Jenko, N., & Wernham, A. G. H. (2023) · Clinical and Experimental Dermatology · doi:10.1093/ced/llad197 · Cited by 98 · see citing works
Potential applications of ChatGPT in dermatology
Kluger, N. (2023) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.19152 · Cited by 34 · see citing works
Chat GPT in dermatology: Progressive or problematic?
Porter, E., Murphy, M., & O’Connor, C. (2023) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.19174 · Cited by 24 · see citing works
The complex ethics of applying ChatGPT and language model artificial intelligence in dermatology
Ferreira, A. L., & Lipoff, J. B. (2023) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2023.05.054 · Cited by 18 · see citing works
Artificial intelligence in Dermatopathology
Jartarkar, S. R., Cockerell, C. J., Patil, A., Kassir, M., Babaei, M., Weidenthaler‐Barth, B., Grabbe, S., & Goldust, M. (2023) · Journal of Cosmetic Dermatology · doi:10.1111/jocd.15565 · Cited by 17 · see citing works
Immunohistochemistry in Dermatopathology
Fuertes, L., Santonja, C., Kutzner, H., et al. (2016) · American Journal of Dermatopathology · doi:10.1097/dad.0000000000000361 · Cited by 15 · see citing works
Utilization and Utility of Immunohistochemistry in Dermatopathology
Naert, K. A., & Trotter, M. J. (2012) · American Journal of Dermatopathology · doi:10.1097/dad.0b013e31825d4f73 · Cited by 11 · see citing works
The introduction of “Dr AI”: What dermatologists should consider
Dave, P., Nambudiri, V. E., & Grant-Kels, J. M. (2022) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2022.01.014 · Cited by 9 · see citing works
Agarwal, A., & Stoff, B. K. (2023) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2023.04.030 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chen, S., Yu, J., Chamouni, S., Wang, Y., & Li, Y. (2024) · BMC Medicine · doi:10.1186/s12916-024-03566-x · Cited by 73 · see citing works
- Authors
- McCrary, M. R., Galambus, J., & Chen, W.
- Affiliations
- University of South Florida (US)
- Department of Anatomic and Clinical Pathology Morsani College of Medicine, University of South Florida Tampa Florida USA
- Venue
- Journal of Cutaneous Pathology
- Publisher
- Wiley
- Published
- 2024-05-14
- Volume
- 51
- Issue
- 9
- Pages
- 689-695
- ISSN
- 0303-6987 (ISSN-L)
- 1600-0560
- OpenAlex
- W4396936548
- Cited by
- 12
- References
- 15
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 5
- FWCI
- 1.12
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Cutaneous Pathology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, chatbots, ChatGPT, dermatopathology, skin diseases, immunophenotype, diagnostic accuracy, hallucinations
- MeSH terms
- Dermatology, Humans, Immunohistochemistry, Skin Diseases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
