Large Language Models in Hematology Case Solving: A Comparative Study of ChatGPT-3.5, Google Bard, and Microsoft Bing
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Background Large language models (LLMs), such as ChatGPT-3.5, Google Bard, and Microsoft Bing, have shown promising capabilities in various natural language processing (NLP) tasks. However, their performance and accuracy in solving domain-specific questions, particularly in the field of hematology, have not been extensively investigated. Objective This study aimed to explore the capability of LLMs, namely, ChatGPT-3.5, Google Bard, and Microsoft Bing (Precise), in solving hematology-related cases and comparing their performance. Methods This was a cross-sectional study conducted in the Department of Physiology and Pathology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India. We curated a set of 50 cases on hematology covering a range of topics and complexities. The dataset included queries related to blood disorders, hematologic malignancies, laboratory test parameters, calculations, and treatment options. Each case and related question was prepared with a set of correct answers to compare with. We utilized ChatGPT-3.5, Google Bard Experiment, and Microsoft Bing (Precise) for question-answering tasks. The answers were checked by two physiologists and one pathologist. They rated the answers on a rating scale from one to five. The average score of the three models was compared by Friedman's test with Dunn's post-hoc test. The performance of the LLMs was compared with a median of 2.5 by a one-sample median test as the curriculum from which the questions were curated has a 50% pass grade. Results The scores among the three LLMs were significantly different (p-value < 0.0001) with the highest score by ChatGPT (3.15±1.19), followed by Bard (2.23±1.17) and Bing (1.98±1.01). The score of ChatGPT was significantly higher than 50% (p-value = 0.0004), Bard's score was close to 50% (p-value = 0.38), and Bing's score was significantly lower than the pass score (p-value = 0.0015). Conclusion The LLMs reveal significant differences in solving case vignettes in hematology. ChatGPT exhibited the highest score, followed by Google Bard and Microsoft Bing. The observed performance trends suggest that ChatGPT holds promising potential in the medical domain. However, none of the models was capable of answering all questions accurately. Further research and optimization of language models can offer valuable contributions to healthcare and medical education applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“These models have demonstrated various levels of accuracy in performing medical examinations, solving complex medical issues, or interpreting radiology reports [4,5,6,7,8].”
BackgroundKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Dave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204
Self-citationBhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460
Shahsavar, Y., & Choudhury, A. (2023) · JMIR Human Factors · doi:10.2196/47564 · Cited by 341 · cited 1×
“Some of the potential use of LLMs are summarized in Table 2 [12,13,14,15,16].”
BackgroundUsing ChatGPT in Medical Research: Current Status and Future Directions
Ruksakulpiwat, S., Kumar, A., & Ajibade, A. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s413470 · Cited by 156 · cited 1×
“Some of the potential use of LLMs are summarized in Table 2 [12,13,14,15,16].”
BackgroundChatGPT in medical school: how successful is AI in progress testing?
Friederichs, H., Friederichs, W. J., & März, M. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2220920 · Cited by 122 · cited 1×
“Hence, further studies are required in the future to fully explore their capability [11].”
BackgroundThe Pros and Cons of Using ChatGPT in Medical Education: A Scoping Review
Mohammad, B., Supti, T., Alzubaidi, M., Shah, H., Alam, T., Shah, Z., & Househ, M. (2023) · Studies in Health Technology and Informatics · doi:10.3233/shti230580 · Cited by 114 · cited 1×
“Some of the potential use of LLMs are summarized in Table 2 [12,13,14,15,16].”
BackgroundMondal, H., Mondal, S., & Podder, I. (2023) · Indian Dermatology Online Journal · doi:10.4103/idoj.idoj_72_23 · Cited by 99 · cited 1×
“Some of the potential use of LLMs are summarized in Table 2 [12,13,14,15,16].”
BackgroundSelf-citationAgarwal, M., Sharma, P., & Goswami, A. (2023) · Cureus · doi:10.7759/cureus.40977 · Cited by 72 · cited 1×
“Prominent among these LLMs are ChatGPT, Google Bard, and Microsoft Bing, which have garnered substantial interest due to their capacity to comprehend textual information and generate contextually relevant responses [3].”
BackgroundGhosh, A., & Bir, A. (2023) · Cureus · doi:10.7759/cureus.37023 · Cited by 67
Dhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.42972 · Cited by 59 · cited 1×
“These models have demonstrated various levels of accuracy in performing medical examinations, solving complex medical issues, or interpreting radiology reports [4,5,6,7,8].”
BackgroundSelf-citation
Cited by
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05630-9 · Cited by 103
“Our findings support Torres-Zegarra et al.’s observations that the previous version of ChatGPT-4 and Microsoft Bing were superior to Google Bard [33], while other research groups did not confirm it [30–32].”
BackgroundResultChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe14030043 · Cited by 69
“Another study comparing ChatGPT 3.5, Google Bard, and Microsoft Bing in hematology-related cases noted distinct performance disparities, with ChatGPT achieving the highest accuracy [11].”
InfluentialAnalyzing evaluation methods for large language models in the medical field: a scoping review
Lee, J., Park, S., Shin, J., & Cho, B. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02709-7 · Cited by 52
Comparative analysis of ChatGPT and Gemini (Bard) in medical inquiry: a scoping review
Fattah, F. H., Salih, A. M., Salih, A. M., Asaad, S. K., Ghafour, A. K., Bapir, R., Abdalla, B. A., Othman, S., Ahmed, S. M., Hasan, S. J., et al. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1482712 · Cited by 50
Ilgaz, H. B., & Çelik, Z. (2023) · Cureus · doi:10.7759/cureus.45301 · Cited by 49
“These models have been helpful in solving complex medical problems, interpreting radiology reports, being used to diagnose diseases, writing scientific articles or answering and generating different medical exam questions and have shown varying degrees of accuracy in these fields [2-5, 7,8 ,”
Metadata
- Authors
- Kumari, A., Kumari, A., Singh, A., Singh, S. K., Juhi, A., Dhanvijay, A. K. D., Pinjar, M. J., & Mondal, H.
- Venue
- Cureus
- Published
- 2023-08-21
- Volume
- 15
- Issue
- 8
- Pages
- e43861-e43861
- ISSN
- 2168-8184 (ISSN-L)
- OpenAlex
- W4386033569
- Cited by
- 62
- References
- 13
- FWCI
- 2.15
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.