Large language model use in clinical oncology
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) are undergoing intensive research for various healthcare domains. This systematic review and meta-analysis assesses current applications, methodologies, and the performance of LLMs in clinical oncology. A mixed-methods approach was used to extract, summarize, and compare methodological approaches and outcomes. This review includes 34 studies. LLMs are primarily evaluated on their ability to answer oncologic questions across various domains. The meta-analysis highlights a significant performance variance, influenced by diverse methodologies and evaluation criteria. Furthermore, differences in inherent model capabilities, prompting strategies, and oncological subdomains contribute to heterogeneity. The lack of use of standardized and LLM-specific reporting protocols leads to methodological disparities, which must be addressed to ensure comparability in LLM research and ultimately leverage the reliable integration of LLM technologies into clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · cited 2×
“Weaddressedthisgap withthedevelopmentofourevaluationframework,whichisbasedonthe principlesofQUADAS-2 11 .”
BackgroundFlesch, R. (1948) · Journal of Applied Psychology · doi:10.1037/h0057532 · Cited by 5,391
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · cited 3×
“This has the potential to provide accessible information to patients and to facilitate seamless communication between clinicians, thereby enhancing patient empowerment and satisfaction 7 .”
BackgroundSelf-citationYeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615
Leveraging large language models for decision support in personalized oncology
Benary, M., Wang, X. D., Schmidt, M., Soll, D., Hilfenhaus, G., Nassir, M., Sigler, C., Knödler, M., Keller, U., Beule, D., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.43689 · Cited by 306
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289 · cited 5×
“Sources encompassed of fi cial information forums (3/34) 14,16,18 , guidelines (7/34) 15,17,28,29,33,35,45 , frequently asked questions (FAQs) from various sources, social media, hospital websites and Google Trends © (8/34) 19,30,31,36,37,39,40,42 , clinical cases (8/34) 20 – 22, 24 – 27 ,”
BackgroundMethodInfluentialUse of Artificial Intelligence Chatbots for Cancer Treatment Information
Chen, S., Kann, B. H., Foote, M. B., Aerts, H. J. W. L., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2954 · Cited by 251 · cited 6×
“Anevaluationframeworkconsistingof18Itemswascreated(Table1).”
BackgroundMethodInfluentialESMO guidance for reporting oncology real-world evidence (GROW)
Castelo-Branco, L., Pellat, A., Martins-Branco, D., Valachis, A., Derksen, J. W. G., Suijkerbuijk, K. P. M., Dafni, U., Dellaporta, T., Vogel, A., Prelaj, A., et al. (2023) · Annals of Oncology · doi:10.1016/j.annonc.2023.10.001 · Cited by 247
Large language model AI chatbots require approval as medical devices
Gilbert, S., Harvey, H., Melvin, T., Vollebregt, E., & Wicks, P. (2023) · Nature Medicine · doi:10.1038/s41591-023-02412-6 · Cited by 207
Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
Holmes, J., Liu, Z., Zhang, L., Ding, Y., Sio, T. T., McGee, L. A., Ashman, J. B., Li, X., Liu, T., Shen, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01938 · Cited by 190
Lukac, S., Dayan, D., Fink, V., Leinert, E., Hartkopf, A., Veselinovic, K., Janni, W., Rack, B., Pfister, K., Heitmeir, B., et al. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07130-5 · Cited by 130 · cited 3×
“Questioning procedure continued “ Prompt Engineering ” PromptEngineering(PE)wasreportedin9outof34studies 15,17,20,21,23,27,33,34,41 .”
BackgroundMethodEppler, M., Ganjavi, C., Ramacciotti, L. S., Piazza, P., Rodler, S., Checcucci, E., Gomez Rivas, J., Kowalewski, K. F., Belenchón, I. R., Puliatti, S., et al. (2024) · European Urology · doi:10.1016/j.eururo.2023.10.014 · Cited by 117 · cited 1×
“TrustintheaccuracyofLLM-generatedinformationissplit,with 29.6%ofparticipantsexpressingbothtrustandscepticism 65 .”
InfluentialApplications of large language models in cancer care: current evidence and future perspectives
Iannantuono, G. M., Bracken-Clarke, D., Floudas, C. S., Roselli, M., Gulley, J. L., & Karzai, F. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1268915 · Cited by 115
Derraz, B., Breda, G., Kaempf, C., Baenke, F., Cotte, F., Reiche, K., Köhl, U., Kather, J. N., Eskenazy, D., & Gilbert, S. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00517-w · Cited by 94
Self-citationGebrael, G., Sahu, K. K., Chigarira, B., Tripathi, N., Mathew Thomas, V., Sayegh, N., Maughan, B. L., Agarwal, N., Swami, U., & Li, H. (2023) · Cancers · doi:10.3390/cancers15143717 · Cited by 85 · cited 2×
“…explored the prompting of medical questions and analysing the output of the LLMs ( medical knowledge , 32/34) 14 – 45 , while a (14/34) 16,18,19,22,28,29,31,32,34,36,37,39,45,46 as well as treatment recommendations (30/34) 14 Testing the encoded medical knowledge of LLMs This section…”
BackgroundLarge language models and multimodal foundation models for precision oncology
Truhn, D., Eckardt, J.-N., Ferber, D., & Kather, J. N. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00573-2 · Cited by 79
Self-citationRodler, S., Kopliku, R., Ulrich, D., Kaltenhauser, A., Casuscelli, J., Eismann, L., Waidelich, R., Buchner, A., Butz, A., Cacciamani, G. E., et al. (2024) · European Urology Focus · doi:10.1016/j.euf.2023.10.020 · Cited by 65
Evaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms
Nguyen, D., Swanson, D., Newbury, A., & Kim, Y. H. (2024) · Academic Radiology · doi:10.1016/j.acra.2023.11.002 · Cited by 38 · cited 4×
“Questioning procedure continued “ Prompt Engineering ” PromptEngineering(PE)wasreportedin9outof34studies 15,17,20,21,23,27,33,34,41 .”
BackgroundMethodInfluential
Cited by
Large language model integrations in cancer decision-making: a systematic review and meta-analysis
Hao, Y., Qiu, Z., Holmes, J., Löckenhoff, C. E., Liu, W., Ghassemi, M., & Kalantari, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01824-7 · Cited by 67
ESMO guidance on the use of Large Language Models in Clinical Practice (ELCAP)
Wong, E. Y. T., Verlingue, L., Aldea, M., Franzoi, M. A., Umeton, R., Halabi, S., Harbeck, N., Indini, A., Prelaj, A., Romano, E., et al. (2025) · Annals of Oncology · doi:10.1016/j.annonc.2025.09.001 · Cited by 44
Self-citationSezgin, E., Jackson, D. I., Kocaballi, A. B., Bibart, M., Zupanec, S., Landier, W., Audino, A., Ranalli, M., & Skeens, M. (2025) · Cancer Medicine · doi:10.1002/cam4.70554 · Cited by 30
Saady, M., Eissa, M., Yacoub, A. S., Hamed, A. B., & Azzazy, H. M. E.-S. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103066 · Cited by 15
Hack, S., Karni, R. J., Maniaci, A., Fundakowski, C. E., Castellani, L., Incandela, F., Accorona, R., Mayo-Yanez, M., Violati, M., Giannini, L., et al. (2026) · Oral Oncology · doi:10.1016/j.oraloncology.2026.107877 · Cited by 10
Metadata
- Authors
- Carl, N., Schramm, F., Haggenmüller, S., Kather, J. N., Hetz, M. J., Wies, C., Michel, M. S., Wessels, F., & Brinker, T. J.
- Venue
- npj Precision Oncology
- Published
- 2024-10-23
- Volume
- 8
- Issue
- 1
- Pages
- 240-240
- ISSN
- 2397-768X (ISSN-L)
- OpenAlex
- W4403680361
- Cited by
- 80
- References
- 23
- FWCI
- 6.41
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.