GPT-4 in a Cancer Center — Institute-Wide Deployment Challenges and Lessons Learned
Open accessThe DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Generative AI has both significant untapped potential and unique risks for healthcare. Here we report on the implementation challenges, technology solutions, and lessons learned in broad deployment of large language models to the workforce of Dana-Farber Cancer Institute. In order that other organizations might benefit from our experience, we detail the aspects of sponsorship, governance, technical implementation and configurations, program launch, socialization, and ongoing support that went into making generative AI available to our entire workforce in a compliant, auditable, and secure manner. ----- The enormous potential for generative pre-trained transformers and other artificial intelligence large language models to improve healthcare has become increasingly clear since the release of ChatGPT in November 2022. Software tools based on large language models have been shown to perform as well as or better than humans on many healthcare-related tasks, including generation of clinical documentation, extraction of structured data from medical records, performance on a growing number of medical board exams, and writing accurate and empathetic responses to patients’ medical questions. However, healthcare settings pose unique ethical, legal, regulatory, and technical challenges for large-scale deployment and adoption of large language models, including the essentiality of patient data privacy and security, the direct negative consequences of errors and biases, the need for model interpretability and supporting evidence, the need to safeguard intellectual property and proprietary data, and the difficulty of modifying clinical and operational workflows. Because of these risks and challenges, few large language models are in use in hospitals outside of controlled research studies or small pilots, and none to our knowledge are yet broadly deployed in a dedicated cancer center. In this case study, we report on the implementation challenges and lessons learned in the evaluation and deployment of large language models at Dana-Farber Cancer Institute for use by all business areas including research and clinical operations. In early discussions about whether and how to proceed, it became clear that while some inherent risks could be mitigated by clear policy guardrails and a secure technical environment, significant risks would remain, including compliance with rapidly evolving regulations (e.g., for academic use of LLMs). We also recognized the substantial, ongoing work that would be required for appropriate ethical consideration of each use case, and to ensure patient- and human-centric decision-making. After robust discussion over many months, we believed it would be better for our institution to tackle these questions and challenges together as a community, with broad participation and discussion, in an audited, continuous-learning environment, than to prohibit all use of LLMs. Here we detail aspects of sponsorship, governance, technical implementation, program launch, socialization and ongoing support that went into making generative AI broadly available to our 12,500-member workforce in a compliant, auditable, and secure manner. We also share our source code and infrastructure-as-code to benefit others who may follow the same path. We hope other institutions can benefit from our experience as they consider deployment of generative AI that furthers their medical and research missions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
ChatGPT for Language Teaching and Learning
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · RELC Journal · doi:10.1177/00336882231162868 · Cited by 1,371
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
Cited by
Dennstädt, F., Hastings, J., Putora, P. M., Schmerder, M., & Cihoric, N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01476-7 · Cited by 144
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81
ESMO guidance on the use of Large Language Models in Clinical Practice (ELCAP)
Wong, E. Y. T., Verlingue, L., Aldea, M., Franzoi, M. A., Umeton, R., Halabi, S., Harbeck, N., Indini, A., Prelaj, A., Romano, E., et al. (2025) · Annals of Oncology · doi:10.1016/j.annonc.2025.09.001 · Cited by 44
Self-citation
Metadata
- Authors
- Umeton, R., Kwok, A., Maurya, R., Leco, D., Lenane, N., Willcox, J., Abel, G. A., Tolikas, M., & Johnson, J. M.
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2024-03-28
- Volume
- 1
- Issue
- 4
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4393259351
- Cited by
- 36
- References
- 34
- FWCI
- 2.89
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: GPT-4 in a Cancer Center — Institute-Wide Deployment Challenges and Lessons Learned. https://app.cochat.ai/papers/10.1056/aics2300191
- The paper
- Umeton, R., Kwok, A., Maurya, R., Leco, D., Lenane, N., Willcox, J., Abel, G. A., Tolikas, M., & Johnson, J. M. (2024). GPT-4 in a Cancer Center — Institute-Wide Deployment Challenges and Lessons Learned. NEJM AI. https://doi.org/10.1056/aics2300191
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.