Generalization—a key challenge for responsible AI in patient-facing clinical applications
Open accessAbstract
Generalization – the ability of AI systems to apply and/or extrapolate their knowledge to new data which might differ from the original training data – is a major challenge for the effective and responsible implementation of human-centric AI applications. Current debate in bioethics proposes selective prediction as a solution. Here we explore data-based reasons for generalization challenges and look at how selective predictions might be implemented technically, focusing on clinical AI applications in real-world healthcare settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities.
“The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities19,54–57.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [55]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,525 · see citing works
Learning From Noisy Labels With Deep Neural Networks: A Survey
Song, H., Kim, M., Park, D., Shin, Y., & Lee, J.-G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3152527 · Cited by 1,407 · see citing works
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853 · see citing works
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D. E., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 582 · see citing works
Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494 · see citing works
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467 · see citing works
Underspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398 · see citing works
Ding, Y., Hou, K., Burch, K. S., Lapinska, S., Privé, F., Vilhjálmsson, B. J., Sankararaman, S., & Paşaniuc, B. (2021) · Nature Genetics · doi:10.1038/s41588-021-00961-5 · Cited by 136 · see citing works
Yalaza, M., İnan, A., & Bozer, M. (2016) · Meme sağlığı dergisi/Meme sağlığı dergisi · doi:10.5152/tjbh.2015.2711 · Cited by 132 · see citing works
Salehi, M., Mirzaei, H., Hendrycks, D., Li, Y., Rohban, M. H., & Sabokrou, M. (2021) · arXiv · doi:10.48550/arxiv.2110.14051 · Cited by 118 · see citing works
Tang, S., Ghorbani, A., Yamashita, R., Rehman, S., Dunnmon, J., Zou, J., & Rubin, D. L. (2021) · Scientific Reports · doi:10.1038/s41598-021-87762-2 · Cited by 88 · see citing works
Yoder, R., Kimler, B. F., Staley, J. M., Schwensen, K., Wang, Y. Y., Finke, K., O’Dea, A. P., Nye, L. E., Elia, M., Crane, G. J., et al. (2022) · npj Breast Cancer · doi:10.1038/s41523-022-00448-4 · Cited by 80 · see citing works
Guha Roy, A., Ren, J., Azizi, S., Loh, A., Natarajan, V., Mustafa, B., Pawlowski, N., Freyberg, J., Liu, Y., Beaver, Z., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102274 · Cited by 77 · see citing works
Machine learning to guide the use of adjuvant therapies for breast cancer
Alaa, A. M., Gurdasani, D., Harris, A. L., Rashbass, J., & van der Schaar, M. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00353-8 · Cited by 55 · see citing works
Self-citationPlex: Towards Reliability using Pretrained Large Model ExtensionsPreprint
Tran, D., Liu, J., Dusenberry, M. W., Phan, D., Collier, M. A., Ren, J., Han, K., Wang, Z., Mariet, Z., Hu, H., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.07411 · Cited by 39 · see citing works
Seedat, N., Imrie, F., & van der Schaar, M. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3345805 · Cited by 36 · see citing works
Self-citationThe selective deployment of AI in healthcare
Vandersluis, R., & Savulescu, J. (2024) · Bioethics · doi:10.1111/bioe.13281 · Cited by 25 · see citing works
Self-citationOptimal delivery of male breast cancer follow-up care: improving outcomes
Ruddy, K. J., & Ferzoco, R. M. (2015) · Breast Cancer Targets and Therapy · doi:10.2147/bctt.s75630 · Cited by 18 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [3]Gulrajani, I. & Lopez-Paz, D. In Search of Lost Domain Generalization. In International Conference on Learning Representations (2020).
- [13]Jaeger, P. F., Lüth, C. T., Klein, L. & Bungert, T. J. A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification. In The Eleventh International Conference on Learning Representations (2022).
- [20]Vovk, V., Gammerman, A. & Shafer, G. Algorithmic learning in a random world Vol. 29 (Springer, New York, 2005).
- [23]Van Breugel B Kyono T Berrevoets J Van der Schaar M Decaf: Generating fair synthetic data using causally-aware generative networks Adv. Neural Inf. Process. Syst. 2021 34 22221 22233
Cited by
Sorted by citations
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
- Authors
- Goetz, L., Seedat, N., Vandersluis, R., & van der Schaar, M.
- Affiliations
- GlaxoSmithKline (United Kingdom) (GB)
- University of Cambridge (GB)
- Corresponding author
- Goetz, Lea
- Seedat, Nabeel
- Venue
- npj Digital Medicine
- Published
- 2024-05-21
- Volume
- 7
- Issue
- 1
- Pages
- 126-126
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11109198
- OpenAlex
- W4398183239
- Cited by
- 124
- References
- 18
- Citations by year
- 2026: 51 · 2025: 56 · 2024: 15 · 2023: 2
- FWCI
- 11.41
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Cystic Fibrosis Trust
- GlaxoSmithKline foundation
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- generalization, responsible AI, selective prediction, clinical AI, human-centered AI, bioethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
