The DOI matches its Crossref record; 61 of the 68 references it cites were found in a registry or an index; 10 of the 31 sources its claims cite were checked for support.
A scoping review of robustness concepts for machine learning in healthcare
Open accessAbstract
While machine learning-based solutions have demonstrated strong performance, robustness remains ambiguous and often overlooked. Eight general concepts of robustness emerged from 274 included records.
Overview
CoChat Summary
TL;DR
This scoping review mapped how robustness is defined in healthcare machine learning literature and found 8 distinct robustness concepts across 274 eligible records. The concepts differed by data type and model type, suggesting that “robustness” is not a single uniform property in healthcare ML.
Key findings
- The review included 274 eligible records from 8585 initially retrieved records; 190 (69%) were journal articles and 81 (30%) were from conferences, workshops, or symposiums.
- Across 526 extracted combinations, the authors identified 8 robustness concepts: input perturbations and alterations, missing data, label noise, imbalanced data, feature extraction and selection, model specification and learning, external data and domain shift, and adversarial attacks.
- The most frequently addressed concept was input perturbations and alterations (142/526, 27%), while imbalanced data was the least commonly tackled (15/526, 3%).
- Robustness concepts varied by data and model: feature extraction/selection was mainly emphasized in image-derived data (33%) and omics (22%); adversarial attacks were mainly tackled in image (22%) and physiological signal data (7%); missing data was mostly addressed in clinical data (20%); deep learning applications covered the most concepts (8) and hybrid models the fewest (4).
Methods
- This was a scoping review conducted and reported per Joanna Briggs Institute methodology and PRISMA-ScR, with a protocol uploaded on Open Science Framework.
- The authors searched PubMed, IEEE Xplore, and Web of Science (search date: 1 March 2023), plus grey literature sources (Google Scholar, conference/workshop websites) and reference lists; they included English records with abstracts on supervised or semi-supervised ML models for healthcare decision support.
- Data extraction captured medical application, model, data type, notion of robustness, and robustness metrics; extracted records were categorized by medical specialty, model class, and data type, then robustness notions were grouped into general concepts through iterative reviewer discussion.
Limitations
- The paper states it was not meant to provide or assess methods for “robustifying” a model; it only maps robustness concepts.
- The review is mostly restricted to studies using the terms “robust,” “noise,” and “perturbation”; the authors note that using other terms (e.g., stability, resilience, reliability, vulnerability) might have yielded different studies.
- The review is temporally limited and does not include more recent foundation-model studies published after the search date.
- Evident from the text: the search was limited to English-language records with abstracts, and the screening/search strategy may therefore have missed relevant non-English or abstractless studies.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
8 Valid · 0 Not valid · 10 Not checked · 43 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“studies that evaluate the robustness to various data distributions through external validation inherently address the generalizability of the model19–22.”
§ A scoping review of robustness concepts for machine learning in healthcare > Discussion
What the sources say
- Not checked — no full text of the sourcecites [19] Kyung, S. et al. Improved performance and robustness of multi-task representation learning with consistency loss between pretexts for intracranial hemorrhage identification in head CT. Med. Image Anal. 81, 102489 (2022). source
- Not checked — no full text of the sourcecites [20] Valliani, A. A. et al. Robust Prediction of Non-home Discharge After Thoracolumbar Spine Surgery With Ensemble Machine Learning and Validation on a Nationwide Cohort. World Neurosurg. 165, e83–e91 (2022). source
- Supportedcites [21] Huo, J., Wu, L. & Zang, Y. Development and Validation of a Robust Immune-Related Prognostic Signature for Gastric Cancer. J. Immunol. Res. 2021, 5554342 (2021). source
Section 2.4 reports external validation in three independent cohorts. The Results (§3.4) describe consistent findings across those cohorts, and the Discussion says the risk score showed “universal applicability via internal and external validation.” Together, these support the cited paper as an example of using external validation to assess robustness and generalizability. (LLM, conf=0.85)
- Partly supportedcites [22] Zhang, W. et al. A Novel and Robust Prognostic Model for Hepatocellular Carcinoma Based on Enhancer RNAs-Regulated Genes. Front. Oncol. 12, 849242 (2022). sourceExperimental — accuracy not yet measured
Methods §2.2 says the authors used an external HCC dataset to assess the model’s predictive capability and generalization, and Discussion §4 links its external-validation performance to robustness. The paper does not report testing robustness across various data distributions; Discussion §4 also says a multicenter study is needed to further establish generalizability. Page numbers were not provided in our copy. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the large number based on omics can be ascribed to The Cancer Genome Atlas Program (TCGA), a publicly available collection of genomes that was developed mainly for cancer-related research42.”
§ A scoping review of robustness concepts for machine learning in healthcare > Discussion
What the sources say
- Partly supportedcites [42] Weinstein, J. N. et al. The Cancer Genome Atlas Pan-Cancer Analysis Project. Nat. Genet. 45, 1113–1120 (2013). sourceExperimental — accuracy not yet measured
The Pan-Cancer Project section describes TCGA’s cancer-focused datasets across genomic, epigenomic, transcriptional and proteomic platforms. From Many Tumours to the Individual Patient says the datasets are publicly available and a resource for cancer analysis. It does not establish that TCGA accounts for the large number of omics-based applications. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“addressing the robustness of the model to specific minority groups directly relates to the notion of fairness23–25.”
§ A scoping review of robustness concepts for machine learning in healthcare > Discussion
What the sources say
- Partly supportedcites [23] Guan, Y. et al. Assessment of the timeliness and robustness for predicting adult sepsis. iScience 24, 102106 (2021). sourceExperimental — accuracy not yet measured
The Results section evaluates sepsis-model performance across age, gender, and racial groups, including a small Asian subgroup for which performance was slightly worse (page unavailable). That supports the subgroup-robustness part of the claim, but our copy does not explicitly connect this evaluation to fairness. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [24] Khoshnevisan, F. & Chi, M. Unifying Domain Adaptation and Domain Generalization for Robust Prediction Across Minority Racial Groups. in Machine Learning and Knowledge Discovery in Databases. Research Track (eds. Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J. & Lozano, J. A.) 521–537 (Springer International Publishing, Cham, 2021). https://doi.org/10.1007/978-3-030-86486-6_32. source
- Not checked yetcites [25] Lu, Y. et al. Robust Speech and Natural Language Processing Models for Depression Screening. in 2020 IEEE Signal Processing in Medicine and Biology Symposium (SPMB) 1–5 (2020). https://doi.org/10.1109/SPMB50085.2020.9353611. source
Checked before — its earlier verdict is held back until a new check runs.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Studies addressing missing data were not limited to a particular missingness mechanism (e.g., missing completely at random, missing at random, missing not at random)31,32.”
§ A scoping review of robustness concepts for machine learning in healthcare > Discussion
What the sources say
- Partly supportedcites [31] Çallı, E. et al. Deep learning with robustness to missing data: A novel approach to the detection of COVID-19. PloS One 16, e0255301 (2021). sourceExperimental — accuracy not yet measured
The Data and methods > Data section (page unavailable) describes experiments with random subsets of inputs and two clinically motivated subsets, so the study did not examine only randomly selected missing inputs. It does not classify those scenarios as missing completely at random, missing at random, or missing not at random, or establish robustness across those mechanisms. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [32] Ramoni, M., Sebastiani, P. & Dybowski, R. Robust outcome prediction for intensive-care patients. Methods Inf. Med. 40, 39–45 (2001). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Label noise can also derive from human uncertainty when using the consensus of multiple annotators, often needed due to the absence of a pathologically confirmed diagnosis label, thereby introducing inherent uncertainty35–37.”
§ A scoping review of robustness concepts for machine learning in healthcare > Discussion
What the sources say
- Not checked yetcites [35] Ju, L. et al. Improving Medical Images Classification With Label Noise Using Dual-Uncertainty Estimation. IEEE Trans. Med. Imaging 41, 1533–1546 (2022). source
Checked before — its earlier verdict is held back until a new check runs.
- Not checked — no full text of the sourcecites [36] Peng, T. et al. Noise Robust Learning with Hard Example Aware for Pathological Image classification. in 2020 IEEE 6th International Conference on Computer and Communications (ICCC) 1903–1907 (2020). https://doi.org/10.1109/ICCC51575.2020.9344937. source
- Supportedcites [37] Hekler, A. et al. Effects of Label Noise on Deep Learning-Based Skin Cancer Classification. Front. Med. 7, 177 (2020). source
The Introduction says that non-biopsy-verified training images are often labeled by a consensus of several dermatologists and that this introduces high label noise. It also identifies inter- and intra-rater variability as a source of label noise. The paper’s own study used biopsy-verified images, but the claim describes the broader practice the Introduction discusses. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not checked yet
29 more claims
Create a free account to see all 61 claims and to verify the 8 references still open.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 14
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Even the most accurate AI model will fail if oncologists cannot use it effectively [139,140,141].”
From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [140]
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742 · cited 1×
“Given the need to clarify the concept of machine learning robustness and identify the key characteristics related to that concept, we decided to conduct a scoping review 16 – 18 .”
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0611-x · Cited by 11,385
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092
Guidance for conducting systematic scoping reviews
Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000050 · Cited by 7,472
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 1×
“For instance, the ChestX-ray8 dataset, a publicly available dataset of 108,948 chest X-rays covering eight diseases 41 , contributes to the high number of applications in pulmonology.”
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · cited 1×
“Hence, their apparent performance does not take into account external sources of perturbations and variations that can affect the model behaviour 8,9 .”
Doubly robust estimation of causal effects
Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., & Davidian, M. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwq439 · Cited by 1,158 · cited 1×
“Records about treatment recommendations were excluded if the notion tackled by the study was related to so-called “ doubly-robust ” methods 52 .”
Research works printed but not linked: 1
- Saab, K. et al. Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity. in Proceedings of the 7th Machine Learning for Healthcare Conference 760–784 (PMLR, 2022).
Cited by
AI-Driven Real-Time Monitoring of Cardiovascular Conditions With Wearable Devices: Scoping Review
Abedi, A., Verma, A., Jain, D., Kaetheeswaran, J., Chui, C., Lankarany, M., & Khan, S. S. (2025) · JMIR mhealth and uhealth · doi:10.2196/73846 · Cited by 44
“Balendran et al [50] conducted a scoping review to explore robustness concepts in the application of machine learning models in health care.”
Gangwal, A., & Lavecchia, A. (2026) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.70072 · Cited by 13
Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery
Kuno, M., Osumi, H., Udagawa, S., Yoshikawa, K., Ooki, A., Shinozaki, E., Ishikawa, T., Oba, J., Yamaguchi, K., & Sakurada, K. (2025) · Current Oncology · doi:10.3390/curroncol32110588 · Cited by 7
Xue, M., Yang, J., Wang, H., Yan, Z., Chen, X., Gao, W., Luo, R., Lv, X., & Ye, Z. (2025) · Journal of Anesthesia and Translational Medicine · doi:10.1016/j.jatmed.2025.06.003 · Cited by 10
Eisenhofer, G., Peitzsch, M., Mantik, K., Schulze, M., Constantinescu, G. A., Lu, Z. X., Remde, H., Fuß, C. T., Williams, T. A., Gruber, S., et al. (2025) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-0200 · Cited by 8
Metadata
- Authors
- Balendran, A., Beji, C., Bouvier, F., Khalifa, O., Evgeniou, T., Ravaud, P., & Porcher, R.
- Affiliations
- Inserm (FR)
- INSEAD (FR)
- Corresponding author
- Balendran, Alan
- Venue
- npj Digital Medicine
- Published
- 2025-01-17
- Volume
- 8
- Issue
- 1
- Pages
- 38-38
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11742061
- OpenAlex
- W4406494794
- Cited by
- 43
- References
- 68
- Citations by year
- 2026: 25 · 2025: 18
- FWCI
- 10.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Bpifrance
- Agence Nationale de la Recherche (ANR-23-IACL-0008, ANR-19-P3IA-0001)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- robustness, machine learning, artificial intelligence, health care, model stability, generalizability, model validation, model deployment, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A scoping review of robustness concepts for machine learning in healthcare. https://app.cochat.ai/papers/10.1038/s41746-024-01420-1
- The paper
- Balendran, A., Beji, C., Bouvier, F., Khalifa, O., Evgeniou, T., Ravaud, P., & Porcher, R. (2025). A scoping review of robustness concepts for machine learning in healthcare. npj Digital Medicine. https://doi.org/10.1038/s41746-024-01420-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.