Preventing dataset shift from breaking machine-learning biomarkers
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Machine learning brings the hope of finding new biomarkers extracted from cohorts with rich biomedical measurements. A good biomarker is one that gives reliable detection of the corresponding condition. However, biomarkers are often extracted from a cohort that differs from the target population. Such a mismatch, known as a dataset shift, can undermine the application of the biomarker to new individuals. Dataset shifts are frequent in biomedical research, e.g., because of recruitment biases. When a dataset shift occurs, standard machine-learning techniques do not suffice to extract and validate biomarkers. This article provides an overview of when and how dataset shifts break machine-learning–extracted biomarkers, as well as detection and correction strategies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001779 · Cited by 12,728 · cited 2×
“The same behavior is visible on real epidemiologic data with age shifts, such as predicting the smoking status of participants in the UKBiobank study [23], as shown in Figure 2.”
BackgroundAustin, P. C. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.568786 · Cited by 12,242 · cited 1×
“Indeed, an over-constrained model may be able to fit the data well only in some regions of the input feature space (Fig.”
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · cited 1×
“Even more frequently, prior probability shift arises when one rare class is over-represented in the training data so that the dataset is more balanced, as when extracting a biomarker from a case-control cohort, or when the dataset is resampled as a strategy to handle the class imbalance problem [61].”
Adversarial Discriminative Domain Adaptation
Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.316 · Cited by 5,154 · cited 1×
“Therefore in predictive settings, where the goal is machine-learning models that generalize well, large and diverse datasets are desirable.”
Learning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505 · cited 1×
“…methods strive to extract features that are predictive of the target while having similar distributions in the source and target domains [e.g. Long et al., 2015], or while preventing an adversary to distinguish source and target data [“domain-adversarial” learning, e.g. Tzeng et al., 2017].”
MethodA survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Nature · doi:10.1038/466029a · Cited by 3,441
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844 · cited 1×
“However the usual (unweighted) estimator is not consistent for an over-constrained (misspecified) model.”
Strimbu, K., & Tavel, J. A. (2010) · Current Opinion in HIV and AIDS · doi:10.1097/coh.0b013e32833ed177 · Cited by 2,062
Return of Frustratingly Easy Domain Adaptation
Sun, B., Feng, J., & Saenko, K. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10306 · Cited by 2,004 · cited 1×
“This research has been conducted using the UK Biobank Resource under Application No. 45551. in the support of p t .”
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Improving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, H. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(00)00115-4 · Cited by 1,678
Liu, M. C., Oxnard, G. R., Klein, E. A., Swanton, C., Seiden, M. V., Liu, M. C., Oxnard, G. R., Klein, E. A., Smith, D., Richards, D., et al. (2020) · Annals of Oncology · doi:10.1016/j.annonc.2020.02.011 · Cited by 1,495
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 2×
“Not only can these shifts reduce overall predictive performance, they can also lead to discriminative clinical decisions for poorly represented populations [30, 31, 32, 33, 34].”
BackgroundBuilding better biomarkers: brain models in translational neuroimaging
Woo, C.-W., Chang, L. J., Lindquist, M. A., & Wager, T. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4478 · Cited by 1,056
Learning and evaluating classifiers under sample selection bias
Zadrozny, B. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015425 · Cited by 800 · cited 1×
“We summarize 2 frequently met scenarios that are easier to handle than the general case and can call for different adjustments: covariate shift and prior probability shift.”
Establishment of Best Practices for Evidence for Prediction: A Review
Poldrack, R. A., Huckins, G., & Varoquaux, G. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.3671 · Cited by 738
Self-citationRepresentation of the Elderly, Women, and Minorities in Heart Failure Clinical Trials
Heiat, A., Gross, C. P., & Krumholz, H. M. (2002) · Archives of Internal Medicine · doi:10.1001/archinte.162.15.1682 · Cited by 723 · cited 4×
“Beyond explicit selection criteria, many factors such as age, ethnicity, or socioeconomic status influence participation in biomedical studies [27, 28, 20, 29].”
BackgroundInfluentialExternal validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685 · cited 1×
“More generally, machine-learning systems may fail on data from different imaging devices, hospitals, populations with a different age distribution, and so forth.”
Cost-sensitive learning by cost-proportionate example weighting
Zadrozny, B., Langford, J., & Abe, N. (2004) · Third IEEE International Conference on Data Mining · doi:10.1109/icdm.2003.1250950 · Cited by 680
The seven tools of causal inference, with reflections on machine learning
Pearl, J. (2019) · Communications of the ACM · doi:10.1145/3241036 · Cited by 659 · cited 1×
“Unfortunately, the biases in participant recruitment may be similar in independently collected datasets.”
Machine Learning and Health Care Disparities in Dermatology
Adamson, A. S., & Smith, A. (2018) · JAMA Dermatology · doi:10.1001/jamadermatol.2018.2348 · Cited by 639 · cited 1×
“For instance, there is a risk that machinelearning analysis of dermoscopic images under-diagnoses malignant moles on skin tones that are typically under-represented in the training set [55].”
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553 · cited 1×
“We now discuss some misconceptions and confusions with problems not directly related to dataset shift.”
Selection bias and information bias in clinical research
Tripepi, G., Jager, K. J., Dekker, F. W., & Zoccali, C. (2010) · Nephron Clinical Practice · doi:10.1159/000312871 · Cited by 425 · cited 1×
“– [7, 8] may cause cohorts to capture only a small range of possible patients and disease manifestations in the presence of spectrum effects [9, 10].”
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411 · cited 1×
“Not only can these shifts reduce overall predictive performance, they can also lead to discriminative clinical decisions for poorly represented populations [30, 31, 32, 33, 34].”
Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects
Glocker, B., Robinson, R., Castro, D. C., Dou, Q., & Konukoglu, E. (2019) · arXiv · doi:10.48550/arxiv.1910.04597 · Cited by 74 · cited 1×
“Selected participants verified many criteria, including being a smoker and not having recent medical problems such as pneumonia.”
Best practices for authors of healthcare-related artificial intelligence manuscripts
Kakarmath, S., Esteva, A., Arnaout, R., Harvey, H., Kumar, S., Muse, E., Dong, F., Wedlund, L., & Kvedar, J. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00336-w · Cited by 46 · cited 2×
“Testing on a dataset collected independently is therefore a useful check but no silver bullet to rule out dataset shift issues.”
Cited by
Crosby, D., Bhatia, S., Brindle, K. M., Coussens, L. M., Dive, C., Emberton, M., Esener, S., Fitzgerald, R. C., Gambhir, S. S., Kuhn, P., et al. (2022) · Science · doi:10.1126/science.aay9040 · Cited by 1,379
“AI andMLmodels are often developed (or trained) on datasets derived from selected populations that do not represent the real population where the AI-derived test would be used, so they cannot be extrapolated to real-life conditions (91).”
Machine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
Self-citationData drift in medical machine learning: implications and potential remedies
Sahiner, B., Chen, W., Samala, R. K., & Petrick, N. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20220878 · Cited by 279
Evaluating Machine Learning Models and Their Diagnostic Value
Varoquaux, G., & Colliot, O. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_20 · Cited by 242
Self-citationData leakage inflates prediction performance in connectome-based machine learning models
Rosenblatt, M., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024) · Nature Communications · doi:10.1038/s41467-024-46150-w · Cited by 236
Metadata
- Authors
- Dockès, J., Varoquaux, G., & Poline, J.-B.
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-09
- Volume
- 10
- Issue
- 9
- ISSN
- 2047-217X (ISSN-L)
- OpenAlex
- W3183267169
- Cited by
- 86
- References
- 30
- FWCI
- 4.94
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.