AutoPrognosis 2.0: Democratizing diagnostic and prognostic modeling in healthcare with automated machine learning
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Diagnostic and prognostic models are increasingly important in medicine and inform many clinical decisions. Recently, machine learning approaches have shown improvement over conventional modeling techniques by better capturing complex interactions between patient covariates in a data-driven manner. However, the use of machine learning introduces technical and practical challenges that have thus far restricted widespread adoption of such techniques in clinical settings. To address these challenges and empower healthcare professionals, we present an open-source machine learning framework, AutoPrognosis 2.0, to facilitate the development of diagnostic and prognostic models. AutoPrognosis leverages state-of-the-art advances in automated machine learning to develop optimized machine learning pipelines, incorporates model explainability tools, and enables deployment of clinical demonstrators,without requiring significant technical expertise. To demonstrate AutoPrognosis 2.0, we provide an illustrative application where we construct a prognostic risk score for diabetes using the UK Biobank, a prospective study of 502,467 individuals. The models produced by our automated framework achieve greater discrimination for diabetes than expert clinical risk scores. We have implemented our risk score as a web-based decision support tool, which can be publicly accessed by patients and clinicians. By open-sourcing our framework as a tool for the community, we aim to provide clinicians and other medical practitioners with an accessible resource to develop new risk scores, personalized diagnostics, and prognostics using machine learning techniques. Software:https://github.com/vanderschaarlab/AutoPrognosis
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
MissForest—Non-parametric missing value imputation for mixed-type data
Stekhoven, D. J., & Bühlmann, P. (2012) · Bioinformatics · doi:10.1093/bioinformatics/btr597 · Cited by 6,156
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212 · cited 3×
“…the optimization of ML pipelines involving data processing, model development, and model training, we reduce the burden on technical experts and turn deriving ML models from an art to a science, democratizing machine learning and opening the field to non-ML domain experts, such as clinicians.”
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · cited 1×
“In particular, there are substantial technical challenges in developing, understanding, and deploying ML systems which currently render them largely inaccessible for medical practitioners [3, 4, 5, 6].”
Flexible Imputation of Missing Data, Second Edition
van Buuren, S. (2018) · doi:10.1201/9780429492259 · Cited by 2,273
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 1×
“Without a transparent understanding of how a model makes predictions it may act in unintended and undesirable ways, for example learning incorrect or aberrant features unique to the training data [23, 24].”
BackgroundAuto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
Ensuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341
Ethical and legal challenges of artificial intelligence-driven healthcare
Gerke, S., Minssen, T., & Cohen, G. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00012-5 · Cited by 1,170
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044
Variable selection strategies and its importance in clinical prediction modelling.
Chowdhury, M. Z. I., & Turin, T. C. (2020) · Family Medicine and Community Health · doi:10.1136/fmch-2019-000262 · Cited by 1,011
Sun, T. Q., & Medaglia, R. (2019) · Government Information Quarterly · doi:10.1016/j.giq.2018.09.008 · Cited by 948 · cited 2×
“Our framework employs automated machine learning [11] to tackle the challenges faced by clinical users.”
Gale, E. A. M., & Gillespie, K. M. (2001) · Diabetologia · doi:10.1007/s001250051573 · Cited by 558
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540
Challenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366 · cited 1×
“…the myriad of choices that must be made when developing a new predictive model for diagnosis or prognosis, such as: what imputation strategy should be used; how should the data be preprocessed; what (ML) model is best suited for the specific task; what configuration of hyperpara-meters should be…”
Winkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.1735 · Cited by 365 · cited 1×
“Without a transparent understanding of how a model makes predictions it may act in unintended and undesirable ways, for example learning incorrect or aberrant features unique to the training data [23, 24].”
BackgroundAi for social good: unlocking the opportunity for positive impact
Tomašev, N., Cornebise, J., Hutter, F., Mohamed, S., Picciariello, A., Connelly, B., Belgrave, D. C. M., Ezer, D., Haert, F. C. V. D., Mugisha, F., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15871-z · Cited by 357
Steps to avoid overuse and misuse of machine learning in clinical research
Volovici, V., Syn, N. L., Ercole, A., Zhao, J. J., & Liu, N. (2022) · Nature Medicine · doi:10.1038/s41591-022-01961-6 · Cited by 110
Prognostication and risk factors for cystic fibrosis via automated machine learning
Alaa, A. M., & van der Schaar, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-29523-2 · Cited by 80 · cited 2×
“AutoPrognosis can be used with data from many different origins, such as biobanks [12], registries [13, 14], and private hospital data [17].”
Transparency of machine-learning in healthcare: The GDPR & European health law
Mourby, M., Ó Cathaoir, K., & Collin, C. B. (2021) · Computer Law & Security Review · doi:10.1016/j.clsr.2021.105611 · Cited by 48 · cited 2×
“Predictive models alone are not sufficient and more must be done to engender model trust from both clinical users [5] and regulatory bodies [26, 27, 28].”
BackgroundInflation of test accuracy due to data leakage in deep learning-based classification of OCT images
Tampu, I. E., Eklund, A., & Haj-Hosseini, N. (2022) · Scientific Data · doi:10.1038/s41597-022-01618-6 · Cited by 1
Research works printed but not linked: 48
- EJ Topol, 2019, High-performance medicine: The convergence of human and artificial intelligence, Nat Med, 25, 44, 10.1038/s41591-018-0300-7 · DOI · Paper page
- L Petersson, 2022, Challenges to implementing artificial intelligence in healthcare: A qualitative interview study with healthcare leaders in Sweden, BMC Health Serv Res, 22, 850, 10.1186/s12913-022-08215-8 · DOI · Paper page
- Alaa A, van der Schaar M. AutoPrognosis: Automated clinical prognostic modeling via Bayesian optimization with structured kernel learning. In: Proceedings of the 35th International Conference on Machine Learning. 2018;80:139–148.
- J Bergstra, 2011, Algorithms for hyper-parameter optimization, Advances in Neural Information Processing Systems, 24
- M Feurer, 2015, Efficient and robust automated machine learning, Advances in Neural Information Processing Systems, 28, 2755
- AM Alaa, 2019, Cardiovascular disease risk prediction using automated machine learning: A prospective study of 423,604 UK Biobank participants, PLoS One, 14, 1
- AM Alaa, 2021, Machine learning to guide the use of adjuvant therapies for breast cancer, Nat Mach Intell, 3, 716, 10.1038/s42256-021-00353-8 · DOI · Paper page
- H Rahbar, 2020, The value of patient and tumor factors in predicting preoperative breast MRI outcomes, Radiol Imaging Cancer, 2, e190099, 10.1148/rycan.2020190099 · DOI
- Z Qian, 2021, CPAS: The UK’s national machine learning-based hospital capacity planning system for COVID-19, Machine Learning, 110, 15, 10.1007/s10994-020-05921-4 · DOI
- AA Shah, 2021, Development of a novel, potentially universal machine learning algorithm for prediction of complications after total hip arthroplasty, J Arthroplasty, 36, 1655, 10.1016/j.arth.2020.12.040 · DOI
- SK Devana, 2021, A novel, potentially universal machine learning algorithm to predict complications in total knee arthroplasty, Arthroplast Today, 10, 135, 10.1016/j.artd.2021.06.020 · DOI
- AA Shah, 2022, Machine learning-driven identification of novel patient factors for prediction of major complications after posterior cervical spinal fusion, Eur Spine J, 31, 1952, 10.1007/s00586-021-06961-7 · DOI
- AA Shah, 2022, A risk calculator for the prediction of C5 nerve root palsy after instrumented cervical fusion, World Neurosurg, 166, e703, 10.1016/j.wneu.2022.07.082 · DOI
- T Callender, 2023, Assessing eligibility for lung cancer screening: Parsimonious multi-country ensemble machine learning models for lung cancer prediction, medRxiv, 2023
- S van Buuren, 2011, mice: Multivariate imputation by chained equations in R, J Stat Softw, 45, 1
- Jarrett D, Cebere BC, Liu T, Curth A, van der Schaar M. HyperImpute: Generalized iterative imputation with automatic model selection. In: Proceedings of the 39th International Conference on Machine Learning. 2022;162:9916–9937.
- J Liu, 2013, On the stationary distribution of iterative imputations, Biometrika, 101, 155, 10.1093/biomet/ast044 · DOI
- Wang Z, Li C, Jegelka S, Kohli P. Batched high-dimensional Bayesian optimization via structural kernel learning. In: Proceedings of the 34th International Conference on Machine Learning. 2017; p. 3656–3664.
- L Li, 2018, Hyperband: A novel bandit-based approach to hyperparameter optimization, J Mach Learn Res, 18, 1
- SF Crone, 2006, The impact of preprocessing on data mining: An evaluation of classifier sensitivity in direct marketing, Eur J Oper Res, 173, 781, 10.1016/j.ejor.2005.07.023 · DOI · Paper page
- MW Kattan, 2016, American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine, CA Cancer J Clin, 66, 370, 10.3322/caac.21339 · DOI · Paper page
- SM Lundberg, 2017, A unified approach to interpreting model predictions, Advances in Neural Information Processing Systems, 30
- J Crabbe, 2021, Explaining latent representations with a corpus of examples, Advances in Neural Information Processing Systems, 34, 12154
- J Crabbe, 2020, Learning outside the Black-Box: The pursuit of interpretable models, Advances in Neural Information Processing Systems, 33, 17838
- G Luo, 2017, Automating construction of machine learning models with clinical big data: Proposal rationale and methods, JMIR Res Protoc, 6, e175, 10.2196/resprot.7757 · DOI
- Elshawi R, Maher M, Sakr S. Automated machine learning: State-of-the-art and open challenges. arXiv preprint arXiv:190602287. 2019;.
- D Sculley, 2015, Hidden technical debt in machine learning systems, Advances in Neural Information Processing Systems, 28
- B Nestor, 2018, Rethinking clinical prediction: Why machine learning must consider year of care and feature aggregation, Machine Learning for Health (ML4H) Workshop at NeurIPS
- DR Cox, 1972, Regression models and life-tables, J R Stat Soc Series B Stat Methodol, 34, 187, 10.1111/j.2517-6161.1972.tb00899.x · DOI · Paper page
- O Akbilgic, 2019, The promise of machine learning: When will it be delivered?, J Card Fail, 25, 484, 10.1016/j.cardfail.2019.04.006 · DOI
- MA Schulz, 2020, Different scaling of linear models and deep learning in UKBiobank brain images versus machine-learning datasets, Nat Commun, 11, 4238, 10.1038/s41467-020-18037-z · DOI · Paper page
- E Steyerberg, 2008, Clinical prediction models: A practical approach to development, validation, and updating
- I Guyon, 2003, An introduction to variable and feature selection, J Mach Learn Res, 3, 1157
- AJ DeGrave, 2021, AI for radiographic COVID-19 detection selects shortcuts over signal, Nat Mach Intell, 3, 610, 10.1038/s42256-021-00338-7 · DOI · Paper page
- CH Yoon, 2022, Machine learning in medicine: Should the pursuit of enhanced interpretability be abandoned?, J Med Ethics, 48, 581, 10.1136/medethics-2020-107102 · DOI · Paper page
- N Laiteerapong, 2015, The pace of change in medical practice and health policy: Collision or coexistence?, J Gen Intern Med, 30, 848, 10.1007/s11606-015-3182-0 · DOI
- DM Gupta, 2017, The physician’s experience of changing clinical practice: a struggle to unlearn, Implement Sci, 12, 28, 10.1186/s13012-017-0555-2 · DOI · Paper page
- RJ LeVeque, 2012, Reproducible research for scientific computing: Tools and strategies for changing the culture, Comput Sci Eng, 14, 13, 10.1109/MCSE.2012.38 · DOI · Paper page
- M Miłkowski, 2018, Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail, J Comput Neurosci, 45, 163, 10.1007/s10827-018-0702-z · DOI · Paper page
- C Sudlow, 2015, UK Biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age, PLoS Med, 12, 1, 10.1371/journal.pmed.1001779 · DOI · Paper page
- L Adamska, 2015, Challenges of linking to routine healthcare records in UK Biobank, Trials, 16, O68, 10.1186/1745-6215-16-S2-O68 · DOI
- H Bang, 2009, Development and validation of a patient self-assessment score for diabetes risk, Ann Intern Med, 151, 775, 10.7326/0003-4819-151-11-200912010-00005 · DOI · Paper page
- J Lindström̈, 2003, The Diabetes Risk Score: A practical tool to predict type 2 diabetes risk, Diabetes Care, 26, 725, 10.2337/diacare.26.3.725 · DOI · Paper page
- J Hippisley-Cox, 2017, Development and validation of QDiabetes-2018 risk prediction algorithm to estimate future risk of type 2 diabetes: cohort study, BMJ, 359
- AJ Vickers, 2008, Decision analysis for the evaluation of diagnostic tests, prediction models, and molecular markers, Am Stat, 62, 314, 10.1198/000313008X370302 · DOI · Paper page
- J Cohen, 2013, Statistical power analysis for the behavioral sciences, 10.4324/9780203771587 · DOI · Paper page
- J Nano, 2017, Gamma-glutamyltransferase levels, prediabetes and type 2 diabetes: A Mendelian randomization study, Int J Epidemiol, 46, 1400, 10.1093/ije/dyx006 · DOI
- International Diabetes Federation, 2013, IDF Diabetes Atlas
Cited by
Learning the natural history of human disease with generative transformers
Shmatko, A., Jung, A. W., Gaurav, K., Brunak, S., Mortensen, L. H., Birney, E., Fitzgerald, T., & Gerstung, M. (2025) · Nature · doi:10.1038/s41586-025-09529-3 · Cited by 110
Yuan, H. (2024) · Health Care Science · doi:10.1002/hcs2.114 · Cited by 37
Timeseriesflattener: A Python package for summarizing features from (medical) time series.
Bernstorff, M., Enevoldsen, K., Damgaard, J., Danielsen, A., & Hansen, L. (2023) · Journal of Open Source Software · doi:10.21105/joss.05197 · Cited by 10
Metadata
- Authors
- Imrie, F., Cebere, B., McKinney, E. F., & van der Schaar, M.
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-06-22
- Volume
- 2
- Issue
- 6
- Pages
- e0000276
- ISSN
- 2767-3170 (ISSN-L)
- OpenAlex
- W4381611976
- Cited by
- 45
- References
- 24
- FWCI
- 3.95
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.