Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine
Open accessAbstract
Objective: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. Materials and methods: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. Results: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. Discussion: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. Conclusion: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · see citing works
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242 · see citing works
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,107 · see citing works
A large-scale evaluation of computational protein function prediction
Radivojac, P., Clark, W. T., Oron, T. R., Schnoes, A. M., Wittkop, T., Sokolov, A., Graim, K., Funk, C., Verspoor, K., Ben-Hur, A., et al. (2013) · Nature Methods · doi:10.1038/nmeth.2340 · Cited by 1,028 · see citing works
Self-citationGoldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936 · see citing works
Critical assessment of methods of protein structure prediction (CASP)—Round <scp>XIV</scp>
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26237 · Cited by 503 · see citing works
A decade of CASP: progress, bottlenecks and prognosis in protein structure prediction
Moult, J. (2005) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2005.05.011 · Cited by 485 · see citing works
Zhou, N., Jiang, Y., Bergquist, T. R., Lee, A. J., Kacsoh, B. Z., Crocker, A. W., Lewis, K. A., Georghiou, G., Nguyen, H. N., Hamid, M. N., et al. (2019) · Genome Biology · doi:10.1186/s13059-019-1835-8 · Cited by 452 · see citing works
Self-citationAn expanded evaluation of protein function prediction methods shows an improvement in accuracy
Jiang, Y., Oron, T. R., Clark, W. T., Bankapur, A., D’Andrea, D., Lepore, R., Funk, C. S., Kahanda, I., Verspoor, K. M., Ben‐Hur, A., et al. (2016) · Genome biology · doi:10.1186/s13059-016-1037-6 · Cited by 390 · see citing works
A Review of Challenges and Opportunities in Machine Learning for Health
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., Chen, I. Y., & Ranganath, R. (2018) · PubMed Central · doi:10.48550/arxiv.1806.00388 · Cited by 347 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Self-citationCrowdsourcing biomedical research: leveraging communities as innovation engines
Sáez-Rodríguez, J., Costello, J. C., Friend, S. H., Kellen, M., Mangravite, L. M., Meyer, P., Norman, T., & Stolovitzky, G. A. (2016) · Nature Reviews Genetics · doi:10.1038/nrg.2016.69 · Cited by 165 · see citing works
Weng, S. F., Vaz, L., Qureshi, N., & Kai, J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0214365 · Cited by 118 · see citing works
Omberg, L., Ellrott, K., Yuan, Y., Kandoth, C., Wong, C., Kellen, M., Friend, S. H., Stuart, J. M., Liang, H., & Margolin, A. A. (2013) · Nature Genetics · doi:10.1038/ng.2761 · Cited by 113 · see citing works
Chen, J. H., Alagappan, M., Goldstein, M. K., Asch, S. M., & Altman, R. B. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.03.006 · Cited by 101 · see citing works
The self‐assessment trap: can we all be better than average?
Norel, R., Rice, J. J., & Stolovitzky, G. A. (2011) · Molecular Systems Biology · doi:10.1038/msb.2011.70 · Cited by 93 · see citing works
Jauk, S., Kramer, D., Großauer, B., Rienmüller, S., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa113 · Cited by 84 · see citing works
Alternative models for sharing confidential biomedical data
Guinney, J., & Sáez-Rodríguez, J. (2018) · Nature Biotechnology · doi:10.1038/nbt.4128 · Cited by 72 · see citing works
Self-citationCAFA and the Open World of protein function predictions
Dessimoz, C., Škunca, N., & Thomas, P. D. (2013) · Trends in Genetics · doi:10.1016/j.tig.2013.09.005 · Cited by 57 · see citing works
Reports from the fifth edition of CAGI: The Critical Assessment of Genome Interpretation
Andreoletti, G., Pal, L. R., Moult, J., & Brenner, S. E. (2019) · Human Mutation · doi:10.1002/humu.23876 · Cited by 56 · see citing works
Daneshjou, R., Wang, Y., Bromberg, Y., Bovo, S., Martelli, P. L., Babbi, G., Lena, P. D., Casadio, R., Edwards, M. D., Gifford, D. K., et al. (2017) · Human Mutation · doi:10.1002/humu.23280 · Cited by 43 · see citing works
Self-citationPredicting mortality of elderly patients acutely admitted to the Department of Internal Medicine
Smolin, B., Levy, Y., Sabbach-Cohen, E., et al. (2014) · International Journal of Clinical Practice · doi:10.1111/ijcp.12564 · Cited by 39 · see citing works
Bergquist, T. R., Yan, Y., Schaffter, T., Yu, T. V., Pejaver, V. R., Hammarlund, N. E., Prosser, J., Guinney, J., & Mooney, S. D. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa083 · Cited by 16 · see citing works
Self-citationCai, B., Li, B., Kiga, N., Thusberg, J., Bergquist, T. R., Chen, Y., Niknafs, N., Carter, H. K., Tokheim, C. J., Guthrie, V. B., et al. (2017) · Human Mutation · doi:10.1002/humu.23265 · Cited by 14 · see citing works
Self-citationFahey, M., Rudd, A. G., Béjot, Y., Wolfe, C. D. A., & Douiri, A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014607 · Cited by 9 · see citing works
Racial treatment disparities after machine learning surgical risk-adjustment
Hammarlund, N. E. (2021) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-020-00231-7 · Cited by 3 · see citing works
Self-citationRacial Treatment Disparities after Machine Learning Surgical Risk-AdjustmentPreprint
Hammarlund, N. E. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3057607 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [48]48. Davis, S. E., Lasko, T. A., Chen, G. & Matheny, M. E. Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality. AMIA Annu. Symp. Proc. 2017 , 625-634 (2017).
Cited by
Sorted by citations
MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
Gao, J., Rahman, M., Caskey, J., Oguss, M., O’Rourke, A., Brown, R., Stey, A., Mayampurath, A., Churpek, M. M., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02219-4 · Cited by 2 · see citing works
“In a prospective study, two research teams were deployed to screen, consent, and enroll 2096 consented patients between September 2021 and February 2024, into the Tobacco, Alcohol, Prescription Medications, and other Substance (TAPS) screening tool, recommended by the National Institute on Drug…”
Self-citation
- Authors
- Bergquist, T., Schaffter, T., Yan, Y., Yu, T., Prosser, J., Gao, J., Chen, G., Charzewski, Ł., Nawalany, Z., Brugere, I., Retkute, R., Prusokiene, A., Prusokas, A., Choi, Y., Lee, S., Choe, J., Lee, I., Kim, S., Kang, J., Mooney, S. D., Guinney, J., the Patient Mortality Prediction DREAM Challenge Consortium, Lee, A., Salehzadeh-Yazdi, A., Prusokas, A., Basu, A., Belouali, A., Becker, A.-K., Israel, A., Prusokas, A., Winter, B., Moreno, C. V., Kurz, C., Waltemath, D., Schweinoch, D., Glaab, E., Luo, G., Chen, G., Zacharias, H. U., Qiao, H., Lee, I., Brugere, I., Kang, J., Gao, J., Truthmann, J., Choe, J., Stephens, K. A., Kaderali, L., Varshney, L. R., Vollmer, M., Pandi, M.-T., Gunn, M. L., Yetisgen, M., Nath, N., Hammarlund, N., Müller-Stricker, O., Togias, P., Heagerty, P. J., Muir, P., Banda, P., Retkute, R., Henkel, R., Madgi, S., Gupta, S., Lee, S., Mooney, S., Kannattikuni, S., Sarhadi, S., Omar, S., Wang, S., Ghosh, S., Neumann, S., Simm, S., Madhavan, S., Kim, S., Von Yu, T., Satagopam, V., Pejaver, V., Gupta, Y., Choi, Y., Nawalany, Z., Charzewski, Ł., Lee, A., Salehzadeh-Yazdi, A., Prusokas, A., Basu, A., Belouali, A., Becker, A.-K., Israel, A., Prusokas, A., Winter, B., Moreno, C. V., Kurz, C., Waltemath, D., Schweinoch, D., Glaab, E., Luo, G., Chen, G., Zacharias, H. U., Qiao, H., Lee, I., Brugere, I., Kang, J., Gao, J., Truthmann, J., Choe, J., Stephens, K. A., Kaderali, L., Varshney, L. R., Vollmer, M., Pandi, M.-T., Gunn, M. L., Yetisgen, M., Nath, N., Hammarlund, N., Müller-Stricker, O., Togias, P., Heagerty, P. J., Muir, P., Banda, P., Retkute, R., Henkel, R., Madgi, S., Gupta, S., Lee, S., Mooney, S., Kannattikuni, S., Sarhadi, S., Omar, S., Wang, S., Ghosh, S., Neumann, S., Simm, S., Madhavan, S., Kim, S., Von Yu, T., Satagopam, V., Pejaver, V., Gupta, Y., Choi, Y., Nawalany, Z., Charzewski, Ł., Lee, A., Salehzadeh-Yazdi, A., Prusokas, A., Basu, A., Belouali, A., Becker, A.-K., Israel, A., Prusokas, A., Winter, B., Moreno, C. V., Kurz, C., Waltemath, D., Schweinoch, D., Glaab, E., Luo, G., Chen, G., Zacharias, H. U., Qiao, H., Lee, I., Brugere, I., Kang, J., Gao, J., Truthmann, J., Choe, J., Stephens, K. A., Kaderali, L., Varshney, L. R., Vollmer, M., Pandi, M.-T., Gunn, M. L., Yetisgen, M., Nath, N., Hammarlund, N., Müller-Stricker, O., Togias, P., Heagerty, P. J., Muir, P., Banda, P., Retkute, R., Henkel, R., Madgi, S., Gupta, S., Lee, S., Mooney, S., Kannattikuni, S., Sarhadi, S., Omar, S., Wang, S., Ghosh, S., Neumann, S., Simm, S., Madhavan, S., Kim, S., Von Yu, T., Satagopam, V., Pejaver, V., Gupta, Y., Choi, Y., Nawalany, Z., Charzewski, Ł., Lee, A., Salehzadeh-Yazdi, A., Prusokas, A., Basu, A., Belouali, A., Becker, A.-K., Israel, A., Prusokas, A., Winter, B., Moreno, C. V., Kurz, C., Waltemath, D., Schweinoch, D., Glaab, E., Luo, G., Chen, G., Zacharias, H. U., Qiao, H., Lee, I., Brugere, I., Kang, J., Gao, J., Truthmann, J., Choe, J., Stephens, K. A., Kaderali, L., Varshney, L. R., Vollmer, M., Pandi, M.-T., Gunn, M. L., Yetisgen, M., Nath, N., Hammarlund, N., Müller-Stricker, O., Togias, P., Heagerty, P. J., Muir, P., Banda, P., Retkute, R., Henkel, R., Madgi, S., Gupta, S., Lee, S., Mooney, S., Kannattikuni, S., Sarhadi, S., Omar, S., Wang, S., Ghosh, S., Neumann, S., Simm, S., Madhavan, S., Kim, S., Von Yu, T., Satagopam, V., Pejaver, V., Gupta, Y., Choi, Y., Nawalany, Z., Charzewski, Ł., Lee, A., Salehzadeh-Yazdi, A., Prusokas, A., Basu, A., Belouali, A., Becker, A.-K., Israel, A., Prusokas, A., Winter, B., Moreno, C. V., Kurz, C., Waltemath, D., Schweinoch, D., Glaab, E., Luo, G., Chen, G., Zacharias, H. U., Qiao, H., Lee, I., Brugere, I., Kang, J., Gao, J., Truthmann, J., Choe, J., Stephens, K. A., Kaderali, L., Varshney, L. R., Vollmer, M., Pandi, M.-T., Gunn, M. L., Yetisgen, M., Nath, N., Hammarlund, N., Müller-Stricker, O., Togias, P., Heagerty, P. J., Muir, P., Banda, P., Retkute, R., Henkel, R., Madgi, S., Gupta, S., Lee, S., Mooney, S., Kannattikuni, S., Sarhadi, S., Omar, S., Wang, S., Ghosh, S., Neumann, S., Simm, S., Madhavan, S., Kim, S., Von Yu, T., Satagopam, V., Pejaver, V., Gupta, Y., Choi, Y., Nawalany, Z., Charzewski, Ł., Lee, A., Salehzadeh-Yazdi, A., Prusokas, A., Basu, A., Belouali, A., Becker, A.-K., Israel, A., Prusokas, A., Winter, B., Moreno, C. V., Kurz, C., Waltemath, D., Schweinoch, D., Glaab, E., Luo, G., Chen, G., Zacharias, H. U., Qiao, H., Lee, I., Brugere, I., Kang, J., Gao, J., Truthmann, J., Choe, J., Stephens, K. A., Kaderali, L., Varshney, L. R., Vollmer, M., Pandi, M.-T., Gunn, M. L., Yetisgen, M., Nath, N., Hammarlund, N., Müller-Stricker, O., Togias, P., Heagerty, P. J., Muir, P., Banda, P., Retkute, R., Henkel, R., Madgi, S., Gupta, S., Lee, S., Mooney, S., Kannattikuni, S., Sarhadi, S., Omar, S., Wang, S., Ghosh, S., Neumann, S., Simm, S., Madhavan, S., Kim, S., Von Yu, T., Satagopam, V., Pejaver, V., Gupta, Y., Choi, Y., Nawalany, Z., & Charzewski, Ł.
- Affiliations
- Sage Bionetworks (US)
- Sage Bionetworks , Seattle, WA, United States
- University of Washington (US)
- Newcastle University (GB)
- Department of Biomedical Informatics and Medical Education, University of Washington , Seattle, WA, United States
- Corresponding author
- Guinney, Justin
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-08-21
- Volume
- 31
- Issue
- 1
- Pages
- 35-44
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10746301
- OpenAlex
- W4386025889
- Semantic Scholar
- e2fdd998f9d46bf63ab4c722febe0ca3422d20e9
- Cited by
- 8
- References
- 32
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 1 · 2022: 2 · 2021: 1
- FWCI
- 0.47
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Open Repository and Bibliography (University of Luxembourg) (submittedVersion, other-oa)
- Funders
- Clinical and Translational Science Awards Program
- ational Center for Data to Health
- National Center for Advancing Translational Sciences (U24TR002306, UL1 TR002319)
- National Institutes of Health (U24TR002306, UL1 TR002319)
- Institute for Translational Health Sciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- all-cause mortality prediction, community problems, prospective validation
- MeSH terms
- Machine Learning, Algorithms, Artificial Intelligence, Humans, Medicine, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
