Peeking into a black box, the fairness and generalizability of a MIMIC-III benchmarking model.
Open accessThe DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
As artificial intelligence (AI) makes continuous progress to improve quality of care for some patients by leveraging ever increasing amounts of digital health data, others are left behind. Empirical evaluation studies are required to keep biased AI models from reinforcing systemic health disparities faced by minority populations through dangerous feedback loops. The aim of this study is to raise broad awareness of the pervasive challenges around bias and fairness in risk prediction models. We performed a case study on a MIMIC-trained benchmarking model using a broadly applicable fairness and generalizability assessment framework. While open-science benchmarks are crucial to overcome many study limitations today, this case study revealed a strong class imbalance problem as well as fairness concerns for Black and publicly insured ICU patients. Therefore, we advocate for the widespread use of comprehensive fairness and performance assessment frameworks to effectively monitor and validate benchmark pipelines built on open data resources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The researchers found that imbalanced representation of racial groups had a significant effect on model performance (yielding recall rates as low as 25%), raising critical concerns for the model’s potential application in real clinical settings [44].”
From Bias in medical AI: Implications for clinical decision-making (2024) · cites this paper as [44]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“These results build on Obermeyer’s previous work and strengthen the case that Black patients and other systemically disadvantaged groups are the ones left behind by AI revolutionizing standards of care 9 .”
ResultThe relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Quan, H., Li, B., Couris, C. M., Fushimi, K., Graham, P., Hider, P., Januel, J.-M., & Sundararajan, V. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwq433 · Cited by 5,843
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · cited 7×
“Observed performance differences can therefore only be attributed to unfair model predictions in two scenarios such that the effect of class imbalance can be excluded: (1) the demographic groups have similar event rates or (2) their event rates are the converse of what the metric would imply (i.”
BackgroundMethodInfluentialImplicit bias in healthcare professionals: a systematic review
FitzGerald, C., & Hurst, S. (2017) · BMC Medical Ethics · doi:10.1186/s12910-017-0179-8 · Cited by 2,556
D'Ignazio, C., & Klein, L. F. (2020) · doi:10.7551/mitpress/11805.001.0001 · Cited by 1,738
Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms
Vyas, D. A., Eisenstein, L. G., & Jones, D. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmms2004740 · Cited by 1,731
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395
Ensuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152 · cited 1×
“These tools approach and sometimes surpass expert clinicians in certain specialties 2 .”
BackgroundMultitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958 · cited 9×
“More details about the LSTM model architecture, the data preprocessing and the parameters of the training procedure can be found in Harutyunyan’s publication 25 .”
BackgroundMethodInfluentialArtificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956 · cited 1×
“In the absence of tight controls, AI could hence unconsciously reinforce pre-existing biases through dangerous feedback loops 15,16 .”
BackgroundAI can be sexist and racist — it’s time to make it fair
Zou, J., & Schiebinger, L. (2018) · Nature · doi:10.1038/d41586-018-05707-8 · Cited by 765 · cited 2×
“However, emerging evidence suggests that many of these data-driven clinical decision support tools may be biased and not equally benefit all populations 3–6 .”
BackgroundEthical Machine Learning in Healthcare
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092820-114757 · Cited by 645
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Hernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., & Shah, N. H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa088 · Cited by 410
Self-citationCan AI Help Reduce Disparities in General Medical and Mental Health Care?
(2019) · AMA Journal of Ethics · doi:10.1001/amajethics.2019.167 · Cited by 401
STRIDE--An integrated standards-based translational research informatics platform.
Lowe, H., Ferris, T., Hernandez, P. M., & Weber, S. (2009) · PubMed · Cited by 397
Ozenne, B., Subtil, F., & Maucort-Boulch, D. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.02.010 · Cited by 379
Lee, P., Le Saux, M., Siegel, R., Goyal, M., Chen, C., Ma, Y., & Meltzer, A. C. (2019) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2019.06.014 · Cited by 317
Singhal, A., Tien, Y.-Y., & Hsia, R. Y. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159224 · Cited by 305
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294 · cited 2×
“…and class imbalance, such as exhibited by this benchmark model, is the fact that such intrinsic modeling problems frequently get hidden by inadequate and too simplistic sum-mary performance metrics such as accuracy, or to a lesser degree even in current reporting standards such as AUROC 36–38 .”
BackgroundEthical limitations of algorithmic fairness solutions in health care machine learning
McCradden, M. D., Joshi, S., Mazwi, M., & Anderson, J. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30065-0 · Cited by 291
Paulus, J. K., & Kent, D. M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0304-9 · Cited by 274
Valverde-Albacete, F. J., & Peláez-Moreno, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084217 · Cited by 212
Why is my classifier discriminatory?
Chen, I., Johansson, F. D., & Sontag, D. (2018) · arXiv · doi:10.48550/arxiv.1805.12002 · Cited by 185
Interpretability and fairness evaluation of deep learning models on MIMIC-IV dataset
Meng, C., Trinh, L., Xu, N., Enouen, J., & Liu, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-11012-2 · Cited by 174
A Path for Translation of Machine Learning Products into Healthcare Delivery
(2020) · EMJ Innovations · doi:10.33590/emjinnov/19-00172 · Cited by 172
Johnson, A., Pollard, T., & Mark, R. (2015) · PhysioNet · doi:10.13026/c2xw26 · Cited by 165
Johnson, A. E. W., Pollard, T., & Mark, R. G. (2015) · PhysioNet · doi:10.13026/cd7z-wg25 · Cited by 161
On the necessity and design of studies comparing statistical methods
Boulesteix, A., Binder, H., Abrahamowicz, M., Sauerbrei, W., & Initiative, F. T. S. P. O. T. S. (2017) · Biometrical Journal · doi:10.1002/bimj.201700129 · Cited by 116
Fairness in Precision Medicine
Ferryman, K. (2018) · doi:10.69985/tbql1779 · Cited by 94
The Quality Of Care Delivered To Patients Within The Same Hospital Varies By Insurance Type
Spencer, C. S., Gaskin, D. J., & Roberts, E. T. (2013) · Health Affairs · doi:10.1377/hlthaff.2012.1400 · Cited by 87 · cited 1×
“This disparity in comorbidity burden points to a more systemic problem in our healthcare system, where socioeconomically disadvantaged patients receive unequal quality of care, contributing to many preventable deaths 13,14,41,42 .”
BackgroundInsurance status and hospital care for myocardial infarction, stroke, and pneumonia
Hasan, O., Orav, E. J., & Hicks, L. S. (2010) · Journal of Hospital Medicine · doi:10.1002/jhm.687 · Cited by 86
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2019) · arXiv · doi:10.48550/arxiv.1901.10002 · Cited by 82
Bias at warp speed: how AI may contribute to the disparities gap in the time of COVID-19
Röösli, E., Rice, B., & Hernandez-Boussard, T. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa210 · Cited by 76
Self-citationPetersen, C., Smith, J., Freimuth, R. R., Goodman, K. W., Jackson, G. P., Kannry, J., Liu, H., Madhavan, S., Sittig, D. F., & Wright, A. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa319 · Cited by 65
Improving Hospital Mortality Prediction with Medical Named Entities and Multimodal Learning
Jin, M., Bahadori, M. T., Colak, A., Bhatia, P., Celikkaya, B., Bhakta, R., Senthivel, S., Khalilia, M. A., Navarro, D., Zhang, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12276 · Cited by 56
Automated EHR score to predict COVID-19 outcomes at US Department of Veterans Affairs
Osborne, T. F., Veigulis, Z. P., Arreola, D. M., Röösli, E., & Curtin, C. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236554 · Cited by 45
Self-citationLearning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks
Oh, J., Wang, J., & Wiens, J. (2018) · PubMed · doi:10.48550/arxiv.1808.06725 · Cited by 42
Using Predictive Analytics to Guide Patient Care and Research in a National Health System
Nelson, K. M., Chang, E. T., Zulman, D. M., Rubenstein, L. V., Kirkland, F. D., & Fihn, S. D. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-04961-4 · Cited by 40
MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset
Meng, C., Trinh, L., Xu, N., & Liu, Y. (2021) · doi:10.21203/rs.3.rs-402058/v1 · Cited by 21
Learning Representations of Missing Data for Predicting Patient OutcomesPreprint
Malone, B., García-Durán, A., & Niepert, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04752 · Cited by 16
YerevaNN/mimic3-benchmarks: MIMIC3-Benchmarks v1.0.0-alpha
Harutyunyan, H., Kale, D., Khachatrian, H., et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1306527 · Cited by 2
YerevaNN/mimic3-benchmarks: MIMIC3-Benchmarks v1.0.0-alpha
Harutyunyan, H., Kale, D., Khachatrian, H., Natny, Badger, T. G., S., & Romanov, A. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1306526 · Cited by 2
The Fairness and Generalizability Assessment Framework
Röösli, E., Bozkurt, S., & Hernandez-Boussard, T. (2021) · Stanford Digital Repository · doi:10.25740/tb877wd0973 · Cited by 1
Self-citationHosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9<965::aid-sim509>3.0.co;2-o
Research works printed but not linked: 4
- Gupta, P., Malhotra, P., Vig, L. & Shrof, G. Using features from pre-trained timenet for clinical predictions. In Proceedings of the 3rd International Workshop on Knowledge Discovery in Healthcare Data at IJCAI-ECAI, 38–44 (Stockholm, Sweden). (2018).
- Gupta, P., Malhotra, P., Vig, L. & Shrof, G. Transfer learning for clinical time series analysis using recurrent neural networks. In Machine Learning for Medicine and Healthcare Workshop at ACM KDD 2018 Conference (London, United Kingdom). (2018).
- Chang, C.-H., Mai, M. & Goldenberg, A. Dynamic measurement scheduling for adverse event forecasting using deep RL. In Machine Learning for Health (ML4H) Workshop at NeurIPS (Montreal, Canada). (2018).
- Ali, A., Shamsuddin, S. & Ralescu, A. Classification with class imbalance problem: A review. SOCO. (2015).
Cited by
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
Bias in medical AI: Implications for clinical decision-making
Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000651 · Cited by 607
“Another noteworthy study aimed to systematically quantify the effect of class imbalance on the fairness and generalizability of an AI model across different patient groups [44].”
BackgroundMethodChen, F., Wang, L., Hong, J., Jiang, J., & Zhou, L. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae060 · Cited by 178
Sanmarchi, F., Fanconi, C., Golinelli, D., Gori, D., Hernandez‐Boussard, T., & Capodici, A. (2023) · Journal of Nephrology · doi:10.1007/s40620-023-01573-4 · Cited by 176
Self-citationDemographic bias in misdiagnosis by computational pathology models
Vaidya, A. J., Chen, R. J., Williamson, D. F. K., Song, A. H., Jaume, G., Yang, Y., Hartvigsen, T., Dyer, E. C., Lu, M. Y., Lipková, J., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02885-z · Cited by 177
Metadata
- Authors
- Röösli, E., Bozkurt, S., & Hernandez-Boussard, T.
- Affiliations
- Stanford Medicine (US)
- Venue
- Scientific Data
- Published
- 2022-01-24
- Volume
- 9
- Issue
- 1
- Pages
- 24-24
- ISSN
- 2052-4463 (ISSN-L)
- PMC
- PMC8786878
- OpenAlex
- W4207057807
- Cited by
- 73
- References
- 45
- Citations by year
- 2026: 13 · 2025: 13 · 2024: 24 · 2023: 18 · 2022: 5
- Type
- data paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Data (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (R01 LM013362-03)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- algorithmic fairness, risk prediction model, MIMIC-III, benchmarking model, generalizability, health disparities, class imbalance, Black patients, publicly insured patients, intensive care unit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.