An empirical characterization of fair machine learning for clinical risk prediction
Open accessAbstract
The use of machine learning to guide clinical decision making has the potential to worsen existing health disparities. Several recent works frame the problem as that of algorithmic fairness, a framework that has attracted considerable attention and criticism. However, the appropriateness of this framework is unclear due to both ethical as well as technical considerations, the latter of which include trade-offs between measures of fairness and model performance that are not well-understood for predictive models of clinical outcomes. To inform the ongoing debate, we conduct an empirical study to characterize the impact of penalizing group fairness violations on an array of measures of model performance and group fairness. We repeat the analysis across multiple observational healthcare databases, clinical outcomes, and sensitive attributes. We find that procedures that penalize differences between the distributions of predictions across groups induce nearly-universal degradation of multiple performance metrics within groups. On examining the secondary impact of these procedures, we observe heterogeneity of the effect of these procedures on measures of fairness in calibration and ranking across experimental conditions. Beyond the reported trade-offs, we emphasize that analyses of algorithmic fairness in healthcare lack the contextual grounding and causal awareness necessary to reason about the mechanisms that lead to health disparities, as well as about the potential of algorithmic fairness methods to counteract those mechanisms. In light of these limitations, we encourage researchers building predictive models for clinical use to step outside the algorithmic fairness frame and engage critically with the broader sociotechnical context surrounding the use of machine learning in healthcare.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PD-ai models use unstructured information such as gender, sex, ethnicity, BMI, medical history, and medical prescription as input for outcome prediction.
“Unstructured information such as gender, sex, ethnicity, body mass index (BMI), medical history, and medical prescription is utilized as the input for outcome prediction in PD-ai models [92,111].”
From Artificial intelligence bias in medical system designs: a systematic review (2023) · cites this paper as [111]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
Crenshaw, K. (2018) · Feminist Legal Theory · doi:10.4324/9780429500480-5 · Cited by 9,510 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
Practical Methods of Optimization
Fletcher, R. (2000) · Wiley eBooks · doi:10.1002/9781118723203 · Cited by 7,649 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 3× · see citing works
“As a result, algorithmic fairness criteria may be misleading in several situations, including when the observed outcome is a biased surrogate for the construct of interest [3, 49] and when predictive models are not appropriately contextualized in terms of the heterogeneous and dynamic impact of the complex interventio…”
BackgroundCrenshaw, K. W. (1998) · Oxford University Press eBooks · doi:10.1093/oso/9780198782063.003.0016 · Cited by 5,707 · see citing works
Structural racism and health inequities in the USA: evidence and interventions
Bailey, Z. D., Krieger, N., Agénor, M., Graves, J., Linos, N., & Bassett, M. T. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30569-x · Cited by 5,203 · see citing works
(2012) · MPG.PuRe (Max Planck Society) · doi:10.5555/2188385.2188410 · Cited by 3,603 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Racial Residential Segregation: A Fundamental Cause of Racial Disparities in Health
Williams, D. R., & Collins, C. (2001) · Public Health Reports · doi:10.1093/phr/116.5.404 · Cited by 3,065 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2× · see citing works
“[40] M.”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Hall, W. J., Chapman, M. V., Lee, K. M., Merino, Y. M., Thomas, T. W., Payne, B. K., Eng, E., Day, S. H., & Coyne-Beasley, T. (2015) · American Journal of Public Health · doi:10.2105/ajph.2015.302903 · Cited by 2,176 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics 2016
Murray, I., & Graham, M. (2016) · Cited by 2,079 · see citing works
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951 · see citing works
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · see citing works
Self-citationHidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms
Vyas, D. A., Eisenstein, L. G., & Jones, D. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmms2004740 · Cited by 1,731 · see citing works
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · see citing works
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581 · see citing works
Proceedings of the 2nd International Conference on Neural Information Processing Systems
Touretzky, D. S. (1989) · Cited by 1,413 · see citing works
Ensuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341 · see citing works
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149 · see citing works
(2013) · Cited by 1,126 · see citing works
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,107 · cited 1× · see citing works
“[80] George Hripcsak, Jon D Duke, Nigam H Shah, Christian G Reich, Vojtech Huser, Martijn J Schuemie, Marc A Suchard, Rae Woong Park, Ian Chi Kei Wong, Peter R Rijnbeek, Johan Van Der Lei, Nicole Pratt, G Niklas Norén, Yu-Chuan Chuan Li, Paul E Stang, David Madigan, and Patrick B Ryan.”
BackgroundKusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · see citing works
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · see citing works
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941 · see citing works
Addressing bias in artificial intelligence in health care
Parikh, R. B., Teeple, S., & Navathe, A. S. (2019) · JAMA · doi:10.1001/jama.2019.18058 · Cited by 764 · see citing works
The public health critical race methodology: praxis for antiracism research.
Ford, C. L., & Airhihenbuwa, C. O. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2010.07.030 · Cited by 757 · see citing works
Good intentions are not enough: how informatics interventions can worsen inequality
Veinot, T. C., Mitchell, H., & Ancker, J. S. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy052 · Cited by 704 · see citing works
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685 · see citing works
Inequality and the health-care system in the USA
Dickman, S. L., Himmelstein, D. U., & Woolhandler, S. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30398-7 · Cited by 656 · see citing works
Fairness Constraints: Mechanisms for Fair Classification
Zafar, M. B., Valera, I., Rodriguez, M. G., & Gummadi, K. P. (2015) · arXiv · doi:10.48550/arxiv.1507.05259 · Cited by 645 · see citing works
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · see citing works
VanderWeele, T. J., & Robinson, W. R. (2014) · Epidemiology · doi:10.1097/ede.0000000000000105 · Cited by 567 · see citing works
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553 · see citing works
Validation of a common data model for active safety surveillance research
Overhage, J. M., Ryan, P. B., Reich, C. G., Hartzema, A. G., & Stang, P. E. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000376 · Cited by 548 · see citing works
Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535 · see citing works
On Fairness and CalibrationPreprint
Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J. M., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02012 · Cited by 509 · see citing works
Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
Mireille Hildebrandt, C. C., & del Castillo, C. C. (2020) · doi:10.1145/3351095 · Cited by 486 · see citing works
On Wasserstein Two-Sample Testing and Related Families of Nonparametric Tests
Ramdas, A., Trillos, N., & Cuturi, M. (2017) · Entropy · doi:10.3390/e19020047 · Cited by 471 · see citing works
Building Classifiers with Independency Constraints
Calders, T., Kamiran, F., & Pechenizkiy, M. (2009) · 2009 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2009.83 · Cited by 468 · see citing works
Why ethnic minority groups are under-represented in clinical trials: A review of the literature
Hussain-Gambles, M., Atkin, K., & Leese, B. (2004) · Health and Social Care in the Community · doi:10.1111/j.1365-2524.2004.00507.x · Cited by 447 · see citing works
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444 · see citing works
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · see citing works
Learning Adversarially Fair and Transferable RepresentationsPreprint
Madras, D., Creager, E., Pitassi, T., & Zemel, R. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06309 · Cited by 418 · see citing works
Race and Reification in Science
Duster, T. (2005) · Science · doi:10.1126/science.1110303 · Cited by 399 · see citing works
Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 381 · cited 1× · see citing works
“Age Group Race and Ethnicity Cohort construction Model training and evaluation Hospital Mortality Prolonged Length of Stay Attribute: Sex Female Male Attribute: Race and Ethnicity Asian Black Hispanic Other White Attribute: Age Group [18-30) [30-45) [45-55) [55-65) [65-75) [75-90) 30-Day Readmission Performance Metric…”
BackgroundAssessing risk, automating racism
Benjamin, R. (2019) · Science · doi:10.1126/science.aaz3873 · Cited by 378 · see citing works
Fairness in Recommendation Ranking through Pairwise Comparisons
Beutel, A., Chen, J., Doshi, T. S., Qian, H., Wei, L., Wu, Y., Heldt, L., Zhao, Z., Hong, L., Ed, H., et al. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330745 · Cited by 374 · see citing works
The Frontiers of Fairness in Machine LearningPreprint
Chouldechova, A., & Roth, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08810 · Cited by 372 · see citing works
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362 · see citing works
Avoiding Discrimination through Causal Reasoning
Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., & Schölkopf, B. (2017) · arXiv · doi:10.48550/arxiv.1706.02744 · Cited by 358 · see citing works
When the implication is not to design (technology)
Baumer, E. P. S., & Silberman, M. ‘. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979275 · Cited by 353 · see citing works
Inherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 347 · see citing works
On the apparent conflict between individual and group fairness
Binns, R. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372864 · Cited by 312 · see citing works
Austin, P. C., & Steyerberg, E. W. (2019) · Statistics in Medicine · doi:10.1002/sim.8281 · Cited by 311 · see citing works
50 Years of Test (Un)fairness: Lessons for Machine Learning
Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287600 · Cited by 297 · see citing works
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295 · cited 3× · see citing works
“Age Group Race and Ethnicity Cohort construction Model training and evaluation Hospital Mortality Prolonged Length of Stay Attribute: Sex Female Male Attribute: Race and Ethnicity Asian Black Hispanic Other White Attribute: Age Group [18-30) [30-45) [45-55) [55-65) [65-75) [75-90) 30-Day Readmission Performance Metric…”
BackgroundMaking Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294 · see citing works
Self-citationEthical limitations of algorithmic fairness solutions in health care machine learning
McCradden, M. D., Joshi, S., Mazwi, M., & Anderson, J. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30065-0 · Cited by 291 · see citing works
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2021) · Communications of the ACM · doi:10.1145/3433949 · Cited by 262 · see citing works
Residential Segregation and Disparities in Health Care Services Utilization
Gaskin, D. J., Dinwiddie, G. Y., Chan, K. S., & McCleary, R. R. (2011) · Medical Care Research and Review · doi:10.1177/1077558711420263 · Cited by 248 · see citing works
On the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247 · see citing works
Racial Categories in Medical Practice: How Useful Are They?
Braun, L., Fausto-Sterling, A., Fullwiley, D., Hammonds, E. M., Nelson, A., Quivers, W., Reverby, S. M., & Shields, A. E. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040271 · Cited by 236 · see citing works
Delayed Impact of Fair Machine Learning
Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., & Hardt, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/862 · Cited by 236 · see citing works
Treating health disparities with artificial intelligence
Chen, I. Y., Joshi, S., & Ghassemi, M. (2020) · Nature Medicine · doi:10.1038/s41591-019-0649-2 · Cited by 235 · see citing works
Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Celis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287586 · Cited by 231 · see citing works
2010 IEEE International Conference on Data Mining Workshops
(2010) · doi:10.1109/icdmw18178.2010 · Cited by 225 · see citing works
Don’t ask if artificial intelligence is good or fair, ask how it shifts power
Kalluri, P. (2020) · Nature · doi:10.1038/d41586-020-02003-2 · Cited by 213 · see citing works
Reps, J. M., Schuemie, M. J., Suchard, M. A., Ryan, P. B., & Rijnbeek, P. R. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy032 · Cited by 205 · see citing works
"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196 · see citing works
Racial inequalities in health: Framing future research
Hicken, M. T., Kravitz‐Wirtz, N., Durkee, M. I., & Jackson, J. S. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.027 · Cited by 194 · see citing works
Why is my classifier discriminatory?
Chen, I., Johansson, F. D., & Sontag, D. (2018) · arXiv · doi:10.48550/arxiv.1805.12002 · Cited by 185 · cited 1× · see citing works
“Age Group Race and Ethnicity Cohort construction Model training and evaluation Hospital Mortality Prolonged Length of Stay Attribute: Sex Female Male Attribute: Race and Ethnicity Asian Black Hispanic Other White Attribute: Age Group [18-30) [30-45) [45-55) [55-65) [65-75) [75-90) 30-Day Readmission Performance Metric…”
BackgroundThe Variational Fair AutoencoderPreprint
Louizos, C., Swersky, K., Li, Y., Welling, M., & Zemel, R. S. (2015) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1511.00830 · Cited by 178 · see citing works
Under-representation of elderly and women in clinical trials
Vitale, C., Fini, M., Spoletini, I., Lainščak, M., Seferovic, P. M., & Rosano, G. M. C. (2017) · International Journal of Cardiology · doi:10.1016/j.ijcard.2017.01.018 · Cited by 161 · see citing works
Identifying and Correcting Label Bias in Machine Learning
Jiang, H., & Nachum, O. (2019) · arXiv · doi:10.48550/arxiv.1901.04966 · Cited by 152 · see citing works
The Racism-Race Reification Process
Sewell, A. A. (2016) · Sociology of Race and Ethnicity · doi:10.1177/2332649215626936 · Cited by 137 · see citing works
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
(2021) · doi:10.1145/3461702 · Cited by 130 · see citing works
Does mitigating ML's impact disparity require treatment disparity?Preprint
Lipton, Z. C., Chouldechova, A., & McAuley, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07076 · Cited by 125 · see citing works
Hurtful words: quantifying biases in clinical contextual word embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384448 · Cited by 122 · see citing works
Algorithmic Fairness from a Non-ideal Perspective
Fazelpour, S., & Lipton, Z. C. (2020) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3375627.3375828 · Cited by 119 · see citing works
50 Years of Test (Un)fairness: Lessons for Machine LearningPreprint
Hutchinson, B., & Mitchell, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10104 · Cited by 118 · see citing works
McCradden, M. D., Joshi, S., Anderson, J. A., Mazwi, M., Goldenberg, A., & Zlotnik Shaul, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa085 · Cited by 111 · see citing works
The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC MetricPreprint
Kallus, N., & Zhou, A. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1902.05826 · Cited by 95 · see citing works
Fairness in Precision Medicine
Ferryman, K. (2018) · doi:10.69985/tbql1779 · Cited by 94 · see citing works
Learning Controllable Fair RepresentationsPreprint
Song, J., Kalluri, P., Grover, A., Zhao, S., & Ermon, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.04218 · Cited by 94 · see citing works
A fairer way forward for AI in health care
Nordling, L. (2019) · Nature · doi:10.1038/d41586-019-02872-2 · Cited by 90 · see citing works
Residual unfairness in fair machine learning from prejudiced data
Kallus, N., & Zhou, A. (2018) · arXiv · doi:10.48550/arxiv.1806.02887 · Cited by 83 · see citing works
A framework for making predictive models useful in practice
Jung, K., Kashyap, S., Avati, A., Harman, S. M., Shaw, H., Li, R., Smith, M. A., Shum, K., Javitz, J., Vetteth, Y., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa318 · Cited by 82 · see citing works
Self-citationThe false promise of risk assessments: epistemic reform and the limits of fairness
Green, B. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372869 · Cited by 80 · see citing works
On integral probability metrics, ϕ-divergences and binary classificationPreprint
Sriperumbudur, B. K., Fukumizu, K., Gretton, A., Schölkopf, B., & Lanckriet, G. R. G. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0901.2698 · Cited by 79 · see citing works
Creating Fair Models of Atherosclerotic Cardiovascular Disease Risk
Pfohl, S. R., Marafino, B. J., Coulet, A., Rodriguez, F., Palaniappan, L., & Shah, N. (2018) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314278 · Cited by 79 · see citing works
Self-citationMetric Learning for Individual FairnessPreprint
Ilvento, C. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.forc.2020.2 · Cited by 69 · see citing works
Machine Learning, Health Disparities, and Causal Reasoning
Goodman, S. N., Goel, S., & Cullen, M. R. (2018) · Annals of Internal Medicine · doi:10.7326/m18-3297 · Cited by 57 · see citing works
Fair regression for health care spending
Zink, A., & Rose, S. (2019) · Biometrics · doi:10.1111/biom.13206 · Cited by 57 · see citing works
Cotter, A., Jiang, H., Wang, S., Narayan, T., Gupta, M. R., You, S., & Sridharan, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04198 · Cited by 57 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2011) · arXiv · doi:10.48550/arxiv.1104.3913 · Cited by 55 · see citing works
Proceedings of the ACM Conference on Health, Inference, and Learning
(2020) · doi:10.1145/3368555 · Cited by 53 · see citing works
A new paradigm for accelerating clinical data science at Stanford MedicinePreprint
Datta, S., Posada, J., Olson, G., Li, W., O’Reilly, C., Balraj, D., Mesterhazy, J., Pallas, J. I., Desai, P., & Shah, N. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10534 · Cited by 50 · see citing works
Self-citationMIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIPreprint
Wang, S., McDermott, M. B. A., Chauhan, G., Hughes, M. C., Naumann, T., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08322 · Cited by 47 · see citing works
Calibration for the (Computationally-Identifiable) MassesPreprint
Hébert-Johnson, Ú., Kim, M. P., Reingold, O., & Rothblum, G. N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08513 · Cited by 44 · see citing works
Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · arXiv · doi:10.48550/arxiv.2003.11515 · Cited by 37 · see citing works
Sun, C., Asudeh, A., Jagadish, H. V., Howe, B., & Stoyanovich, J. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357853 · Cited by 35 · see citing works
"The Human Body is a Black Box": Supporting Clinical Decision-Making with Deep LearningPreprint
Sendak, M., Elish, M. C., Gao, M. C., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O’Brien, C. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08089 · Cited by 35 · see citing works
The implicit fairness criterion of unconstrained learningPreprint
Liu, L. T., Simchowitz, M., & Hardt, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.10013 · Cited by 32 · see citing works
Classification with Fairness Constraints: A Meta-Algorithm with Provable GuaranteesPreprint
Celis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06055 · Cited by 30 · see citing works
Counterfactual Reasoning for Fair Clinical Risk PredictionPreprint
Pfohl, S., Duan, T., Ding, D. Y., & Shah, N. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06260 · Cited by 30 · see citing works
Self-citationMartin, D., Prabhakaran, V., Kuhlberg, J., Smart, A., & Isaac, W. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07572 · Cited by 28 · see citing works
Causal Modeling for Fairness in Dynamical SystemsPreprint
Creager, E., Madras, D., Pitassi, T., & Zemel, R. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09141 · Cited by 28 · see citing works
Magnetoconductance responses in organic charge-transfer-complex molecules
Lee, T., Li, J.-H., Huang, W.-S., Hu, B., Huang, J. C. A., Guo, T., & Wen, T. (2011) · Applied Physics Letters · doi:10.1063/1.3627170 · Cited by 25 · see citing works
Fairness and Transparency in the Age of the Algorithm
Olhede, S. C., & Rodrigues, R. (2017) · Significance · doi:10.1111/j.1740-9713.2017.01012.x · Cited by 24 · see citing works
Measuring Fairness in an Unfair World
Herington, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375854 · Cited by 22 · see citing works
The Frontiers of Machine Learning: 2017 Raymond and Beverly Sackler U.S.-U.K. Scientific Forum
National Academy of Sciences, & The Royal Society (2018) · doi:10.17226/25021 · Cited by 17 · see citing works
Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent ConstraintsPreprint
Cotter, A., Gupta, M. R., Jiang, H., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00028 · Cited by 15 · see citing works
When Your Only Tool Is A Hammer
McCradden, M. D., Mazwi, M., Joshi, S. D., & Anderson, J. A. (2020) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3375627.3375824 · Cited by 14 · see citing works
Martin, D., Prabhakaran, V., Kuhlberg, J., Smart, A., & Isaac, W. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09663 · Cited by 14 · see citing works
MIMIC-ExtractPreprint
Wang, S., McDermott, M. B. A., Chauhan, G., Hughes, M. C., Naumann, T., & Ghassemi, M. (2020) · arXiv (Cornell University) · Cited by 12 · see citing works
Fair Predictors under Distribution ShiftPreprint
Singh, H., Singh, R., Mhasawade, V., & Chunara, R. (2019) · arXiv (Cornell University) · Cited by 6 · see citing works
Understanding racial bias in health using the Medical Expenditure Panel Survey dataPreprint
Singh, M., & Ramamurthy, K. N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01509 · Cited by 6 · see citing works
Noise Induces Loss Discrepancy Across Groups for Linear Regression.Preprint
Khani, F., & Liang, P. (2019) · arXiv (Cornell University) · Cited by 5 · see citing works
A framework for making predictive models useful in practicePreprint
Jung, K., Kashyap, S., Avati, A., et al. (2020) · medRxiv · doi:10.1101/2020.07.10.20149419 · Cited by 5 · see citing works
A Calibration Metric for Risk Scores with Survival Data
Yadlowsky, S., Basu, S., & Tian, L. (2019) · Machine Learning for Healthcare Conference · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [38]Agarwal A, Beygelzimer A, Dudik M, Langford J, Wallach H, A reductions approach to fair classification, in: International Conference on Machine Learning, 2018, pp. 60– 69.
Cited by
Sorted by citations
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750 · see citing works
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358 · see citing works
Timmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221134490 · Cited by 263 · see citing works
Trustworthy Augmented Intelligence in Health Care
Crigger, E., Reinbold, K., Hanson, C., Kao, A., Blake, K., & Irons, M. (2022) · Journal of Medical Systems · doi:10.1007/s10916-021-01790-z · Cited by 132 · see citing works
A toolbox for surfacing health equity harms and biases in large language models
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03258-2 · Cited by 113 · see citing works
Self-citationGuo, L. L., Pfohl, S. R., Fries, J., Johnson, A. E. W., Posada, J., Aftandilian, C., Shah, N., & Sung, L. (2022) · Scientific Reports · doi:10.1038/s41598-022-06484-1 · Cited by 103 · see citing works
Self-citationArtificial Intelligence in Cardiovascular Care—Part 2: Applications
Jain, S. S., Elias, P., Poterucha, T., Randazzo, M., Lopez Jimenez, F., Khera, R., Perez, M., Ouyang, D., Pirruccello, J., Salerno, M., et al. (2024) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2024.03.401 · Cited by 74 · see citing works
The AI life cycle: a holistic approach to creating ethical AI for health decisions
Ng, M. Y., Kapur, S., Blizinsky, K. D., & Hernandez-Boussard, T. (2022) · Nature Medicine · doi:10.1038/s41591-022-01993-y · Cited by 71 · see citing works
Artificial intelligence bias in medical system designs: a systematic review
Kumar, A., Aelgani, V., Vohra, R., Gupta, S. K., Bhagawati, M., Paul, S., Saba, L., Suri, N., Khanna, N. N., Laird, J. R., et al. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16029-x · Cited by 65 · see citing works
Algorithmic fairness of machine learning models for Alzheimer disease progression
Yuan, C., Linn, K. A., & Hubbard, R. A. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.42203 · Cited by 49 · see citing works
Mccradden, M., Odusi, O., Joshi, S., Akrout, I., Ndlovu, K., Glocker, B., Maicas, G., Liu, X., Mazwi, M., Garnett, T., et al. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594096 · Cited by 33 · see citing works
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33 · see citing works
Pfohl, S., Xu, Y., Foryciarz, A., Ignatiadis, N., Genkins, J., & Shah, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533166 · Cited by 30 · see citing works
Self-citation
- Authors
- Pfohl, S. R., Foryciarz, A., & Shah, N. H.
- Affiliations
- Stanford University (US)
- Corresponding author
- Pfohl, Stephen R.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2021-01
- Volume
- 113
- Pages
- 103621
- ISSN
- 1532-0464 (ISSN-L)
- PMC
- PMC7871979
- OpenAlex
- W3043023066
- Cited by
- 164
- References
- 133
- Citations by year
- 2026: 39 · 2025: 35 · 2024: 24 · 2023: 25 · 2022: 31 · 2021: 8 · 2020: 2
- FWCI
- 22.70
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- arXiv (Cornell University) (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-1656518)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- algorithmic fairness, clinical risk prediction, health disparities, fairness-performance trade-off, ranking fairness, sensitive attributes, clinical decision support
- MeSH terms
- Machine Learning, Delivery of Health Care, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
