Ensuring fairness in machine learning to advance health equity.
Open accessAbstract
Machine learning is used increasingly in clinical care to improve diagnosis, treatment selection, and health system efficiency. Because machine-learning models learn from historically collected data, populations that have experienced human and structural biases in the past-called protected groups-are vulnerable to harm by incorrect predictions or withholding of resources. This article describes how model design, biases in data, and the interactions of model predictions with clinicians and patients may exacerbate health care disparities. Rather than simply guarding against these harms passively, machine-learning systems should be used proactively to advance health equity. For that goal to be achieved, principles of distributive justice must be incorporated into model design, deployment, and evaluation. The article describes several technical implementations of distributive justice-specifically those that ensure equality in patient outcomes, performance, and resource allocation-and guides clinicians as to when they should prioritize each principle. Machine learning is providing increasingly sophisticated decision support and population-level monitoring, and it should encode principles of justice to ensure that models benefit all patients.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transparent decision-making processes in healthcare AI, with patients and providers understanding the rationale behind AI-driven recommendations, foster trust and accountability.
“Transparent decision-making processes are vital in healthcare AI. Patients and healthcare providers must understand the rationale behind AI-driven recommendations, fostering trust and accountability.[23]”
From Ethical implications of AI and robotics in healthcare: A review (2023) · cites this paper as [23]
The “What Is Machine Learning?” section says complex models can be hard to understand and trust, supporting a link between understandability and trust. The Abstract recommends involving stakeholders in model design. Neither passage says patients and providers must understand the rationale for individual AI recommendations or that doing so fosters accountability (pages not provided in our copy). (LLM, conf=0.82)
Experimental — accuracy not yet measuredParticipatory approaches play a vital role in bridging the gaps between model development and deployment and in promoting fairness in ML applications.
“Participatory approaches play a vital role in bridging the gaps between model development and deployment and in promoting fairness in ML applications (Rajkomar et al., 2018).”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Rajkomar et al., 2018
SDoH variables reflect structural inequities, and model outputs using them could inadvertently reinforce disparities if misapplied.
“Because SDoH variables reflect structural inequities, there is a risk that model outputs could inadvertently reinforce disparities if misapplied [91,92].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [91]
How principles of algorithmic fairness might apply in clinical decision-making has received little attention in the medical literature.
“how principles of algorithmic fairness might apply in clinical decision-making has received little attention in the medical literature7,8.”
From Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities (2020) · cites this paper as [7]
One of the main challenges to develop trustworthy AI is to define the meaning of fairness in machine learning practice.
“One of the main challenges to develop trustworthy AI is to define the meaning of fairness in the practice of machine learning111.”
From Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare (2020) · cites this paper as [111]
AI solutions have led to inadvertent bias of modeling results, with increasing concerns and related examples across diverse populations.
“As AI becomes more pervasive in both public and personal health across diverse populations, there have been increasing concerns, and related examples, of AI solutions leading to inadvertent bias of modeling results (45–48).”
From Accessing Artificial Intelligence for Clinical Decision-Making (2021) · cites this paper as [46]
Biases can be found in AI model design, training data, interactions with clinicians, and interactions with patients.
“In general, biases can be found in AI model design (e.g.label bias and cohort bias), training data (e.g.minority bias, missing data bias), interactions with clinicians (e.g.automation bias and feedback loops bias), and interactions with patients (e.g.privilege bias and agency bias)”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Rajkomar et al., 2018
Some potential solutions have been developed, such as exploring local datasets or local checklists for training AI, and conducting validation checks to mitigate label choice bias, but this challenge seems to persist.
“Some potential solutions have been developed, such as exploring local datasets or local checklists for training AI (Rajkomar et al., 2018(Rajkomar et al., , 2019;;Vollmer et al., 2020), and conducting validation checks to mitigate label choice bias (Char et al., 2020;Obermeyer et al., 2019), but this challenge seems to persist.”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Rajkomar et al., 2018
Biases in AI can emerge from two main sources: the data used for algorithm training (data bias) and inherent design or learning mechanisms of the algorithm itself (algorithmic bias).
“Generally, biases in AI can emerge from two main sources: the data used for algorithm training (data bias) and inherent design or learning mechanisms of the algorithm itself (algorithmic bias).”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
In the medical field, bias and discrimination in AI have been studied in various domains.
“In the medical field, bias and discrimination in AI have been studied in various domains [11, 12].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
In the healthcare context, additional biases may arise because of the complex nature of human interactions and decision-making processes, and these can be classified into biases from AI-clinician interactions and biases from AI-patient interactions.
“However, in the healthcare context, additional biases may arise because of the complex nature of human interactions and decision-making processes. These additional biases can be classified into two types: those that originate from AI–clinician interactions and those that originate from AI–patient interactions [11].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
Minority bias occurs when the number of protected group members in the dataset is insufficient for AI to learn accurate statistical patterns, which can lead to decreased performance and biased results when the algorithm is applied to these underrepresented groups.
“Minority bias occurs when the number of protected group members in the dataset is insufficient for AI to learn accurate statistical patterns. This can lead to decreased performance and biased results when the algorithm is applied to these underrepresented groups.”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
Even with representative data without data bias, AI can exhibit bias because of its inherent design or learning mechanisms.
“Even with representative data without data bias, AI can exhibit bias because of its inherent design or learning mechanisms.”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
Informativeness bias occurs when the features used for detection are not as apparent for certain protected groups, lowering their informativeness when predictions are made.
“Informativeness bias occurs when the features used for detection are not as apparent for certain protected groups, lowering their informativeness when predictions are made.”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
When healthcare professionals interact with AI, biases related to interactions can occur, potentially affecting the algorithm's performance, fairness, and adoption.
“When healthcare professionals interact with AI, biases related to interactions can occur, potentially affecting the algorithm's performance, fairness, and adoption [11].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
Allocation discrepancy occurs when the positive predictive values for protected groups are disproportionately low, leading the AI to withhold necessary resources, and such resource allocation discrepancies can exacerbate disparities in care and outcomes among the affected groups.
“Finally, an allocation discrepancy occurs when the positive predictive values for protected groups are disproportionately low, leading the AI to withhold necessary resources, such as clinical attention or social services. Such resource allocation discrepancies can exacerbate disparities in care and outcomes among the affected groups.”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
Missing data bias occurs when data from protected groups are missing nonrandomly, making it difficult for AI to generate accurate predictions.
“Missing data bias occurs when data from protected groups are missing nonrandomly, making it difficult for AI to generate accurate predictions.”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [11]
AI systems are increasingly in a position to help improve patient and population health outcomes and drive down cost, increase physician job satisfaction, and address health disparities.
“Thus, AI systems are increasingly in a position to help improve patient and population health outcomes and drive down cost, increase physician job satisfaction, and address health disparities20,21,3,22,23.”
From Considerations for addressing bias in artificial intelligence for health equity (2023) · cites this paper as [20]
If the training data used to develop ML and AI models are not representative of the broader population or if they contain historical biases, the resulting algorithms may perpetuate or even amplify these biases.
“If the training data used to develop these models are not representative of the broader population or if they contain historical biases, the resulting algorithms may perpetuate or even amplify these biases [58].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [58]
Breast cancer detection systems may perform differently based on breast density variations across ethnic groups.
“Similarly, breast cancer detection systems may perform differently based on breast density variations across ethnic groups [117,118].”
From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [117]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Capps, L. (2003) · JAMA · doi:10.1001/jama.290.18.2487-b · Cited by 4,558 · see citing works
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1× · see citing works
“For example, a recent model to predict hospital readmissions examined hundreds of thousands of pieces of information, including the free text of clinical notes (24).”
BackgroundSelf-citationBig Data and Machine Learning in Health Care
Beam, A. L., & Kohane, I. S. (2018) · JAMA · doi:10.1001/jama.2017.18391 · Cited by 2,506 · see citing works
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242 · see citing works
Self-citationFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,123 · see citing works
The Effect of Race and Sex on Physicians' Recommendations for Cardiac Catheterization
Schulman, K. A., Berlin, J. A., Harless, W., Kerner, J. F., Sistrunk, S., Gersh, B. J., Dubé, R., Taleghani, C. K., Burke, J. E., Williams, S., et al. (1999) · New England Journal of Medicine · doi:10.1056/nejm199902253400806 · Cited by 1,934 · see citing works
Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning
Poplin, R., Varadarajan, A. V., Blumer, K., Liu, Y., McConnell, M. V., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0195-0 · Cited by 1,865 · see citing works
Self-citationPotential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · see citing works
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · see citing works
Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
Digital Phenotyping: Technology for a New Science of Behavior
Insel, T. R. (2017) · JAMA · doi:10.1001/jama.2017.11295 · Cited by 833 · see citing works
Good intentions are not enough: how informatics interventions can worsen inequality
Veinot, T. C., Mitchell, H., & Ancker, J. S. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy052 · Cited by 704 · see citing works
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2007) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691 · see citing works
Gloster, H. M., & Neal, K. (2006) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2005.08.063 · Cited by 668 · see citing works
Deep Learning—A Technology With the Potential to Transform Health Care
Hinton, G. (2018) · JAMA · doi:10.1001/jama.2018.11100 · Cited by 667 · see citing works
Machine Learning and Health Care Disparities in Dermatology
Adamson, A. S., & Smith, A. (2018) · JAMA Dermatology · doi:10.1001/jamadermatol.2018.2348 · Cited by 639 · cited 1× · see citing works
“For example, if advanced-stage melanoma is diagnosed more frequently in patients with dark skin than in other groups, might a skin cancer detection model fail to detect early-stage disease in patients with dark skin (47, 48)? During training and evaluation, researchers should measure any deviations from equal accuracy…”
MethodLum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611 · see citing works
Validation of the Atherosclerotic Cardiovascular Disease Pooled Cohort Risk Equations
Muntner, P., Colantonio, L. D., Cushman, M., Goff, D. C., Howard, G., Howard, V. J., Kissela, B., Levitan, E. B., Lloyd-Jones, D. M., & Safford, M. M. (2014) · JAMA · doi:10.1001/jama.2014.2630 · Cited by 603 · see citing works
Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.034 · Cited by 564 · see citing works
Self-citationAutomation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451 · cited 1× · see citing works
“African American patients might be harmed if clinical teams started relying on alerts to identify at-risk patients without realizing that the prediction system underdetects patients in that group (automation bias) (16).”
BackgroundDrew, B. J., Harris, P., Zègre-Hemsey, J. K., Mammone, T., Schindler, D., Salas-Boni, R., Bai, Y., Tinoco, A., Ding, Q., & Hu, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0110274 · Cited by 438 · see citing works
How to study improvement interventions: a brief overview of possible study types
Portela, M. C., Pronovost, P. J., Woodcock, T. M., Carter, P., & Dixon‐Woods, M. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003620 · Cited by 319 · see citing works
With an eye to AI and autonomous diagnosis
Keane, P. A., & Topol, E. J. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0048-y · Cited by 304 · see citing works
The legal and ethical concerns that arise from using complex predictive analytics in health care
Cohen, I. G., Amarasingham, R., Shah, A., Xie, B., & Lo, B. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0048 · Cited by 300 · see citing works
Data Decisions and Theoretical Implications when Adversarially Learning Fair RepresentationsPreprint
Beutel, A., Chen, J., Zhao, Z., & Chi, E. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00075 · Cited by 282 · see citing works
Chin, M. H., Clarke, A. R., Nocon, R. S., Casey, A. A., Goddu, A. P., Keesecker, N. M., & Cook, S. C. (2012) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2082-9 · Cited by 276 · see citing works
Self-citation(2018) · Cited by 230 · see citing works
Learning Non-Discriminatory Predictors
Woodworth, B., Gunasekar, S., Ohannessian, M. I., & Srebro, N. (2017) · arXiv · doi:10.48550/arxiv.1702.06081 · Cited by 182 · see citing works
Reduction of Peripartum Racial and Ethnic Disparities
Howell, E. A., Brown, H. L., Brumley, J., Bryant, A., Caughey, A. B., Cornell, A., Grant, J., Gregory, K. D., Gullo, S. M., Kozhimannil, K. B., et al. (2018) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000002475 · Cited by 167 · see citing works
Epstein, A. M., Stern, R. S., Tognetti, J., Begg, C. B., Hartley, R. M., Cumella, E. J., & Ayanian, J. Z. (1988) · New England Journal of Medicine · doi:10.1056/nejm198806163182405 · Cited by 131 · see citing works
Inherent Trade-Offs in Algorithmic Fairness
Kleinberg, J. (2018) · Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems · doi:10.1145/3219617.3219634 · Cited by 125 · see citing works
Lessons for achieving health equity comparing Aotearoa/New Zealand and the United States
Chin, M. H., King, P. T., Jones, R., Jones, B., Ameratunga, S. N., Muramatsu, N., & Derrett, S. (2018) · Health Policy · doi:10.1016/j.healthpol.2018.05.001 · Cited by 106 · see citing works
Self-citationEscobar, G., Turk, B. J., Ragins, A. I., Ha, J., Hoberman, B., Levine, S., Ballesca, M. A., Liu, V. X., & Kipnis, P. (2016) · Journal of Hospital Medicine · doi:10.1002/jhm.2652 · Cited by 102 · see citing works
Howell, E. A., Brown, H. L., Brumley, J., Bryant, A., Caughey, A. B., Cornell, A., Grant, J. H., Gregory, K. D., Gullo, S. M., Kozhimannil, K. B., et al. (2018) · Journal of Obstetric, Gynecologic & Neonatal Nursing · doi:10.1016/j.jogn.2018.03.004 · Cited by 100 · see citing works
In the Era of Precision Medicine and Big Data, Who Is Normal?
Manrai, A. K., Patel, C. J., & Ioannidis, J. P. A. (2018) · JAMA · doi:10.1001/jama.2018.2009 · Cited by 98 · see citing works
Finding patients before they crash: the next major opportunity to improve patient safety
Bates, D. W., & Zimlichman, E. (2014) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003499 · Cited by 97 · see citing works
Systems Practices for the Care of Socially At-Risk Populations
Board on Population Health and Public Health Practice, Board on Health Care Services, Division, H. A. M., et al. (2016) · National Academies Press eBooks · doi:10.17226/21914 · Cited by 42 · see citing works
Emerging from EHR Purgatory — Moving from Process to Outcomes
Goroll, A. H. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1700601 · Cited by 26 · see citing works
Dhruva, S. S., Mazure, C. M., Ross, J. S., & Redberg, R. F. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.3148 · Cited by 17 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- [7]American Medical Association. AMA passes first policy recommendations on augmented intelligence 2018 Accessed at www.ama-assn.org/ama-passes-first-policy-recommendations-augmented-intelligence on 6 July 2018.
- [9]Smedley BD, Stith AY, Nelson AR, eds; Institute of Medicine. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care Washington, DC: National Academies Pr; 2003. 25032386
- [26]Owens DK, Sox HC. Biomedical decision making: probabilistic clinical reasoning. In: Shortliffe EH, Cimino JJ, eds. Biomedical Informatics London: Springer-Verlag; 2014: 67– 107.
- [36]Hardt M, Price E, Srebro N. Equality of opportunity in supervised learning [Abstract]. In: Proceedings from the Conference on Advances in Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2017 La Jolla, CA: Neural Information Processing Systems; 2017: 3315– 23.
- [41]Platt JC. Probabilities for SV machines. In: Smola AJ, Bartlett PJ, Schuurmans D, Schölkopf B, eds. Advances in Large Margin Classifiers Cambridge, MA: MIT Pr; 1999: 61– 74.
- [43]Pleiss G, Raghavan M, Wu F, Kleinberg J, Weinberger KQ. On fairness and calibration. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan,, eds. Proceedings from the Conference on Advances in Neural Information Processing Systems 2017, Long Beach, California, 4–9 December 2017 La Jolla, CA: Neural Information Processing Systems; 2017: 5680– 9.
- [44]Kilbertus N, Rojas-Carulla M, Parascandolo G, Hardt M, Janzing D, Schölkopf B. Avoiding discrimination through causal reasoning. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S,, eds. Proceedings from the Conference on Advances in Neural Information Processing Systems 2017, Long Beach, CA, 4–9 December 2017 La Jolla, CA: Neural Information Processing Systems; 2017: 656– 66.
Cited by
Sorted by citations
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579 · see citing works
Self-citationThe ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · see citing works
Digital technology, tele-medicine and artificial intelligence in ophthalmology: A global perspective
Li, J.-P. O., Liu, H., Ting, D. S. J., Jeon, S., Chan, R. V. P., Kim, J. E., Sim, D. A., Thomas, P. B. M., Lin, H., Chen, Y., et al. (2021) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2020.100900 · Cited by 626 · see citing works
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607 · see citing works
Peiffer-Smadja, N., Rawson, T. M., Ahmad, R., Buchard, A., Georgiou, P., Lescure, F.-X., Birgand, G., & Holmes, A. H. (2020) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2019.09.009 · Cited by 561 · see citing works
Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review
Celi, L. A., Cellini, J., Charpignon, M.-L., Dee, E. C., Dernoncourt, F., Eber, R., Mitchell, W. G., Moukheiber, L., Schirmer, J., Situ, J., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000022 · Cited by 576 · see citing works
Ethical Machine Learning in Healthcare
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092820-114757 · Cited by 645 · see citing works
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553 · see citing works
Artificial intelligence and algorithmic bias: implications for health systems
Panch, T., Mattie, H., & Atun, R. (2019) · Journal of Global Health · doi:10.7189/jogh.09.020318 · Cited by 533 · see citing works
Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety
Huckvale, K., Venkatesh, S., & Christensen, H. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0166-1 · Cited by 433 · see citing works
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483 · see citing works
Self-citationIdentifying Ethical Considerations for Machine Learning Healthcare Applications
Char, D. S., Abràmoff, M. D., & Feudtner, C. (2020) · The American Journal of Bioethics · doi:10.1080/15265161.2020.1819469 · Cited by 415 · see citing works
Krieger, N. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094017 · Cited by 411 · see citing works
Back to the Future: Achieving Health Equity Through Health Informatics and Digital Health
Brewer, L. C., Fortuna, K. L., Jones, C., Walker, R., Hayes, S. N., Patten, C. A., & Cooper, L. A. (2020) · JMIR mHealth and uHealth · doi:10.2196/14512 · Cited by 367 · see citing works
Ethical implications of AI and robotics in healthcare: A review
Elendu, C., Amaechi, D. C., Elendu, T. C., Jingwa, K. A., Okoye, O. K., John Okah, M., Ladele, J. A., Farah, A. H., & Alimi, H. A. (2023) · Medicine · doi:10.1097/md.0000000000036671 · Cited by 370 · see citing works
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358 · see citing works
Considerations for addressing bias in artificial intelligence for health equity
Abràmoff, M. D., Tarver, M. E., Loyo-Berrios, N., Trujillo, S., Char, D., Obermeyer, Z., Eydelman, M. B., Foundational Principles of Ophthalmic Imaging and Algorithmic Interpretation Working Group of the Collaborative Community for Ophthalmic Imaging Foundation, W., & Maisel, W. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00913-9 · Cited by 349 · see citing works
Accessing Artificial Intelligence for Clinical Decision-Making
Giordano, C., Brennan, M., Mohamed, B., Rashidi, P., Modave, F., & Tighe, P. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.645232 · Cited by 326 · see citing works
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · see citing works
Paulus, J. K., & Kent, D. M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0304-9 · Cited by 274 · see citing works
Chin, M. H., Afsar-Manesh, N., Bierman, A. S., Chang, C., Colón-Rodríguez, C. J., Dullabh, P., Duran, D. G., Fair, M., Hernandez-Boussard, T., Hightower, M., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.45050 · Cited by 275 · see citing works
Self-citationIlluminating the dark spaces of healthcare with ambient intelligence
Haque, A., Milstein, A., & Fei-Fei, L. (2020) · Nature · doi:10.1038/s41586-020-2669-y · Cited by 274 · see citing works
Mienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212 · see citing works
Latent bias and the implementation of artificial intelligence in medicine
DeCamp, M., & Lindvall, C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa094 · Cited by 198 · see citing works
Giovanola, B., & Tiribelli, S. (2023) · AI & SOCIETY · doi:10.1007/s00146-022-01455-6 · Cited by 193 · see citing works
Evaluation and Mitigation of Racial Bias in Clinical Machine Learning Models: Scoping Review
Huang, J., Galal, G., Etemadi, M., & Vaidyanathan, M. (2022) · JMIR Medical Informatics · doi:10.2196/36388 · Cited by 186 · see citing works
Structural Racism: The Rules and Relations of Inequity
Gee, G. C., & Hicken, M. T. (2021) · Ethnicity & Disease · doi:10.18865/ed.31.s1.293 · Cited by 181 · see citing works
Does “ AI ” stand for augmenting inequality in the era of covid-19 healthcare?
Leslie, D., Mazumder, A., Peppin, A., Wolters, M. K., & Hagerty, A. (2021) · BMJ · doi:10.1136/bmj.n304 · Cited by 177 · see citing works
Three Epochs of Artificial Intelligence in Health Care
Howell, M. D., Corrado, G. S., & DeSalvo, K. B. (2024) · JAMA · doi:10.1001/jama.2023.25057 · Cited by 185 · see citing works
Self-citationA Path for Translation of Machine Learning Products into Healthcare Delivery
(2020) · EMJ Innovations · doi:10.33590/emjinnov/19-00172 · Cited by 172 · see citing works
“106 Second, the dataset used to train a model 107 or the outcome that a model predicts 108 can have significant implications for how models lessen or worsen disparities in healthcare.”
BackgroundMethodAn empirical characterization of fair machine learning for clinical risk prediction
Pfohl, S. R., Foryciarz, A., & Shah, N. H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103621 · Cited by 164 · see citing works
Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157 · see citing works
“To validate the AI system, performance metrics should be systematically and continuously evaluated after the AI has been deployed [159].”
BackgroundMethodInfluentialAlgorithmic fairness in computational medicine
Xu, J., Xiao, Y., Wang, W. H., Ning, Y., Shenkman, E. A., Bian, J., & Wang, F. (2022) · eBioMedicine · doi:10.1016/j.ebiom.2022.104250 · Cited by 170 · see citing works
Artificial Intelligence and Early Detection of Pancreatic Cancer: 2020 Summative Review
Kenner, B., Chari, S. T., Kelsen, D., Klimstra, D. S., Pandol, S. J., Rosenthal, M., Rustgi, A. K., Taylor, J. A., Yala, A., Abul-Husn, N., et al. (2021) · Pancreas · doi:10.1097/mpa.0000000000001762 · Cited by 158 · see citing works
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155 · see citing works
Generative models improve fairness of medical classifiers under distribution shifts
Ktena, I., Wiles, O., Albuquerque, I., Rebuffi, S.-A., Tanno, R., Roy, A. G., Azizi, S., Belgrave, D., Kohli, P., Cemgil, T., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02838-6 · Cited by 165 · see citing works
Abràmoff, M. D., Tobey, D., & Char, D. S. (2020) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2020.02.022 · Cited by 140 · see citing works
Drukker, K., Chen, W., Gichoya, J. W., Gruszauskas, N. P., Kalpathy-Cramer, J., Koyejo, S., Myers, K. J., Sá, R. C., Sahiner, B., Whitney, H. M., et al. (2023) · Journal of Medical Imaging · doi:10.1117/1.jmi.10.6.061104 · Cited by 149 · see citing works
Understanding and Mitigating Bias in Imaging Artificial Intelligence
Tejani, A. S., Ng, Y. S., Xi, Y., & Rayan, J. C. (2024) · RadioGraphics · doi:10.1148/rg.230067 · Cited by 138 · see citing works
Z-Inspection <sup>®</sup> : A Process to Assess Trustworthy AI
Zicari, R. V., Brodersen, J., Brusseau, J., Dudder, B., Eichhorn, T., Ivanov, T., Kararigas, G., Kringen, P., McCullough, M., Moslein, F., et al. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3066209 · Cited by 134 · see citing works
Ensuring that biomedical AI benefits diverse populations
Zou, J., & Schiebinger, L. (2021) · EBioMedicine · doi:10.1016/j.ebiom.2021.103358 · Cited by 124 · see citing works
Advancing health equity with artificial intelligence
Thomasian, N. M., Eickhoff, C., & Adashi, E. Y. (2021) · Journal of Public Health Policy · doi:10.1057/s41271-021-00319-5 · Cited by 117 · see citing works
A reimbursement framework for artificial intelligence in healthcare
Abràmoff, M. D., Roehrenbeck, C., Trujillo, S., Goldstein, J., Graves, A. S., Repka, M. X., & Silva III, E. “. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00621-w · Cited by 117 · see citing works
Ong, J. C. L., Seng, B. J. J., Law, J. Z. F., Low, L. L., Kwa, A. L. H., Giacomini, K. M., & Ting, D. S. W. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101356 · Cited by 111 · see citing works
Implementing Ethics in Healthcare AI-Based Applications: A Scoping Review
Goirand, M., Austin, E., & Clay-Williams, R. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-021-00336-3 · Cited by 101 · see citing works
Benzinger, L., Ursin, F., Balke, W.-T., Kacprowski, T., & Salloch, S. (2023) · BMC Medical Ethics · doi:10.1186/s12910-023-00929-6 · Cited by 95 · see citing works
Lyell, D., Coiera, E., Chen, J., Shah, P., & Magrabi, F. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100301 · Cited by 95 · see citing works
McInnes, G., Sharo, A. G., Koleske, M. L., Brown, J. E. H., Norstad, M., Adhikari, A. N., Wang, S., Brenner, S. E., Halpern, J., Koenig, B. A., et al. (2021) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2021.03.003 · Cited by 86 · see citing works
“’’(32) More than simply doing no harm, a GLHS should actively support greater health equity.(4,105) Of central importance is whether clinical data, or model parameters if deploying a federated learning approach, could be viewed and secured as a public good insofar as all stakeholders, both healthcare and private indus…”
BackgroundJuhn, Y. J., Ryu, E., Wi, C.-I., King, K. S., Malik, M., Romero-Brufau, S., Weng, C., Sohn, S., Sharp, R. R., & Halamka, J. D. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac052 · Cited by 83 · see citing works
Critical theory, culture change, and achieving health equity in health care settings
Todic´, J., Cook, S. C., Spitzer-Shohat, S., Williams, J. S., Battle, B. A., Jackson, J., & Chin, M. H. (2022) · Academic Medicine · doi:10.1097/acm.0000000000004680 · Cited by 91 · see citing works
Self-citationPeeking into a black box, the fairness and generalizability of a MIMIC-III benchmarking model.
Röösli, E., Bozkurt, S., & Hernandez-Boussard, T. (2022) · Scientific Data · doi:10.1038/s41597-021-01110-7 · Cited by 73 · see citing works
Chen, S., Yu, J., Chamouni, S., Wang, Y., & Li, Y. (2024) · BMC Medicine · doi:10.1186/s12916-024-03566-x · Cited by 73 · see citing works
“If the training data used to develop these models are not representative of the broader population or if they contain historical biases, the resulting algorithms may perpetuate or even amplify these biases [58].”
BackgroundMethodThe Debate on the Ethics of AI in Health Care: a Reconstruction and Critical Review
Morley, J., Machado, C., Burr, C., Cowls, J., Taddeo, M., & Floridi, L. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3486518 · Cited by 66 · see citing works
Ethics of artificial intelligence in medicine
Savulescu, J., Giubilini, A., Vandersluis, R., & Mishra, A. (2024) · Singapore Medical Journal · doi:10.4103/singaporemedj.smj-2023-279 · Cited by 68 · see citing works
Strategies to improve the impact of artificial intelligence on health equity: scoping review
Berdahl, C. T., Baker, L., Mann, S., Osoba, O., & Girosi, F. (2023) · JMIR AI · doi:10.2196/42936 · Cited by 68 · see citing works
Wang, H. E., Landers, M., Adams, R., Subbaswamy, A., Kharrazi, H., Gaskin, D. J., & Saria, S. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac065 · Cited by 63 · see citing works
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 61 · see citing works
“Centering Participation in Data Curation Participatory approaches play a vital role in bridging the gaps between model development and deployment and in promoting fairness in ML applications (Rajkomar et al., 2018).”
BackgroundCallahan, A., McElfresh, D., Banda, J. M., Bunney, G., Char, D., Chen, J., Corbin, C. K., Dash, D., Downing, N. L., Jain, S. S., et al. (2024) · NEJM Catalyst · doi:10.1056/cat.24.0131 · Cited by 61 · see citing works
Machine Learning, Health Disparities, and Causal Reasoning
Goodman, S. N., Goel, S., & Cullen, M. R. (2018) · Annals of Internal Medicine · doi:10.7326/m18-3297 · Cited by 57 · see citing works
Lessons and tips for designing a machine learning study using EHR data
Arbet, J., Brokamp, C., Meinzen-Derr, J., Trinkley, K. E., & Spratt, H. M. (2021) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.513 · Cited by 46 · see citing works
Algorithmic fairness of machine learning models for Alzheimer disease progression
Yuan, C., Linn, K. A., & Hubbard, R. A. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.42203 · Cited by 49 · see citing works
A Review of Applications of Machine Learning in Mammography and Future Challenges
Batchu, S., Liu, F., Amireh, A., Waller, J., & Umair, M. (2021) · Oncology · doi:10.1159/000515698 · Cited by 41 · see citing works
Bias Mitigation in Primary Health Care Artificial Intelligence Models: Scoping Review
Sasseville, M., Ouellet, S., Rhéaume, C., Sahlia, M., Couture, V., Després, P., Paquette, J.-S., Darmon, D., Bergeron, F., & Gagnon, M.-P. (2025) · Journal of Medical Internet Research · doi:10.2196/60269 · Cited by 38 · see citing works
Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · arXiv · doi:10.48550/arxiv.2003.11515 · Cited by 37 · see citing works
Mccradden, M., Odusi, O., Joshi, S., Akrout, I., Ndlovu, K., Glocker, B., Maicas, G., Liu, X., Mazwi, M., Garnett, T., et al. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594096 · Cited by 33 · see citing works
Pfohl, S., Xu, Y., Foryciarz, A., Ignatiadis, N., Genkins, J., & Shah, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533166 · Cited by 30 · see citing works
Holm, S., Stanton, C., & Bartlett, B. (2021) · Health Care Analysis · doi:10.1007/s10728-021-00430-4 · Cited by 32 · see citing works
Predicting inpatient medication orders from electronic health record data
Rough, K., Dai, A. M., Zhang, K., Xue, Y., Vardoulakis, L. M., Cui, C., Butte, A. J., Howell, M. D., & Rajkomar, A. (2020) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1826 · Cited by 28 · see citing works
Self-citationPredicting outcomes of psychotherapy for depression with electronic health record data
Coley, R. Y., Boggs, J. M., Beck, A., & Simon, G. E. (2021) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2021.100198 · Cited by 29 · see citing works
Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination
Kallus, N., Mao, X., & Zhou, A. (2019) · arXiv · doi:10.48550/arxiv.1906.00285 · Cited by 28 · see citing works
Can medical algorithms be fair? Three ethical quandaries and one dilemma
Bærøe, K., Gundersen, T., Henden, E., & Rommetveit, K. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100445 · Cited by 22 · see citing works
Sun, K., Roy, A., & Tobin, J. M. (2024) · Journal of Critical Care · doi:10.1016/j.jcrc.2024.154792 · Cited by 28 · see citing works
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483 · see citing works
Self-citationExploring explainable AI features in the vocal biomarkers of lung disease
Chen, Z., Liang, N., Li, H., Zhang, H., Li, H., Yan, L., Hu, Z., Chen, Y., Zhang, Y., Wang, Y., et al. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108844 · Cited by 26 · see citing works
A call for open data to develop mental health digital biomarkers
Adler, D. A., Wang, F., Mohr, D. C., Estrin, D., Livesey, C., & Choudhury, T. (2022) · BJPsych Open · doi:10.1192/bjo.2022.28 · Cited by 23 · see citing works
A step toward building a unified framework for managing AI bias
Rana, S. A., Azizul, Z. H., & Awan, A. A. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1630 · Cited by 23 · see citing works
Genovesi, S., Mönig, J. M., Schmitz, A., Poretschkin, M., Akila, M., Kahdan, M., Kleiner, R., Krieger, L., & Zimmermann, A. (2024) · AI and Ethics · doi:10.1007/s43681-023-00291-8 · Cited by 19 · see citing works
Fair Machine Learning in Healthcare: A Survey
Feng, Q., Du, M., Zou, N., & Hu, X. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3361836 · Cited by 12 · see citing works
Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery
Kuno, M., Osumi, H., Udagawa, S., Yoshikawa, K., Ooki, A., Shinozaki, E., Ishikawa, T., Oba, J., Yamaguchi, K., & Sakurada, K. (2025) · Current Oncology · doi:10.3390/curroncol32110588 · Cited by 7 · see citing works
Algorithmic bias, generalist models, and clinical medicine
Keeling, G. (2024) · AI and Ethics · doi:10.1007/s43681-023-00329-x · Cited by 5 · see citing works
Hundal, J., Veettil, A. A. V., Chung, C., & Hosseini, M. (2026) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk-26-517280 · Cited by 1 · see citing works
Lee, S.-Y., Saleem, M., Land, A. M., DeLaney, E. W., Garretson, A. R., Patel, M., Hall, A. G., Westrick, S., Ovalle, F., Cherrington, A. L., et al. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106604
- Authors
- Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H.
- Affiliations
- Google (United States) (US)
- University of Chicago (US)
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2018-12-18
- Volume
- 169
- Issue
- 12
- Pages
- 866-872
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- PMC
- PMC6594166
- OpenAlex
- W2902802452
- Cited by
- 1,341
- References
- 41
- Citations by year
- 2026: 380 · 2025: 292 · 2024: 190 · 2023: 155 · 2022: 119 · 2021: 107 · 2020: 71 · 2019: 23 · 2018: 2 · 2012: 1
- FWCI
- 100.84
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092949)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- algorithmic fairness, health equity, health care disparities, distributive justice, protected groups, structural bias, algorithmic bias, resource allocation, clinical decision support, model evaluation
- MeSH terms
- Machine Learning, Health Equity, Critical Care, Humans, Length of Stay, Social Justice, Health Care Rationing, Healthcare Disparities, Patient Outcome Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
