The Sociodemographic Biases in Machine Learning Algorithms: A Biomedical Informatics Perspective
Open accessThe DOI matches its Crossref record; 97 of the 97 references it cites were found in a registry or an index.
Abstract
Artificial intelligence models represented in machine learning algorithms are promising tools for risk assessment used to guide clinical and other health care decisions. Machine learning algorithms, however, may house biases that propagate stereotypes, inequities, and discrimination that contribute to socioeconomic health care disparities. The biases include those related to some sociodemographic characteristics such as race, ethnicity, gender, age, insurance, and socioeconomic status from the use of erroneous electronic health record data. Additionally, there is concern that training data and algorithmic biases in large language models pose potential drawbacks. These biases affect the lives and livelihoods of a significant percentage of the population in the United States and globally. The social and economic consequences of the associated backlash cannot be underestimated. Here, we outline some of the sociodemographic, training data, and algorithmic biases that undermine sound health care risk assessment and medical decision-making that should be addressed in the health care system. We present a perspective and overview of these biases by gender, race, ethnicity, age, historically marginalized communities, algorithmic bias, biased evaluations, implicit bias, selection/sampling bias, socioeconomic status biases, biased data distributions, cultural biases and insurance status bias, conformation bias, information bias and anchoring biases and make recommendations to improve large language model training data, including de-biasing techniques such as counterfactual role-reversed sentences during knowledge distillation, fine-tuning, prefix attachment at training time, the use of toxicity classifiers, retrieval augmented generation and algorithmic modification to mitigate the biases moving forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 4×
“With the advent of big data analytics, EHRs are often the source of training data for AIMs and, subsequently, MLAGs, thus creating a feedback loop of bias (Figure 1) where existing inequities and disparities can be exacerbated through positive reinforcement methods [12,50].”
BackgroundInfluentialOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 2×
“The fact is that MLAGs born of AIMs may not perform as well on women as on men when they are under-represented in the training data [16,74].”
BackgroundA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Implicit bias in healthcare professionals: a systematic review
FitzGerald, C., & Hurst, S. (2017) · BMC Medical Ethics · doi:10.1186/s12910-017-0179-8 · Cited by 2,556 · cited 2×
“Implicit bias imposed by clinicians has an impact on the overall quality of patients’ health and health care and can lead to unequal treatment based on certain demographics [17,23,48].”
BackgroundHidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms
Vyas, D. A., Eisenstein, L. G., & Jones, D. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmms2004740 · Cited by 1,731 · cited 3×
“While recommendations of race-based adjustment have been considered to mitigate these issues, these algorithms can also be negatively biased toward Black individuals, further perpetuating racial bias in health care [40].”
BackgroundAlgorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Measuring Diagnoses: ICD Code Accuracy
O'Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,116
Bias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685 · cited 1×
“The fact is that MLAGs born of AIMs may not perform as well on women as on men when they are under-represented in the training data [16,74].”
BackgroundBrinker, T. J., Hekler, A., Enk, A. H., Klode, J., Hauschild, A., Berking, C., Schilling, B., Haferkamp, S., Schadendorf, D., Holland-Letz, T., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.04.001 · Cited by 605
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598 · cited 1×
“Moving forward, we need a community or consensus that puts forth best-practice guidelines for fair, accurate representation of diverse populations [75] in the USA and globally.”
BackgroundSources of bias in artificial intelligence that perpetuate healthcare disparities—A global review
Celi, L. A., Cellini, J., Charpignon, M.-L., Dee, E. C., Dernoncourt, F., Eber, R., Mitchell, W. G., Moukheiber, L., Schirmer, J., Situ, J., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000022 · Cited by 576 · cited 3×
“In this cited work, the primary authors were not clinicians, the preponderance of authors were male, and the top databases available were mostly from high-income countries [51].”
BackgroundInfluentialSex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553 · cited 1×
“With the advent of big data analytics, EHRs are often the source of training data for AIMs and, subsequently, MLAGs, thus creating a feedback loop of bias (Figure 1) where existing inequities and disparities can be exacerbated through positive reinforcement methods [12,50].”
BackgroundDisparities in dermatology AI performance on a diverse, curated clinical image set
Daneshjou, R., Vodrahalli, K., Novoa, R. A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S. M., Bailey, E. E., Gevaert, O., et al. (2022) · Science Advances · doi:10.1126/sciadv.abq6147 · Cited by 544 · cited 3×
“However, samples from racial and ethnic minorities have limited representation [61–65].”
BackgroundOikonomou, E. K., Williams, M. C., Kotanidis, C. P., Desai, M. Y., Marwan, M., Antonopoulos, A. S., Thomas, K. E., Thomas, S., Akoumianakis, I., Fan, L. M., et al. (2019) · European Heart Journal · doi:10.1093/eurheartj/ehz592 · Cited by 534 · cited 1×
“For example, MLAGs that predict cardiovascular disease often utilize larger proportions of men in their training data [69], yet women are more likely to present with atypical features of cardiovascular disease [70,71], and cardiovascular disease is more often misdiagnosed in women as compared to…”
BackgroundDeep neural networks are superior to dermatologists in melanoma image classification
Brinker, T. J., Hekler, A., Enk, A. H., Berking, C., Haferkamp, S., Hauschild, A., Weichenthal, M., Klode, J., Schadendorf, D., Holland-Letz, T., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.05.023 · Cited by 415
Daneshjou, R., Smith, M. P., Sun, M. D., Rotemberg, V., & Zou, J. (2021) · JAMA Dermatology · doi:10.1001/jamadermatol.2021.3129 · Cited by 403
Racial/Ethnic and Gender Disparities in Health Care Use and Access
Manuel, J. I. (2018) · Health Services Research · doi:10.1111/1475-6773.12705 · Cited by 378 · cited 1×
“In a similar manner, the presence of gender disparities in health care has been well documented for centuries [18,66–68].”
BackgroundFairness in Recommendation Ranking through Pairwise Comparisons
Beutel, A., Chen, J., Doshi, T., Qian, H., Wei, L., Wu, Y., Heldt, L., Zhao, Z., Hong, L., Chi, E. H., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330745 · Cited by 362 · cited 1×
“…through algorithmic modifications that look at machine learning models’ loss of function by integrating “regularization” language specifically for the presumption of fairness, “for clas-sification” [91,92], “for ranking” [93], “for matching tasks” [94,95], and “for recommender systems” [96].”
BackgroundFairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330691 · Cited by 350 · cited 1×
“…through algorithmic modifications that look at machine learning models’ loss of function by integrating “regularization” language specifically for the presumption of fairness, “for clas-sification” [91,92], “for ranking” [93], “for matching tasks” [94,95], and “for recommender systems” [96].”
BackgroundFujisawa, Y., Otomo, Y., Ogata, Y., Nakamura, Y., Fujita, R., Ishitsuka, Y., Watanabe, R., Okiyama, N., Ohara, K., & Fujimoto, M. (2019) · British Journal of Dermatology · doi:10.1111/bjd.16924 · Cited by 317
Gender disparity in the funding of diseases by the U.S National Institutes of Health
Mirin, A. A. (2021) · Journal of Women's Health · doi:10.1089/jwh.2020.8682 · Cited by 269
Machine Learning in Dermatology: Current Applications, Opportunities, and Limitations
Chan, S., Reddy, V., Myers, B., Thibodeaux, Q., Brownstone, N., & Liao, W. (2020) · Dermatology and Therapy · doi:10.1007/s13555-020-00372-0 · Cited by 267 · cited 1×
“Generally, the data used for training deep convolutional neural networks (DCNNs) are biased towards datasets of individuals of predominantly Western European and Asian descent [54,55].”
BackgroundRokach, L. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.07.017 · Cited by 253 · cited 1×
“…multiple models are created to address the initial task, then combined to optimize the results for a more precise, well-rounded, and genuine model [97]; for example, building a model designed to have basic medical and health care knowledge to fall back on when the training does not accurately…”
BackgroundApplications of Artificial Intelligence (AI) in healthcare: A review
Shaheen, M. Y. (2021) · doi:10.14293/s2199-1006.1.sor-.ppvry8k.v1 · Cited by 247
Physician Use of Stigmatizing Language in Patient Medical Records
Park, J., Saha, S., Chee, B., Taylor, J., & Beach, M. C. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.17052 · Cited by 239
Haenssle, H. A., Fink, C., Toberer, F., Winkler, J., Stolz, W., Deinlein, T., Hofmann-Wellenhof, R., Lallas, A., Emmert, S., Buhl, T., et al. (2020) · Annals of Oncology · doi:10.1016/j.annonc.2019.10.013 · Cited by 236
GeDi: Generative Discriminator Guided Sequence Generation
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.424 · Cited by 220
Challenges with quality of race and ethnicity data in observational databases
Polubriaginof, F. C. G., Ryan, P., Salmasian, H., Shapiro, A. W., Perotte, A., Safford, M. M., Hripcsak, G., Smith, S., Tatonetti, N. P., & Vawdrey, D. K. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz113 · Cited by 208 · cited 1×
“In the USA, the diversity and quality of racial and ethnic data in observational databases are suboptimal [53].”
BackgroundHuman-Centered Design to Address Biases in Artificial Intelligence
Chen, Y., Clayton, E. W., Novak, L. L., Anders, S., & Malin, B. (2023) · Journal of Medical Internet Research · doi:10.2196/43251 · Cited by 199
Giovanola, B., & Tiribelli, S. (2023) · AI & SOCIETY · doi:10.1007/s00146-022-01455-6 · Cited by 193 · cited 2×
“Whether implicit, cognitive, or algorithmic, bias has been seen to attenuate the predictive accuracy of MLAGs derived from AIMs [45], and model bias, frequently a result of learned bias that presents in many forms, often trickles down the ML pipeline [8,13].”
BackgroundObaid, H. S., Dheyab, S. A., & Sabry, S. S. (2019) · 2019 9th Annual Information Technology, Electromechanical Engineering and Microelectronics Conference (IEMECON) · doi:10.1109/iemeconx.2019.8877011 · Cited by 158 · cited 1×
“However, some algorithmic bias reduction techniques may diminish the model’s overall accuracy, [46], a tradeoff that may be acceptable.”
BackgroundTowards Understanding and Mitigating Social Biases in Language Models
Liang, P. P., Wu, C., Morency, L.-P., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2106.13219 · Cited by 153 · cited 1×
“In health care settings, where pre-trained language models (LMs) are used, stereotypes can be perpetuated in the form of harmful biases, a direct result of societal stereotypes and adverse generalizations [80].”
BackgroundDExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.522 · Cited by 148 · cited 1×
“…that LLM text generation can be more objective and impartial to mitigate training data bias by (1) fine-tuning or training on balanced datasets [81,83]; (2) incorporating prefix attachment at training time [83]; and (3) using a bias or toxicity/attribute classifier [79,84,85] to train datasets.”
BackgroundArtificial intelligence in public health: the potential of epidemic early warning systems
MacIntyre, C. R., Chen, X., Kunasekaran, M., Quigley, A., Lim, S., Stone, H., Paik, H.-Y., Yao, L., Heslop, D., Wei, W., et al. (2023) · Journal of International Medical Research · doi:10.1177/03000605231159335 · Cited by 135 · cited 1×
“In health care, ML can take on many forms and has been seen to have a multitude of positive applications [42–44].”
BackgroundSocietal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131 · cited 2×
“…that LLM text generation can be more objective and impartial to mitigate training data bias by (1) fine-tuning or training on balanced datasets [81,83]; (2) incorporating prefix attachment at training time [83]; and (3) using a bias or toxicity/attribute classifier [79,84,85] to train datasets.”
BackgroundGuo, L. N., Lee, M. S., Kassamali, B., Mita, C., & Nambudiri, V. E. (2022) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2021.06.884 · Cited by 122 · cited 1×
“However, samples from racial and ethnic minorities have limited representation [61–65].”
BackgroundWhy Is the Electronic Health Record So Challenging for Research and Clinical Care?
Holmes, J. H., Beinlich, J., Boland, M. R., Bowles, K. H., Chen, Y., Cook, T. S., Demiris, G., Draugelis, M., Fluharty, L., Gabriel, P. E., et al. (2021) · Methods of Information in Medicine · doi:10.1055/s-0041-1731784 · Cited by 117 · cited 1×
“To avoid these practices, health care organizations need to improve the quality, accuracy, and usability of EHRs [34].”
BackgroundSelf-citationA Nationwide Network of Health AI Assurance Laboratories
Shah, N. H., Halamka, J. D., Saria, S., Pencina, M., Tazbaz, T., Tripathi, M., Callahan, A., Hildahl, H., & Anderson, B. (2024) · JAMA · doi:10.1001/jama.2023.26930 · Cited by 114 · cited 1×
“…care professionals involved in using these MLAGs derived from AI models that they are rid of bias, exhibit fairness, and are appropriately used [76], especially in populations who have historically faced systematic inequities, such as prejudice, discrimination, and disparities in health care…”
MethodSühr, T., Biega, A. J., Zehlike, M., Gummadi, K. P., & Chakraborty, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330793 · Cited by 112
Three Problems with Big Data and Artificial Intelligence in Medicine
Chin-Yee, B., & Upshur, R. (2019) · Perspectives in Biology and Medicine · doi:10.1353/pbm.2019.0012 · Cited by 111
Black Kidney Function Matters: Use or Misuse of Race?
Powe, N. R. (2020) · JAMA · doi:10.1001/jama.2020.13378 · Cited by 111 · cited 3×
“Recognize race as a social, not genetic, construct, and avoid use in clinical settings/situations [22,23].”
BackgroundEthical Development of Digital Phenotyping Tools for Mental Health Applications: Delphi Study
Martinez-Martin, N., Greely, H. T., & Cho, M. K. (2021) · JMIR mHealth and uHealth · doi:10.2196/27343 · Cited by 111
Kamulegeya, L., Bwanika, J., Okello, M., Rusoke, D., Nassiwa, F., Lubega, W., Musinguzi, D., & Börve, A. (2023) · African Health Sciences · doi:10.4314/ahs.v23i2.86 · Cited by 111
Kamulegeya, L. H., Okello, M., Bwanika, J. M., Musinguzi, D., Lubega, W., Rusoke, D., Nassiwa, F., & Börve, A. (2019) · doi:10.1101/826057 · Cited by 111 · cited 5×
“Generally, the data used for training deep convolutional neural networks (DCNNs) are biased towards datasets of individuals of predominantly Western European and Asian descent [54,55].”
BackgroundMethodInfluentialSrinivasan, R., & Chander, A. (2021) · Communications of the ACM · doi:10.1145/3464903 · Cited by 104
Straw, I., & Wu, H. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100457 · Cited by 104 · cited 2×
“For example, the Indian Liver Patient Dataset (ILPD), used to create predictive algorithms for liver disease, classifies women with a high false negative rate (FNR), resulting in a deficiency of appropriate care and, thus, poor health outcomes [21].”
BackgroundMethodPham, T.-C., Luong, C.-M., Hoang, V.-D., & Doucet, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-96707-8 · Cited by 101
Balayn, A., Lofi, C., & Houben, G.-J. (2021) · The VLDB Journal · doi:10.1007/s00778-021-00671-8 · Cited by 98 · cited 1×
“According to Balayn et al. [90], approaches to mitigating AB have been accomplished through algorithmic modifications that look at machine learning models’ loss of function by integrating “regularization” language specifically for the presumption of fairness, “for clas-sification” [91,92], “for…”
Prejudiced Places: How Contexts Shape Inequality and How Policy Can Change Them
Murphy, M. C., Kroeper, K. M., & Ozier, E. M. (2018) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732217748671 · Cited by 96 · cited 1×
“…care professionals involved in using these MLAGs derived from AI models that they are rid of bias, exhibit fairness, and are appropriately used [76], especially in populations who have historically faced systematic inequities, such as prejudice, discrimination, and disparities in health care [77].”
Explainability in medicine in an era of AI-based clinical decision support systems
Pierce, R. L., Van Biesen, W., Van Cauwenberge, D., Decruyenaere, J., & Sterckx, S. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.903600 · Cited by 95 · cited 1×
“Most algorithmic tools are proprietary “black boxes” that often update, so the end user is unaware of the mechanisms for the decision it produces [9].”
BackgroundKino, S., Hsu, Y.-T., Shiba, K., Chien, Y.-S., Mita, C., Kawachi, I., & Daoud, A. (2021) · SSM - Population Health · doi:10.1016/j.ssmph.2021.100836 · Cited by 91 · cited 1×
“There is overwhelming evidence that there are biases in various artificial intelligence models (AIMs) applied to machine learning algorithms (MLAGs) used in health care and other industries [7,16,35–39].”
BackgroundProcess for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Solaiman, I., & Dennison, C. (2021) · arXiv · doi:10.48550/arxiv.2106.10328 · Cited by 90 · cited 2×
“…that LLM text generation can be more objective and impartial to mitigate training data bias by (1) fine-tuning or training on balanced datasets [81,83]; (2) incorporating prefix attachment at training time [83]; and (3) using a bias or toxicity/attribute classifier [79,84,85] to train datasets.”
BackgroundMethodJuhn, Y. J., Ryu, E., Wi, C.-I., King, K. S., Malik, M., Romero-Brufau, S., Weng, C., Sohn, S., Sharp, R. R., & Halamka, J. D. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac052 · Cited by 83 · cited 2×
“Implicit bias imposed by clinicians has an impact on the overall quality of patients’ health and health care and can lead to unequal treatment based on certain demographics [17,23,48].”
BackgroundAI in healthcare: A narrative review
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.26997.2 · Cited by 69
Gender differences in the clinical features and outcomes of patients with coronary artery disease
Wada, H., Miyauchi, K., & Daida, H. (2018) · Expert Review of Cardiovascular Therapy · doi:10.1080/14779072.2019.1561277 · Cited by 66 · cited 1×
“…MLAGs that predict cardiovascular disease often utilize larger proportions of men in their training data [69], yet women are more likely to present with atypical features of cardiovascular disease [70,71], and cardiovascular disease is more often misdiagnosed in women as compared to men [72,73].”
BackgroundGervasi, S. S., Chen, I. Y., Smith-McLallen, A., Sontag, D., Obermeyer, Z., Vennera, M., & Chawla, R. (2022) · Health Affairs · doi:10.1377/hlthaff.2021.01287 · Cited by 60 · cited 1×
“Implicit Bias Consider the nature of bias and use counterfactual explanations as a predictive tool to detect attributes that are assigned and thus plague certain populations [15,29].”
BackgroundMitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal
Gupta, U., Dhamala, J., Kumar, V., Verma, A., Pruksachatkun, Y., Krishna, S., Gupta, R., Chang, K.-W., Steeg, G. V., & Galstyan, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.55 · Cited by 60
Fairness-Aware Machine Learning Engineering: How Far Are We?
Ferrara, C., Sellitto, G., Ferrucci, F., Palomba, F., & De Lucia, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10402-y · Cited by 60
Uses and Misuses of Patient- and Neighborhood-level Social Determinants of Health Data
Gottlieb, L. M., Francis, D. E., & Beck, A. F. (2018) · The Permanente Journal · doi:10.7812/tpp/18-078 · Cited by 59 · cited 1×
“Socioeconomic status (SES) Bias Work to limit the misunderstanding between patients and providers due to cultural and linguistic differences; keep the positive aspects of the social determinants of health (SDoH) in mind [12,30–32].”
BackgroundOne Algorithm May Not Fit All: How Selection Bias Affects Machine Learning Performance
Yu, A. C., & Eng, J. (2020) · RadioGraphics · doi:10.1148/rg.2020200040 · Cited by 59 · cited 2×
“Review and keep in mind basic clinical epidemiological principles (e.g., selection bias , which affects “real-world” study interpretation and generalizability to a representative population) [13,27].”
BackgroundPreCoF: counterfactual explanations for fairness
Goethals, S., Martens, D., & Calders, T. (2023) · Machine Learning · doi:10.1007/s10994-023-06319-8 · Cited by 59 · cited 1×
“Implicit Bias Consider the nature of bias and use counterfactual explanations as a predictive tool to detect attributes that are assigned and thus plague certain populations [15,29].”
BackgroundThe Use and Misuse of Biomedical Data: Is Bigger Really Better?
Hoffman, S., & Podgurski, A. (2013) · American Journal of Law & Medicine · doi:10.1177/009885881303900401 · Cited by 55 · cited 1×
“This implicit bias runs the risk of becoming encoded into electronic health records (EHRs), which are historically known to house not only inaccurate but deficient data [49].”
BackgroundPaper Matching with Local Fairness Constraints
Kobren, A., Saha, B., & McCallum, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330899 · Cited by 52 · cited 1×
“…through algorithmic modifications that look at machine learning models’ loss of function by integrating “regularization” language specifically for the presumption of fairness, “for clas-sification” [91,92], “for ranking” [93], “for matching tasks” [94,95], and “for recommender systems” [96].”
BackgroundMitigating Bias in Algorithmic Systems—A Fish-eye View
Orphanou, K., Otterbacher, J., Kleanthous, S., Batsuren, K., Giunchiglia, F., Bogina, V., Tal, A. S., Hartman, A., & Kuflik, T. (2023) · ACM Computing Surveys · doi:10.1145/3527152 · Cited by 52 · cited 1×
“Algorithmic bias (AB), on the other hand, is even more murky in its approach, and addressing AB is a dilemma long sought after across a continuum of scientific fields [89].”
BackgroundArtificial Intelligence Bias in Health Care: Web-Based Survey
Vorisek, C. N., Stellmach, C., Mayer, P. J., Klopfenstein, S. A. I., Bures, D., Diehl, A., Henningsen, M., Ritter, K., & Thun, S. (2023) · Journal of Medical Internet Research · doi:10.2196/41089 · Cited by 51 · cited 2×
“There is overwhelming evidence that there are biases in various artificial intelligence models (AIMs) applied to machine learning algorithms (MLAGs) used in health care and other industries [7,16,35–39].”
BackgroundElkin, P. L., Mullin, S., Mardekian, J., Crowner, C., Sakilay, S., Sinha, S., Brady, G., Wright, M., Nolen, K., Trainer, J., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/28946 · Cited by 49
Self-citationGender medicine: its historical roots
Lippi, D., Bianucci, R., & Donell, S. T. (2020) · Postgraduate Medical Journal · doi:10.1136/postgradmedj-2019-137452 · Cited by 48 · cited 1×
“In a similar manner, the presence of gender disparities in health care has been well documented for centuries [18,66–68].”
BackgroundThe Dangers of Human-Like Bias in Machine-Learning Algorithms
Fuchs, D. J. (2018) · Cited by 47
Worrall-Carter, L., Ski, C., Scruth, E., Campbell, M., & Page, K. (2011) · Nursing & Health Sciences · doi:10.1111/j.1442-2018.2011.00645.x · Cited by 43
Fairness of Machine Learning Algorithms for Predicting Foregone Preventive Dental Care for Adults
Schuch, H. S., Borba, M. F., Silva, G. F. D. S., Kawachi, I. I., Filho, A. D. P. C., & Elani, H. W. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.41625 · Cited by 42
A Graphical Catalog of Threats to Validity: Linking Social Science with Epidemiology
Matthay, E. C., & Glymour, M. M. (2020) · Epidemiology · doi:10.1097/ede.0000000000001161 · Cited by 41
(2022) · Journal of Biomed Research · doi:10.46439/biomedres.3.025 · Cited by 40
No Computation without Representation: Avoiding Data and Algorithm Biases through Diversity
Kuhlman, C., Jackson, L., & Chunara, R. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3411074 · Cited by 39
Sharun, K., Banu, S. A., Pawde, A. M., Kumar, R., Akash, S., Dhama, K., & Pal, A. (2023) · Annals of Medicine & Surgery · doi:10.1097/ms9.0000000000001228 · Cited by 38 · cited 1×
“When the limits of MLAGs are “stretched out”, the results are unpredictable outputs, also known as “generative AI hallucinations”, which could impact patient care [10].”
BackgroundRisk of Bias and Error From Data Sets Used for Dermatologic Artificial Intelligence
Tschandl, P. (2021) · JAMA Dermatology · doi:10.1001/jamadermatol.2021.3128 · Cited by 35 · cited 1×
“…machine learning algorithms may lead to missed diagnoses and increased mortality rates from the most deadly form of skin cancer (melanoma), in dark-skinned individuals, especially Black patients, as these models have not been adequately trained or evaluated for these populations [54,62–65].”
BackgroundAnalysis of Gender-Based Inequality in Cardiovascular Health: An Umbrella Review
Bosomworth, J., & Khan, Z. U. (2023) · Cureus · doi:10.7759/cureus.43482 · Cited by 34 · cited 1×
“In a similar manner, the presence of gender disparities in health care has been well documented for centuries [18,66–68].”
BackgroundChest Pain in Women: Considerations From the 2021 AHA/ACC Chest Pain Guideline
Kaur, G., Oliveira-Gomes, D. D., Rivera, F. B., & Gulati, M. (2023) · Current Problems in Cardiology · doi:10.1016/j.cpcardiol.2023.101697 · Cited by 32 · cited 1×
“…MLAGs that predict cardiovascular disease often utilize larger proportions of men in their training data [69], yet women are more likely to present with atypical features of cardiovascular disease [70,71], and cardiovascular disease is more often misdiagnosed in women as compared to men [72,73].”
BackgroundModel-based and actual independence for fairness-aware classification
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2017) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-017-0534-x · Cited by 31 · cited 1×
“…through algorithmic modifications that look at machine learning models’ loss of function by integrating “regularization” language specifically for the presumption of fairness, “for clas-sification” [91,92], “for ranking” [93], “for matching tasks” [94,95], and “for recommender systems” [96].”
BackgroundProposal of a novel Artificial Intelligence Distribution Service platform for healthcare
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K. M., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.36775.1 · Cited by 28
Emerging misunderstood presentations of cardiovascular disease in young women
Bullock-Palmer, R. P., Shaw, L. J., & Gulati, M. (2019) · Clinical Cardiology · doi:10.1002/clc.23165 · Cited by 19
Fairly Predicting Graft Failure in Liver Transplant for Organ AssigningPreprint
Ding, S., Tang, R., Zha, D., Zou, N., Zhang, K., Jiang, X., & Hu, X. (2023) · PubMed Central · doi:10.48550/arxiv.2302.09400 · Cited by 19 · cited 1×
“There is overwhelming evidence that there are biases in various artificial intelligence models (AIMs) applied to machine learning algorithms (MLAGs) used in health care and other industries [7,16,35–39].”
BackgroundLi, A., Mullin, S., & Elkin, P. L. (2024) · JMIR Medical Informatics · doi:10.2196/42271 · Cited by 17
Self-citationLee, D. N., Hutchens, M. J., George, T. J., Wilson-Howard, D. S., Cooks, E., & Krieger, J. L. R. (2022) · Medical Education Online · doi:10.1080/10872981.2022.2107470 · Cited by 13 · cited 1×
“Socioeconomic status (SES) Bias Work to limit the misunderstanding between patients and providers due to cultural and linguistic differences; keep the positive aspects of the social determinants of health (SDoH) in mind [12,30–32].”
BackgroundHTP-NLP: A New NLP System for High Throughput Phenotyping.
Schlegel, D. R., Crowner, C., Lehoullier, F., & Elkin, P. L. (2017) · PubMed · Cited by 12
Self-citationGeskey, J. M., Kodish-Wachs, J., Blonsky, H., Hohman, S. F., & Meurer, S. (2023) · American Journal of Medical Quality · doi:10.1097/jmq.0000000000000112 · Cited by 12
Clinical Artificial Intelligence
McDermott, M. B. A., Nestor, B., & Szolovits, P. (2022) · Clinics in Laboratory Medicine · doi:10.1016/j.cll.2022.09.004 · Cited by 7 · cited 1×
“This type of narrow, non-inclusive data selection process puts forth a challenge for the use of AIMs in developing nations because AIMs used in MLAGs tend to perform best on populations that are like their training data [8,52].”
Resnick, M., LeHouillier, F., Brown, S. H., Campbell, K. E., Montella, D., & Elkin, P. L. (2021) · Studies in health technology and informatics · doi:10.3233/shti210822 · Cited by 6
Self-citationPreCoF: Counterfactual Explanations for FairnessPreprint
Goethals, S., Martens, D., & Calders, T. (2022) · Research Square · doi:10.21203/rs.3.rs-1405346/v1 · Cited by 6
Challenging Race-Based Medicine Through Historical Education About the Social Construction of Race
Skinner-Dorkenoo, A. L., Rogbeer, K. G., Sarmal, A., Ware, C., & Zhu, J. (2023) · Health Equity · doi:10.1089/heq.2023.0036 · Cited by 6
4 Analysis of gender-based inequality in cardiovascular health: an umbrella review
Bosomworth, J. (2022) · Heart · doi:10.1136/heartjnl-2022-bacpr.4 · Cited by 4
Reducing Age Bias in Machine Learning: An Algorithmic Approach
de Alford, A. S. G., Hayden, S. K., Wittlin, N., & Atwood, A. K. (2020) · SMU Scholar (Southern Methodist University) · Cited by 3
ACORN SDOH survey: Terminological representation for use with NLP and CDS
Resnick, M., Montella, D., Brown, S. H., & Elkin, P. L. (2024) · Journal of Clinical and Translational Science · doi:10.1017/cts.2024.24 · Cited by 2 · cited 1×
“…although these are important aspects of other types of bias, but rather to describe the problems associated with gathering, generating, processing, training, and evaluating data that might lead to preconceived notions or prejudices and discrimination on the basis of sociodemographic features [2–6].”
BackgroundSelf-citation
Research works printed but not linked: 2
- Hellström, T., Dignum, V., and Bensch, S. (2020). Bias in Machine Learning—What Is It Good For?. arXiv.
- Liyanage, 2023, Ethical considerations and potential risks in the deployment of large Language Models in diverse societal contexts, J. Comput. Soc. Dyn., 8, 15
Cited by
Mohammed, S. A. A. Q., Osman, Y. M. M., Ibrahim, A. M., & Shaban, M. (2025) · International Nursing Review · doi:10.1111/inr.70010 · Cited by 105
Alhumaidi, N. H., Dermawan, D., Kamaruzaman, H. F., & Alotaiq, N. (2025) · JMIR Medical Informatics · doi:10.2196/68898 · Cited by 109
“By incorporating fairness-aware ML models and building representative datasets, health care practitioners can ensure that ML applications benefit all patient groups, regardless of demographics [20].”
Diet Quality and Caloric Accuracy in AI-Generated Diet Plans: A Comparative Study Across Chatbots
Kaçar, H. K., Kaçar, Ö. F., & Avery, A. (2025) · Nutrients · doi:10.3390/nu17020206 · Cited by 49
“This disparity could arise from algorithmic biases informed by societal norms or training datasets that emphasise greater dietary diversity for women [56,57], possibly due to their unique nutritional requirements during reproductive years or broader dietary trends [58].”
Advancing clinical biochemistry: addressing gaps and driving future innovations
Cao, H., Oghenemaro, E. F., latypova, A., Abosaoda, M. K., Zaman, G. S., & Devi, A. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1521126 · Cited by 47
“Explainability holds great importance in the context of clinical responsibility (192, 193).”
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
Metadata
- Authors
- Franklin, G., Stephens, R., Piracha, M., Tiosano, S., Lehouillier, F., Koppel, R., & Elkin, P.
- Affiliations
- United States Department of Veterans Affairs (US)
- University at Buffalo, State University of New York (US)
- Corresponding author
- Elkin, Peter
- Venue
- Life
- Publisher
- MDPI AG
- Published
- 2024-05-21
- Volume
- 14
- Issue
- 6
- Pages
- 652
- ISSN
- 2075-1729 (ISSN-L)
- PMC
- PMC11204917
- OpenAlex
- W4398169659
- Cited by
- 76
- References
- 95
- Citations by year
- 2026: 29 · 2025: 43 · 2024: 4
- FWCI
- 31.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Life (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- NIH NLM (T15LM012495, R25LM014213, NIAAA R21AA026954, R33AA0226954, NCATS UL1TR001412)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- algorithmic bias, sociodemographic bias, health care disparities, large language models, training data bias, gender bias, racial and ethnic bias, socioeconomic status, insurance status, implicit bias, sampling bias, anchoring effect, clinical decision-making, knowledge distillation, retrieval-augmented generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
![Figure 1 A taxonomy of sociodemographic bias in the clinical machine learning pipeline [13,14,15,16,17,18,19,20].](https://figures.cochat.ai/paper-figures/48764eb2a07844ccaefd605135a6e2d0/the-sociodemographic-biases-in-machine-learning-algorithms-a-biomedical-figure-1.jpg)