CoChat
Sign Up
Cited by 61Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)

A scoping review of fair machine learning techniques when using real-world data

Open access
Huang, Yu, Guo, Jingchuan, Chen, Wei-Han, Lin, Hsin-Yueh, Tang, Huilin, Wang, Fei, Xu, Hua, Bian, Jiang·2024·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2024.104622PMC PMC11146346

Published version of a preprint

Open this paper in CoChat — free account

Abstract

OBJECTIVE: The integration of artificial intelligence (AI) and machine learning (ML) in health care to aid clinical decisions is widespread. However, as AI and ML take important roles in health care, there are concerns about AI and ML associated fairness and bias. That is, an AI tool may have a disparate impact, with its benefits and drawbacks unevenly distributed across societal strata and subpopulations, potentially exacerbating existing health inequities. Thus, the objectives of this scoping review were to summarize existing literature and identify gaps in the topic of tackling algorithmic bias and optimizing fairness in AI/ML models using real-world data (RWD) in health care domains. METHODS: We conducted a thorough review of techniques for assessing and optimizing AI/ML model fairness in health care when using RWD in health care domains. The focus lies on appraising different quantification metrics for accessing fairness, publicly accessible datasets for ML fairness research, and bias mitigation approaches. RESULTS: We identified 11 papers that are focused on optimizing model fairness in health care applications. The current research on mitigating bias issues in RWD is limited, both in terms of disease variety and health care applications, as well as the accessibility of public datasets for ML fairness research. Existing studies often indicate positive outcomes when using pre-processing techniques to address algorithmic bias. There remain unresolved questions within the field that require further research, which includes pinpointing the root causes of bias in ML models, broadening fairness research in AI/ML with the use of RWD and exploring its implications in healthcare settings, and evaluating and addressing bias in multi-modal data. CONCLUSION: This paper provides useful reference material and insights to researchers regarding AI/ML fairness in real-world health care data and reveals the gaps in the field. Fair AI/ML in health care is a burgeoning field that requires a heightened research focus to cover diverse applications and different types of RWD.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2024.104622

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Others such as Huang et al. [9] and Drabiak et al. [10] delve into the application of real-world data and the ethical challenges of ML, but rarely address whether these models meaningfully improve or exacerbate health disparities.

    “Similarly, while others such as Huang et al. [9] and Drabiak et al. [10] delve into the application of real-world data and the ethical challenges of ML, they rarely address whether these models meaningfully improve or exacerbate health disparities.”

    From Advancing Health Equity through Multi-Level Fairness in Health Informatics (2026) · cites this paper as [9]

    The Abstract (p. 2) confirms that Huang et al. review fairness and bias in ML models using real-world healthcare data. But the paper does address potential effects on disparities: the Abstract warns that AI may exacerbate health inequities, and the Discussion (p. 18) considers fairness metrics for assessing disparities. It does not establish whether deploying these models meaningfully changes health disparities in practice, so that narrower criticism remains supported. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. [1]1. Pudjihartono N, Fadason T, Kempa-Liehr AW, O'Sullivan JM. A Review of Feature Selection Methods for Machine Learning-Based Disease Risk Prediction. Front Bioinform . 2022;2:927312.
  2. [2]2. Suri JS, Bhagawati M, Paul S, et al. Understanding the bias in machine learning systems for cardiovascular disease risk assessment: The first of its kind review. Comput Biol Med . 2022;142:105204.
  3. [3]3. Li Q, Yang X, Xu J, et al. Early prediction of Alzheimer's disease and related dementias using real-world electronic health records. Alzheimers Dement . Published online February 23, 2023. doi:10.1002/alz.12967 · DOI · Paper page
  4. [4]4. Kononenko I. Machine learning for medical diagnosis: history, state of the art and perspective. Artif Intell Med . 2001;23(1):89-109.
  5. [5]5. Bakator M, Radosav D. Deep Learning and Medical Diagnosis: A Review of Literature. Multimodal Technologies and Interaction . 2018;2(3):47.
  6. [6]6. Petrova-Antonova D, Spasov I, Krasteva I, Manova I, Ilieva S. A Digital Twin Platform for Diagnostics and Rehabilitation of Multiple Sclerosis. In: Computational Science and Its Applications - ICCSA 2020 . Springer International Publishing; 2020:503-518.
  7. [7]7. Battineni G, Sagaro GG, Chinatalapudi N, Amenta F. Applications of Machine Learning Predictive Models in the Chronic Disease Diagnosis. J Pers Med . 2020;10(2). doi:10.3390/jpm10020021 · DOI
  8. [8]8. Ahsan MM, Luna SA, Siddique Z. Machine-Learning-Based Disease Diagnosis: A Comprehensive Review. Healthcare (Basel) . 2022;10(3). doi:10.3390/healthcare10030541 · DOI · Paper page
  9. [11]11. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence - what is it and what can it tell us? N Engl J Med . 2016;375(23):2293-2297.
  10. [12]12. Concato J, Corrigan-Curay J. Real-world evidence - where are we now? N Engl J Med . 2022;386(18):1680-1682.
  11. [15]15. Shamout F, Zhu T, Clifton DA. Machine Learning for Clinical Outcome Prediction. IEEE Rev Biomed Eng . 2021;14:116-126.
  12. [17]17. Perveen S, Shahbaz M, Keshavjee K, Guergachi A. A Systematic Machine Learning Based Approach for the Diagnosis of Non-Alcoholic Fatty Liver Disease Risk and Progression. Sci Rep . 2018;8(1):2112.
  13. [18]18. Chatterjee A, Gerdes MW, Martinez SG. Identification of Risk Factors Associated with Obesity and Overweight-A Machine Learning Overview. Sensors . 2020;20(9):2734.
  14. [20]20. Xu J, Xiao Y, Wang WH, et al. Algorithmic fairness in computational medicine. eBioMedicine . 2022;84. doi:10.1016/j.ebiom.2022.104250 · DOI · Paper page
  15. [24]24. Li Y, Wang H, Luo Y. Improving Fairness in the Prediction of Heart Failure Length of Stay and Mortality by Integrating Social Determinants of Health. Circ Heart Fail . 2022;15(11):e009473.
  16. [25]25. Park Y, Hu J, Singh M, et al. Comparison of Methods to Reduce Bias From Clinical Prediction Models of Postpartum Depression. JAMA Netw Open . 2021;4(4):e213909.
  17. [26]26. Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A. A Survey on Bias and Fairness in Machine Learning. ACM Comput Surv . 2021;54(6):1-35.
  18. [29]29. Wan M, Zha D, Liu N, Zou N. In-Processing Modeling Techniques for Machine Learning Fairness: A Survey. ACM Trans Knowl Discov Data . 2023;17(3):1-27.
  19. [33]33. Corbett-Davies S, Pierson E, Feller A, Goel S, Huq A. Algorithmic Decision Making and the Cost of Fairness. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . KDD '17. Association for Computing Machinery; 2017:797-806.
  20. [34]34. Feldman M, Friedler SA, Moeller J, Scheidegger C, Venkatasubramanian S. Certifying and Removing Disparate Impact. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . KDD '15. Association for Computing Machinery; 2015:259-268.
  21. [35]35. Foryciarz A, Pfohl SR, Patel B, Shah N. Evaluating algorithmic fairness in the presence of clinical guidelines: the case of atherosclerotic cardiovascular disease risk estimation. BMJ Health Care Inform . 2022;29(1):e100460.
  22. [36]36. Paul W, Hadzic A, Joshi N, Alajaji F, Burlina P. TARA: Training and Representation Alteration for AI Fairness and Domain Generalization. Neural Comput . 2022;34(3):716-753.
  23. [37]37. Speicher T, Heidari H, Grgic-Hlaca N, et al. A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . KDD '18. Association for Computing Machinery; 2018:2239-2248.
  24. [38]38. Chhabra A, Masalkovaite K, Mohapatra P. An overview of fairness in clustering. IEEE Access . 2021;9:130698-130720.
  25. [39]39. Ter-Minassian L, Viani N, Wickersham A, et al. Assessing machine learning for fair prediction of ADHD in school pupils using a retrospective cohort study of linked education and healthcare data. BMJ Open . 2022;12(12):e058058.
  26. [40]40. Wang R, Chaudhari P, Davatzikos C. Bias in machine learning models can be significantly mitigated by careful training: Evidence from neuroimaging studies. Proceedings of the National Academy of Sciences . 2023;120(6):e2211613120.
  27. [41]41. Barda N, Yona G, Rothblum GN, et al. Addressing bias in prediction models by improving subpopulation calibration. J Am Med Inform Assoc . 2021;28(3):549-558.
  28. [42]42. Li F, Wu P, Ong HH, Peterson JF, Wei WQ, Zhao J. Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction. J Biomed Inform . 2023;138:104294.
  29. [43]43. Davoudi A, Sajdeya R, Ison R, et al. Fairness in the prediction of acute postoperative pain using machine learning models. Front Digit Health . 2022;4:970281.
  30. [44]44. Jeanselme V, De-Arteaga M, Zhang Z, Barrett J, Tom B. Imputation Strategies Under Clinical Presence: Impact on Algorithmic Fairness. In: Parziale A, Agrawal M, Joshi S, et al., eds. Proceedings of the 2nd Machine Learning for Health Symposium . Vol 193. Proceedings of Machine Learning Research. PMLR; 2022:1234.
  31. [45]45. Thompson HM, Sharma B, Bhalla S, et al. Bias and fairness assessment of a natural language processing opioid misuse classifier: detection and mitigation of electronic health record data disadvantages across racial subgroups. J Am Med Inform Assoc . 2021;28(11):2393-2403.
  32. [46]46. Johnson AEW, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Sci Data . 2016;3:160035.
  33. [53]53. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-Ray8: Hospitalscale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE; 2017:2097-2106.
  34. [55]55. Jay MA, McGrath-Lone L, Gilbert R. Data Resource: the National Pupil Database (NPD). Int J Popul Data Sci . 2019;4(1):1101.
  35. [56]56. Downs J, Gilbert R, Hayes RD, Hotopf M, Ford T. Linking health and education data to plan and evaluate services for children. Arch Dis Child . 2017;102(7):599-602.
  36. [57]57. Downs JM, Ford T, Stewart R, et al. An approach to linking education, social care and electronic health records for children and young people in South London: a - linkage study of child and adolescent mental health service data. BMJ Open . 2019;9(1):e024355.
  37. [61]61. Hebert-Johnson U, Kim M, Reingold O, Rothblum G. Multicalibration: Calibration for the (Computationally-Identifiable) Masses. In: Dy J, Krause A, eds. Proceedings of the 35th International Conference on Machine Learning . Vol 80. Proceedings of Machine Learning Research. PMLR; 10--15 Jul 2018:1939-1948.
  38. [62]62. Kleinberg J, Mullainathan S, Raghavan M. Inherent trade-offs in the fair determination of risk scores. arXiv [csLG] . Published online September 19, 2016. http://arxiv.org/abs/1609.05807 · arXiv
  39. [63]63. Saraswat D, Bhattacharya P, Verma A, et al. Explainable AI for Healthcare 5.0: Opportunities and Challenges. IEEE Access . 2022;10:84486-84517.
  40. [65]65. Payrovnaziri SN, Chen Z, Rengifo-Moreno P, et al. Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review. J Am Med Inform Assoc . 2020;27(7):1173-1185.
  41. [66]66. Lundberg S, Lee SI. A unified approach to interpreting model predictions. arXiv [csAI] . Published online May 22, 2017. Accessed January 15, 2023. https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67 767-Abstract.html
  42. [67]67. Meng Y, Speier W, Ong M, Arnold CW. HCET: Hierarchical Clinical Embedding With Topic Modeling on Electronic Health Records for Predicting Future Depression. IEEE J Biomed Health Inform . 2021;25(4):1265-1272.
  43. [68]68. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In: 2017 IEEE International Conference on Computer Vision (ICCV) . IEEE; 2017:618-626.
  44. [69]69. He Z, Tang X, Yang X, et al. Clinical trial generalizability assessment in the big data era: A review. Clin Transl Sci . 2020;13(4):675-684.
  45. [70]70. Forrest CB, McTigue KM, Hernandez AF, et al. PCORnet® 2020: current state, accomplishments, and future directions. J Clin Epidemiol . 2021;129:60-67.
  46. [73]73. Booth BM, Hickman L, Subburaj SK, Tay L, Woo SE, D'Mello SK. Bias and Fairness in Multimodal Machine Learning: A Case Study of Automated Video Interviews. In: Proceedings of the 2021 International Conference on Multimodal Interaction . ICMI '21. Association for Computing Machinery; 2021:268-277. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. medRxiv preprint doi: https://doi.org/10.1101/2024.03.03.24303669; this version posted March 4, 2024. The copyright holder for this preprint MIMIC-III (Medical Information Mart for Intensive Care III), iSTAGING (Imaging-based coordinate SysTem for AGing and NeurodeGenerative diseases), GWTG-HF (Get With The Guidelines-Heart Failure), (NPD) National Pupil Database linked with the SLaM CAMHS (South London and Maudsley National Health Service Foundation Trust Child and Adolescent Mental Health Services) Figure 1. The overall search and screening process. <!-- image --> <!-- image --> *Non-public dataset All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. medRxiv preprint doi: https://doi.org/10.1101/2024.03.03.24303669; this version posted March 4, 2024. The copyright holder for this preprint Figure 2. Fair machine learning in health care. Figure 3. Distributions of data types of the real-world datasets used for fair ML research in health care. <!-- image --> Table 1. Overview of the existing bias mitigation approaches. · DOI

Cited by

Sorted by citations

Authors
Huang, Y., Guo, J., Chen, W.-H., Lin, H.-Y., Tang, H., Wang, F., Xu, H., & Bian, J.
Affiliations
  • University of Florida Health (US)
  • University of Florida (US)
  • Cornell University (US)
  • Yale University (US)
Corresponding author
Bian, Jiang
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2024-03
Volume
151
Pages
104622
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2024.104622
PMC
PMC11146346
OpenAlex
W4392515337
Cited by
61
References
15
Citations by year
2026: 28 · 2025: 18 · 2024: 15
FWCI
4.64
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • Digital Commons-TMC (Texas Medical Center) (submittedVersion)
Funders
  • Patient-Centered Outcomes Research Institute (ME-2018C3-14754)
  • National Center for Advancing Translational Sciences (UL1TR001427)
  • Centers for Disease Control and Prevention (U18DP006512)
  • National Institute of Allergy and Infectious Diseases (R01AI172875)
  • National Institute on Aging (RF1AG084178, R01AG080624, R01AG083039)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
algorithmic fairness, algorithmic bias, real-world data, health care, fairness metrics, bias mitigation, health disparities, clinical decision support, multimodal data
MeSH terms
Machine Learning, Artificial Intelligence, Humans, Research Personnel, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review