Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects

In the paper

“Medical publication trends indicate a growing interest in bias in AI (Figure 1).”

In the paper

“Some could be highly accurate but also costly and poorly available, whereas others could neglect intermediate findings or be operator-dependent,27 potentially reducing label applicability and reliability.”

What the sources say

• Not checked yetcites [27] Chen PC Mermel CH Liu Y Evaluation of artificial intelligence on a reference standard based on subjective interpretation. Lancet Digit Health 2021 3 11 e693 e695 10.1016/S2589-7500(21)00216-8 34561202 source

In the paper

“In particular, this bias can lead to skewed data cohorts, incomplete information, heightened uncertainty in outcomes, and potential dismissal of algorithm-assisted medical advice.40”

What the sources say

• Not checked yetcites [40] Olteanu A Castillo C Diaz F Kıcıman E Social data: biases, methodological pitfalls, and ethical boundaries. Front Big Data 2019 2 13 10.3389/fdata.2019.00013 33693336 PMC7931947 source

In the paper

“Specifically, it arises when the correlation between input variables, such as images, and output predictions, such as diagnoses, evolves due to fluctuations in data, such as variations in image acquisition hardware or protocols, shifts in disease prevalence, or advancements in gold-standard technologies.38”

What the sources say

• Not checked yetcites [38] Roland T Böck C Tschoellitsch T et al Domain shifts in machine learning based Covid-19 diagnosis from blood tests. J Med Syst 2022 46 5 23 35348909 PMC8960704 source

In the paper

“The concept of bias in machine learning (ML) research and more generally in the field of predictive modeling is intrinsically tied to the concept of variance.14”

What the sources say

• Not checked yetcites [14] Geman S Bienenstock E Doursat R Neural networks and the bias/variance dilemma. Neural Comput 1992 4 1 1 58 10.1162/neco.1992.4.1.1 source

In the paper

“Uncertainty bias encompasses the influence of uncertainty on decision-making stemming from AI/ML models.39”

What the sources say

• Not checked yetcites [39] Drukker K Chen W Gichoya J et al Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment. J Med Imaging(Bellingham) 2023 10 6 061104 10.1117/1.jmi.10.6.061104 37125409 PMC10129875 source

In the paper

“From a mathematical point of view, noise (the joint probability distribution between training and test/inference samples), bias, and variance are the three components that lead to model performance degradation and negatively affect generalization to new data.17”

What the sources say

• Not checked yetcites [17] Guan X Burton H Bias-variance tradeoff in machine learning: theoretical formulation and implications to structural engineering applications. Structures 2022 46 17 30 10.1016/j.istruc.2022.10.004 source

In the paper

“Behavioral bias pertains to the potential distortions in user behavior seen across various platforms, contexts, or datasets.39”

What the sources say

• Not checked yetcites [39] Drukker K Chen W Gichoya J et al Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment. J Med Imaging(Bellingham) 2023 10 6 061104 10.1117/1.jmi.10.6.061104 37125409 PMC10129875 source

In the paper

“However, it should be made clear that these two entities are interdependent, and reducing one (e.g., variance) typically comes at the expense of increasing the other (i.e., bias), which gives birth to the concept of a bias-variance tradeoff.”

In the paper

“A deployment bias emerges when there is a misalignment between the envisioned purpose of a system or algorithm and its actual application.36”

What the sources say

• Not checked yetcites [36] Rajpurkar P Chen E Banerjee O Topol EJ AI in health and medicine. Nat Med 2022 28 1 31 38 10.1038/s41591-021-01614-0 35058619 source

In the paper

“This starts with a code review that can be carried out by an independent experienced coder/auditor.49”

What the sources say

• Not checked yetcites [49] Murphy-Hill E Jaspan CN Egelman C Cheng L The Pushback effects of race, ethnicity, gender, and age in code review. Commun ACM 2022 65 3 52 57 10.1145/3474097 source

In the paper

“Apart from collecting more data, strategies at the data processing stage may include the creation of more representative training datasets by data augmentation (e.g., by specifically adding underrepresented examples to the data through additional sampling or data generation) or data filtering (e.g., actively undersampling or filtering out undesirable or non-representative samples).68”

What the sources say

• Not checked yetcites [68] Gallegos IO, Rossi RA, Barrow J, et al. Bias and Fairness in large language models: a survey. Published online 2023. source

In the paper

“The composition of the involved teams can influence the AI’s propensity for bias.”

In the paper

“By incorporating the above-mentioned considerations early into the design phase, developers can create systems that are less likely to perpetuate or amplify biases.”

In the paper

“The way data are presented to the model during training (affected by the problem formulation and the labeling methodology) and how model parameters are updated (defined through training setup including the objective function) can introduce bias into the model.13, 68”

What the sources say

• Not checked yetcites [13] Obermeyer Z Powers B Vogeli C Mullainathan S Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019 366 6464 447 453 10.1126/science.aax2342 31649194 source
• Not checked yetcites [68] Gallegos IO, Rossi RA, Barrow J, et al. Bias and Fairness in large language models: a survey. Published online 2023. source

In the paper

“Additionally, algorithms can be categorized according to whether they explicitly or implicitly address bias by accessing or not accessing the bias variables during training.77”

What the sources say

• Not checked yetcites [77] Shrestha R, Kafle K, Kanan C. An investigation of critical issues in bias mitigation techniques. In:IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE; Waikoloa, HI, USA: 2022:2512-2523.

In the paper

“Guiding principles, such as transparency, fairness, non-maleficence, and respect for privacy from the outset, are widely included in recommendations and position papers and can help to prevent bias (Figure 9).67”

What the sources say

• Not checked yetcites [67] Jobin A Ienca M Vayena E The global landscape of AI ethics guidelines. Nat Mach Intell 2019 1 9 389 399

In the paper

“Additionally, these are independent of the model and can be used in a black-box setting by altering the data distribution to increase model fairness.76”

What the sources say

• Not checked yetcites [76] Feldman T, Peake A. End-to-end bias mitigation: removing gender bias in deep learning. Published online June 20, 2021. source

In the paper

“Bias mitigation algorithms can be divided into three types according to the phase in which they are applied: in a preprocessing phase, during model training, or after model training.76”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [76] Feldman T, Peake A. End-to-end bias mitigation: removing gender bias in deep learning. Published online June 20, 2021.

In the paper

“Re-sampling and re-weighting algorithms focus on rebalancing the class distribution by adjusting the sample probability/loss weight for majority/minority samples.78, 79, 80, 81, 82, 83”

What the sources say

• Not checked yetcites [78] Feldman M, Friedler SA, Moeller J, Scheidegger C, Venkatasubramanian S. Certifying and removing disparate impact. In:Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’15. Association for Computing Machinery; 2015:259-268. source
• Not checked yetcites [79] Cui Y, Jia M, Lin TY, Song Y, Belongie S. Class-balanced loss based on effective number of samples. In:IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: 2019:9260-9269. source
• Not checked yetcites [80] Zou Y, Yu Z, Kumar BVKV, Wang J. Domain adaptation for semantic segmentation via class-balanced self-training. Published online October 25, 2018. source

In the paper

“Nonetheless, the current version of PROBAST is not suitable for the evaluation of ML studies, and this is the reason that the PROBAST group has initiated the process of developing an AI-specific version of PROBAST called PROBAST-AI, which is still under development.61”

What the sources say

• Not checked yetcites [61] Collins GS Dhiman P Andaur Navarro CL et al Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open 2021 11 7 e048008 10.1136/bmjopen-2020-048008 PMC8273461 34244270 source

In the paper

“Consider this scenario: AI/ML models can be “confidently wrong,” meaning they may yield incorrect outcomes with a high level of certainty.”

In the paper

“Transparent and comprehensive documentation of the AI system’s design choices, data sources, and any assumptions made during development (e.g., through model cards)75 is crucial and can help spot sources of bias before, during, and after training.”

What the sources say

• Not checked yetcites [75] Mitchell M, Wu S, Zaldivar A, et al. Model cards for model reporting. In:Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM; 2019:220-229.

In the paper

“Teams that are diverse in terms of gender, ethnicity, culture, and professional background bring a wide array of perspectives to the table, which can help identify and eliminate potential biases early in the development process.”

In the paper

“In general, XAI methods can be categorized into two main groups: perturbation-based and backpropagation-based explanations.”

In the paper

“Algorithmic aversion refers to a phenomenon where clinicians or healthcare professionals exhibit reluctance or skepticism toward relying on AI algorithms for making diagnostic or treatment decisions in medical imaging.46”

What the sources say

• Not checked yetcites [46] Cadario R Longoni C Morewedge CK Understanding, explaining, and utilizing medical artificial intelligence. Nat Hum Behav 2021 5 12 1636 1642 34183800 source

In the paper

“Reporting guidelines, such as the Fairness Universality Traceability Usability Robustness Explainability-AI and TRIPOD-AI, can assist authors of AI manuscripts in reporting their studies according to the Fairness principle, promoting the identification of bias sources.58, 63, 64”

What the sources say

• Not checked yetcites [58] Klontzas ME Gatti AA Tejani AS Kahn CE Jr AI reporting guidelines: how to select the best one for your research. Radiol Artif Intell 2023 5 3 e230055 10.1148/ryai.230055 37293341 PMC10245184 source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [63] Lekadir K, Osuala R, Gallin C, et al. FUTURE-AI: Guiding principles and consensus recommendations for trustworthy artificial intelligence in medical imaging. Published online October 30, 2023.
• Not checked yetcites [64] Collins GS Moons KGM Dhiman P et al TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 2024 385 7996 e078378 10.1038/d41586-024-00173-x 38626948 PMC11019967 source

In the paper

“However, there are several challenges in handling bias that can be related to ambiguities in interpreting results, limited diversity in benchmark datasets, and the subjectivity of detecting bias.”

In the paper

“Automation bias refers to the tendency of individuals to rely excessively on automated systems, such as AI algorithms, and to disregard or underutilize their own judgment or critical thinking skills.42”

What the sources say

• Not checked yetcites [42] Khera R Simon MA Ross JS Automation Bias and Assistive AI: Risk of Harm From AI-driven clinical decision support. JAMA 2023 330 23 2255 2257 10.1001/jama.2023.22557 38112824 source

In the paper

“By identifying and addressing biases, these systems can improve healthcare outcomes and equity.”

In the paper

“Understanding that medical imaging AI systems are sensitive to biases is key for their effective real-world integration into clinical practice.”

In the paper

“Before going into detail, it should be acknowledged that expecting completely bias-free systems may be unrealistic.”

In the paper

“Despite the aspiration for unbiased AI, complete inclusivity of all data types and sources remains an unattainable goal in model development.”

In the paper

“Concerning monitoring, Article 61 of the legislation mandates that developers of high-risk AI systems establish ongoing, systematic post-market surveillance mechanisms.”

In the paper

“Additionally, the lack of diversity among developers and researchers can worsen these issues, as teams may unconsciously favor perspectives similar to their own.”

In the paper

“Subjectivity in the detection of bias can be related to personal interpretation and individual perspectives related to the identification of the bias itself.”

In the paper

“Article 10 mandates that for high-risk AI systems, training, validation, and testing datasets must be relevant, representative, error-minimal, and complete for their intended use.”

In the paper

“The recent European Union’s AI Act serves as a pioneering legal framework aimed at regulating AI use, particularly in high-risk applications such as healthcare (as defined in Article 6).”

In the paper

“For example, AI software trained for adult fracture detection is at risk of erroneous results if applied in a pediatric population.”

In the paper

“Publicly accessible benchmarks are essential for comparison for AI models and represent a crucial element of open science.122”

What the sources say

• Not checked yetcites [122] Bell LC Shimron E Sharing Data Is Essential for the Future of AI in Medical Imaging. Radiol Artif Intell 2024 6 1 e230337 10.1148/ryai.230337 38231036 PMC10831510 source

In the paper

“However, although sharing data is essential for developing robust AI tools, patient privacy when collecting medical information can pose significant challenges.124”

What the sources say

• Not checked yetcites [124] Ferrara E Fairness and bias in artificial intelligence: a brief survey of sources, impacts, and mitigation strategies. Sci 2023 6 1 3 10.3390/sci6010003 source

In the paper

“The 2020 Dutch case against the “system risk indication” tool, which violated privacy laws and wrongly identified innocent people as fraud suspects, underscores the impact of such biases.128”

What the sources say

• Not checked yetcites [128] SyRI legislation in breach of European Convention on Human Rights. Accessed April 11, 2024.

In the paper

“Moreover, different sources of bias can contribute to the generation of bias, including the data source, algorithm, and users, which makes the identification more cumbersome.124”

What the sources say

• Not checked yetcites [124] Ferrara E Fairness and bias in artificial intelligence: a brief survey of sources, impacts, and mitigation strategies. Sci 2023 6 1 3 10.3390/sci6010003 source

In the paper

“Set to fully take effect in 2026, the act governs the development, deployment, and use of AI, ensuring safety, transparency, and adherence to ethical standards across the EU.”

In the paper

“The WHO principles 2 and 5 address bias, mandating that AI tools prioritize human well-being, safety, and public interest.”

What the sources say

• Not checked yetcites [127] Ethics and Governance of Artificial Intelligence for Health: WHO guidance. Published online 2021.

In the paper

“The World Health Organization (WHO) poses six key ethical principles for AI in healthcare in their framework (Figure 9): (1) protect autonomy, (2) promote human well-being, human safety, and the public interest, (3) ensure transparency, explainability, and intelligibility, (4) foster responsibility and accountability, (5) ensure inclusiveness and equity, and (6) promote AI that is responsive and s”

What the sources say

• Not checked yetcites [127] Ethics and Governance of Artificial Intelligence for Health: WHO guidance. Published online 2021.