CoChat
Sign Up
Cited by 279Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIExplainable Artificial Intelligence (XAI)

AI pitfalls and what not to do: mitigating bias in AI

Open access
Gichoya, Judy Wawira, Thomas, Kaesha, Celi, Leo Anthony, Safdar, Nabile, Banerjee, Imon, Banja, John D, Seyyed-Kalantari, Laleh, Trivedi, Hari +1 more·2023·The British Journal of Radiology
DOI 10.1259/bjr.20230023PMC PMC10546443
Open this paper in CoChat — free accountOpen PDF

Abstract

Various forms of artificial intelligence (AI) applications are being deployed and used in many healthcare systems. As the use of these applications increases, we are learning the failures of these models and how they can perpetuate bias. With these new lessons, we need to prioritize bias evaluation and mitigation for radiology applications; all the while not ignoring the impact of changes in the larger enterprise AI deployment which may have downstream impact on performance of AI models. In this paper, we provide an updated review of known pitfalls causing AI bias and discuss strategies for mitigating these biases within the context of AI deployment in the larger healthcare enterprise. We describe these pitfalls by framing them in the larger AI lifecycle from problem definition, data set selection and curation, model training and deployment emphasizing that bias exists across a spectrum and is a sequela of a combination of both human and machine factors.

AI snapshot

Written from the paper’s full text.

TL;DR

This review argues that bias in radiology AI arises across the entire lifecycle—problem definition, data acquisition, model development, validation, and deployment—from human, machine, and systems factors. It recommends mitigation at every stage, including diverse teams and data, rigorous validation, ongoing monitoring, and caution with purely mathematical fairness approaches.

Key findings

  • Bias can persist even when race is excluded from model inputs: a commercial algorithm referred fewer Black patients with similar disease burden than White patients because healthcare costs were used as a proxy for need, learning from a system where Black patients are less likely to seek or receive care.
  • Imaging AI data are geographically and demographically narrow: a review of published AI articles from 2015–2019 found data largely from California, Massachusetts, and New York, with none from 34 states; among 23 chest X-ray datasets, only 8.7% reported race or ethnicity and 4.3% reported insurance status; the UK Biobank has 500,000 patients but only 6% of non-European ancestry, and only 8% has been used for research.
  • Validation gaps are common: among 151 FDA-cleared imaging AI products by November 2021, only 64% used clinical data for validation, 4% had patient demographics, 5% reported machine specifications, and 34% had multi-institutional validation; a systematic review of external validation found 86 studies with a median of 240 cases, about 47% positive.
  • Deployment failures and human–machine interactions matter: the Epic sepsis model at Michigan Medicine identified only 7% of patients whose sepsis diagnosis was missed by a clinician, failed to flag 67% of patients with sepsis, and generated alerts on 18% of all hospitalized patients; in another example, pathologist accuracy worsened when a model’s prediction was incorrect.

Methods

  • Narrative/updated review, not a systematic review; it frames AI bias pitfalls across the AI lifecycle: problem definition, dataset selection and curation, model training and deployment, and post-deployment evaluation and monitoring.
  • Uses cited literature and examples—such as Obermeyer et al., the Epic sepsis model, COVID-19 imaging prediction models, RadImageNet, and federated learning—to illustrate bias sources and mitigation strategies.
  • No new primary data, sample, or statistical analysis are reported; a formal methods section is not stated in the paper.

Limitations

  • The paper does not include a dedicated limitations section; as a narrative review, it lacks systematic search or selection criteria and does not quantitatively synthesize evidence (evident from the text).
  • It relies on selected published examples and author experience, so its coverage may not be exhaustive or generalizable (evident from the text).
  • It states field-level limitations: lack of a unified definition of bias, limited model explainability, proprietary “black box inside a black box” models, and difficulty detecting harmful hidden signals in imaging datasets.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1259/bjr.20230023

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 86 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In our own experience, pre-processing of images is important and affects federated model performance.

    “In our own experience, 53 we observed that pre-processing of images is important and affects the federated model performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source

      Not checked in the free run.

  • Findingown data

    Our preliminary work found that signals that identify race are also contained on synthetic data sets.

    “Our preliminary work found that signals that identify race are also contained on synthetic data sets.”
    Quote in the paperSubstantial claim
  • Premisecitation

    COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease.

    “For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source

      Not checked in the free run.

  • Premisecitation

    AI systems have documented real-world failures in healthcare.

    “Despite the promise and hope of artificial intelligence (AI) to improve patient care, several real-world failures of AI systems have been documented.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source

      Not checked in the free run.

    • cites [2] Seyyed-Kalantari, 2021, Underdiagnosis bias of artificial intelligence Algorithms applied to chest Radiographs in under-served patient populations, Nat Med source

      Not checked in the free run.

  • Premisecitation

    One input variable to the Epic sepsis model was antibiotic orders by a provider, which is a type of data leakage because an infection is already being considered at that point.

    “Before the subsequent model overhaul as a result of the publication (rather than some planned post-deployment evaluation and monitoring), it was observed that one input variable to the model was antibiotic orders by a provider—a type of data leakage since an infection is already being considered at that point.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source

      Not checked in the free run.

    • cites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source

      Not checked in the free run.

  • May combine two claims. 29
  • Classified as probably not a substantial claim. 21

  • May combine two claims.

    Findingown data

    FedAvg is biased to research sites contributing larger data sets, with poor performance reported for smaller data sets.

    “Moreover, FedAvg is biased to research sites contributing larger data sets with poor performance reported for smaller data sets.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    In our experimental evaluation of the reading race paper, we demonstrated superhuman ability for AI to predict self-reported race across multiple modalities, and we found that some transfer learning tasks are better in race detection.

    “In our experimental evaluation of the reading race paper where we demonstrated superhuman ability for AI to predict self-reported race across multiple modalities, we found that some transfer learning tasks are better in race detection. 28”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Gichoya, 2022, AI recognition of patient race in medical imaging: a Modelling study, Lancet Digit Health source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    When we used the FedAvg algorithm, the model improved its generalizability on external data sets but at the expense of some degradation in internal validation.

    “When we used the FedAvg algorithm, the model improved its generalizability on external data sets but at the expense of some degradation in internal validation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Existing AI data sets are acquired from limited geographical regions and most lack information necessary to evaluate subgroup performance of models.

    “Existing AI data sets are acquired from limited geographical regions; most lack information necessary to evaluate subgroup performance of models.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [22] Kaushal, 2020, Geographic distribution of US cohorts used to train deep learning Algorithms, JAMA source

      Not checked in the free run.

    • cites [23] Yi, 2022, Demographic reporting in publicly available chest radiograph data SETS: opportunities for mitigating sex and racial disparities in deep learning models, J Am Coll Radiol source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Subgroup evaluations of deployed AI models have shown varying performance, with poor performance especially for historically underserved patients.

    “In addition to observing overall performance decline secondary to differences in data distribution after model deployment, subgroup evaluations have shown varying levels of performance with poor performance especially for historically underserved patients. 1,2”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source

      Not checked in the free run.

    • cites [2] Seyyed-Kalantari, 2021, Underdiagnosis bias of artificial intelligence Algorithms applied to chest Radiographs in under-served patient populations, Nat Med source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The length-of-stay algorithm learned that patients from less affluent zip codes are likely to have longer hospital stays and may not benefit from case management facilitation of early discharge.

    “The algorithm (which was abandoned before deployment) learnt that patients from less affluent zip codes are likely to have longer hospital stays and may not benefit from case management facilitation of early discharge.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [16] Nordling, 2019, A fairer way forward for AI in health care, Nature source

      Not checked in the free run.

  • May combine two claims.

    Premisenone given

    Problems in AI data set development include limited patient diversity, limited resolution and missing clinical confounders, limited or suboptimal quality annotations, and lack of data standards and best practice to ensure quality and reproducibility.

    “Problems in data set development include limited patient diversity, limited resolution and missing clinical confounders, limited or suboptimal quality annotations, and lack of data standards and best practice to ensure quality and reproducibility.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The Epic sepsis model was used by 180 customers representing hundreds of hospitals and was subsequently withdrawn.

    “The failure and subsequent withdrawal of the Epic (TM) sepsis model used by 180 customers representing hundreds of hospitals provides a template to study AI failure across a spectrum. 13,14”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source

      Not checked in the free run.

    • cites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Bias permeates every step of the AI lifecycle and is a sequela of human, machine, and systems factors.

    “We emphasize that bias permeates every step of the lifecycle and is a sequela of human, machine, and systems factors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A review of 23 chest X-ray data sets found that although the majority reported age and sex, only 8.7% reported race or ethnicity and 4.3% reported insurance status.

    “A review of 23 CXR data sets found that although the majority reported age and sex, only 8.7% reported race or ethnicity and 4.3% reported insurance status. 23”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [23] Yi, 2022, Demographic reporting in publicly available chest radiograph data SETS: opportunities for mitigating sex and racial disparities in deep learning models, J Am Coll Radiol source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Efforts to mitigate bias in AI are challenging due to lack of a unified definition of bias, focus on statistical and technocentric definitions without involving patients and communities, and post-hoc reflection on bias rather than as a deliverable by design.

    “Efforts to mitigate these biases are challenging due to—lack of a unified definition of bias 10 ; focus on a statistical definition of bias and a technocentric definition of bias without involving patients and communities; and post-hoc reflection on bias rather than as a deliverable by design. 11”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [10] Narayanan  A . Tutorial: 21 fairness definitions and their politics . 2018 . Available from : https://www.youtube.com/watch?v=jIXIuYdnyyk

      Not checked in the free run.

    • cites [11] Wawira Gichoya, 2021, Equity in essence: a call for Operationalising fairness in machine learning for Healthcare, BMJ Health Care Inform source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Upon testing at Michigan Medicine, the Epic sepsis algorithm identified only 7% of patients whose sepsis diagnosis was missed by a clinician, failed to flag 67% of patients with sepsis, and generated alerts on 18% of all hospitalized patients, resulting in alert fatigue.

    “Upon testing at Michigan Medicine, the algorithm only identified 7% of patients whose sepsis diagnosis was missed by a clinician, failed to flag 67% of patients with sepsis despite generating alerts on 18% of all hospitalized patients resulting in alert fatigue.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source

      Not checked in the free run.

    • cites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Model auditing and bias detection, which benefit from diverse perspectives during training and validation, are more difficult in federated learning where only the weights are shared across the groups of modelers.

    “It should be noted that model auditing and bias detection, which benefit from diverse perspectives during training and validation, are more difficult in federated learning where only the weights are shared across the groups of modelers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Some groups have suggested datasheets-a checklist to guide model developers on how to use data-to address the problem of bias in available data sets.

    “Some groups have suggested datasheets 39,40 —a checklist to guide model developers on how to use data—to address this problem.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [39] Gebru  T , Morgenstern  J , Vecchione  B , Vaughan  JW , Wallach  H , Daumé III  H . Datasheets for Datasets . arXiv  2018 .

      Not checked in the free run.

    • cites [40] Garbin  C , Rajpurkar  P , Irvin  J , Lungren  MP , Marques  O . Structured dataset documentation: a datasheet for CheXpert . arXiv  2021 .

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The availability of a radiology-specific data set to facilitate transfer learning is a step towards efficiency in model development.

    “The availability of a radiology specific data set to facilitate transfer learning is a step towards efficiency in model development.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Chest X-rays have been used to predict biological age, cardiac disease outcomes, and healthcare cost at 1, 3, and 5 years.

    “CXRs have also been used to predict biological age, cardiac disease outcomes and healthcare cost at 1, 3 and 5 years. 30”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Sohn, 2022, Prediction of future Healthcare expenses of patients from chest Radiographs using deep learning: a pilot study, Sci Rep source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    For local model performance, personalized federated learning models such as FedBN are preferable, while for external performance, FedAvg is preferable.

    “Overall, when local model performance matters—then personalized federated learning models (such as FedBN) should be prioritized, while for external performance, FedAvg should be used. 53”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisenone given

    Federated learning-whereby models are trained across multiple institutions without the data being shared except for the model weights-has been proposed as a solution to improve model performance across diverse patient groups.

    “Federated learning—whereby models are trained across multiple institutions without the data being shared except for the model weights—has been proposed as a solution to improve model performance across diverse patient groups.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Such a data set will discourage the use of Frankenstein data sets-assembled from other data sets and redistributed under a new name-as pretraining can be done on RadImageNet and fine-tuned on a smaller data set.

    “Such data set will also discourage the use of Frankenstein data sets—assembled from other data sets and redistributed under a new name—as pretraining can be done on RadImageNet and fine-tuned on a smaller data set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Class imbalance also occurs due to distribution of positive versus negative cases and across various demographic distributions within data sets.

    “Beyond individual images, class imbalance also occurs due to distribution of positive versus negative cases (since most studies are normal), and across various demographic distributions within data sets.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Class imbalance occurs frequently in radiology.

  • May combine two claims.

    Premisecitation

    Only 8% of the UK Biobank has been used for research, with most of this research focused on patients of European ancestry.

    “In fact, only 8% of the UK Biobank has been used for research, with most of this research focused on patients of European ancestry.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Privé, 2022, Using the UK Biobank as a global reference of worldwide populations: application to measuring ancestry diversity from GWAS summary Statistics, Bioinformatics source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Fusion models have a larger feature space used for training, which requires feature selection early in the training process.

    “These models have a larger feature space used for training, which require feature selection early in the training process.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Radiology AI performance is affected by other underlying practice patterns which can appear as hidden signals in AI data sets.

    “Radiology AI performance is affected by other underlying practice patterns which can appear as hidden signals in AI data sets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Bias risk assessment tools can facilitate the process of mitigating bias.

    “Bias risk assessment tools can facilitate this process. 65”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [65] Moons, 2019, PROBAST: A tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration, Ann Intern Med source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Bias at each stage of model development can occur due to machine factors, human factors or a combination of both.

    “As highlighted in Figure 1, bias at each stage of model development can occur due to machine factors, human factors or a combination of both.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    This test data must be created by the local IT team and then curated carefully by the local clinicians who may not be familiar with the requisite subgroup model performance evaluation and other unintended consequences of AI bias.

    “This test data must be created by the local IT team and then curated carefully by the local clinicians who may not be familiar with the requisite subgroup model performance evaluation and other unintended consequences of AI bias. 61”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [61] Ahluwalia, 2023, The subgroup imperative: chest X-ray Classifier generalization gaps in patient, setting, and pathology subgroups, Radiology: Artificial Intelligence

      Not checked in the free run.

    • Rests on an assumption: In many cases, models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.

  • Classified as probably not a substantial claim.

    Premisecitation

    Following guidelines for research documentation and complete reporting increases transparency.

    “Following guidelines for research documentation and complete reporting increases transparency. 63,64”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [63] Mongan, 2020, Checklist for artificial intelligence in medical imaging (CLAIM): A guide for authors and reviewers, Radiol Artif Intell source

      Not checked in the free run.

    • cites [64] Bossuyt, 2015, STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies, Radiology source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Bias at one stage can be propagated to other stages.

    “Moreover, bias at one stage can be propagated to other stages, hence efforts to detect and mitigate bias should be at each stage.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    In a study using a machine learning algorithm by pathologists to differentiate liver cancer types, pathologist accuracy did not improve when the model's prediction was correct, but worsened when the model's prediction was incorrect.

    “For example, the use of a machine learning algorithm by pathologists to differentiate different liver cancer types found that the pathologist accuracy did not improve when the model’s prediction was correct; however, the pathologist performance worsened when the model’s prediction was incorrect. 62”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [62] Kiani, 2020, Impact of a deep learning assistant on the histopathologic classification of liver cancer, NPJ Digit Med source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Neglecting continents and ethnically diverse locations perpetuates health inequities and hinders the growth and applicability of AI models.

    “Neglecting continents and ethnically diverse locations perpetuates health inequities and hinders the growth and applicability of AI models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A review of the use of clinical trials for machine learning in healthcare found only 41 randomized clinical trials, and most did not adhere to accepted reporting guidelines and had limited participation from minority groups.

    “In fact, a review of the use of clinical trials for machine learning in healthcare found only 41 randomized clinical trials, and most did not adhere to accepted reporting guidelines and had limited participation from minority groups. 58”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [58] Plana, 2022, Randomized clinical trials of machine learning interventions in health care: A systematic review, JAMA Netw Open source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    A review of 151 imaging AI products cleared by the FDA by November 2021 showed that only 64% of these products used clinical data for their validation; these clinical data had limited information on demographic and technical confounders (only 4% had patient demographics and 5% reported machine specifications).

    “A review of 151 imaging AI products cleared by the FDA by November 2021 showed that only 64% of these products used clinical data for their validation; and these clinical data had limited information on demographic and technical confounders (only 4% had patient demographics and 5% reported machine specifications). 56”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [56] Khunte, 2023, Trends in clinical validation and usage of US food and Drug Administration-cleared artificial intelligence Algorithms for medical imaging, Clin Radiol source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Relying on data sets with narrow geographic and other dimensions of diversity can further biases and limits generalizability of AI technologies.

    “Similarly, relying on data sets with narrow geographic and other dimensions of diversity can further biases and limits generalizability of AI technologies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Four federal US agencies recently issued a joint statement against biased models and their use including non-medical use like employment discrimination.

    “The regulatory landscape is changing as exemplified by the actions of four federal US agencies who recently issued a joint statement against biased models and their use including non-medical use like employment discrimination. 59”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] Chopra  R , Khan  LM . Joint statement on enforcement efforts against discrimination and bias in automated systems . Available from : https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CFPB-AI-Joint-Statement%28final%29.pdf

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    Failure to provide transparent documentation and detailed methodologies especially for commercial algorithms has potential to perpetuate biases and prevent critical analysis.

    “Failure to provide transparent documentation and detailed methodologies especially for commercial algorithms has potential to perpetuate biases and prevent critical analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Ensuring that AI works for all will require diverse teams in addition to diverse data sets.

    “Ensuring that AI works for all will require diverse teams in addition to diverse data sets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    External validation is infrequent or non-existent for FDA-approved and peer-reviewed articles, and when performed, it is typically limited in scope.

    “Despite infrequent to even non-existent external validation for both FDA approved and peer reviewed articles, when external validation is performed, it is typically limited in scope.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The most challenging issue for bias in medical imaging arises from techniques to address class imbalance and how to evaluate and mitigate harmful "hidden signals" in imaging data sets, as they can lead to shortcut learning and reinforce existing biases.

    “The most challenging issue for bias in medical imaging arises from techniques to address class imbalance and how to evaluate and mitigate harmful “hidden signals” in imaging data sets, as they can lead to shortcut learning and reinforce existing biases.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Bias occurs across the AI life cycle.

    “Bias occurs across the AI life cycle, and hence must be tackled at each step from problem definition all the way through post-deployment monitoring.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    These organizations issue statements like "consider evaluation of bias", without clear instructions on how the evaluation should be performed, and what consequences result from not performing such evaluation.

    “For example, these organizations issue statements like “consider evaluation of bias”, without clear instructions on how the evaluation should be performed, and what consequences result from not performing such evaluation. 11”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] Wawira Gichoya, 2021, Equity in essence: a call for Operationalising fairness in machine learning for Healthcare, BMJ Health Care Inform source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Due to limited AI explainability, hidden signals can be difficult to detect and have led to algorithmic bias that encrypts health inequities into practice.

    “Due to limited AI explainability, these hidden signals can be difficult to detect and have led to algorithmic bias that encrypts health inequities into practice.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    In the same 86 studies, nearly half reported a modest increase in external performance and nearly a quarter reported a substantial decrease.

    “Even on what would be considered a small sample size for external validation, nearly half of the 86 studies reported a modest increase in external performance with nearly a quarter reporting a substantial decrease, reiterating on the challenge of validation. 57”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In a systematic review of 86 studies, the median number of cases used for external validation was 240, with about 47% positive cases.

    “A recent systematic review found that 86 studies have a median of 240 cases used for external validation, with about 47% of which are positive cases. 57”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Because the approval process requires resubmission when models are updated, there are no incentives to address algorithmic bias during the initial application.

    “because the approval process require resubmission when models are updated, there are no incentives to address algorithmic bias during the initial application.”
    Quote in the paperNot a substantial claim
44 of 44

  • The paper relies on this without support.

    Bias principles are largely considered too late, if at all, when designing AI.

    “Bias principles are largely considered too late, if at all, when designing AI.”

  • The paper relies on this without support.

    Comprehensive AI data sets are limited and expensive to curate.

    “Comprehensive AI data sets are limited and expensive to curate.”

  • The paper relies on this without support.

    Presently, AI models are trained on data collected from tertiary care academic institutions.

    “While there have been initiatives to build data sets from “data deserts”, presently, AI models are trained on data collected from tertiary care academic institutions.”

  • The paper relies on this without support.

    Class imbalance occurs frequently in radiology.

    “Class imbalance occurs frequently in radiology.”

  • The paper relies on this without support.

    Radiology images tend to have high pixel dimensions and resolutions, yet pathology is represented on a small percentage of the pixels.

    “Radiology images tend to have high pixel dimensions and resolutions, yet pathology is represented on a small percentage of the pixels.”

  • The paper relies on this without support.

    Model validation remains a barrier that limits incorporation of AI into clinical care.

    “Model validation remains a barrier that limits incorporation of AI into clinical care.”

  • The paper relies on this without support.

    Many factors affect AI validation, including high costs and lack of consensus on proper validation protocol.

    “Many factors affect AI validation—including high costs of validation (technical pipeline for data set preparation and curation, diverse team of machine learning and clinical experts, regulatory approval burden), and lack of consensus on what is the proper protocol of validation.”

  • The paper relies on this without support.

    AI models 'fail silently' i.e. they do not provide an output to the user when they do not recognize the model input.

    “In these cases, the AI models “fail silently” i.e. they do not provide an output to the user when they do not recognize the model input.”

  • The paper relies on this without support.

    AI models will make a guess estimate of the prediction for wrong input data without providing the level of uncertainty.

    “These AI models will make a guess estimate of the prediction for wrong input data without providing the level of uncertainty.”

  • The paper relies on this without support.

    The greatest challenge of bias detection and mitigation remains the lack of a consensus definition of algorithmic bias despite many toolkits developed for this purpose.

    “The greatest challenge of bias detection and mitigation remains the lack of a consensus definition of algorithmic bias despite many toolkits developed for this purpose.”

  • The paper relies on this without support.

    External validation can ensure models generalize well across populations.

    “External validation can ensure models generalize well across populations.”

  • The paper relies on this without support.

    Bias evaluation and mitigation frequently occurs post-hoc.

    “Bias evaluation and mitigation frequently occurs post-hoc”

  • The paper relies on this without support.

    A review of AI reporting guidelines noted that references to bias are either absent, made in passing or included in the guideline supplement.

    “A review of AI reporting guidelines noted that references to bias are either absent, made in passing or included in the guideline supplement.”

  • The paper relies on this without support.

    Clinical trials have limitations including high costs, most trials never meet their enrollment proportions, and bias still exists in clinical trials despite their rigor.

    “While some people have called for more clinical trials in validating AI, we must acknowledge the limitations of such trials including high costs, the fact that most trials never meet their enrollment proportions, and that bias still exists in clinical trials despite their rigor.”

  • The paper relies on this without support.

    Proprietary models are usually trained and validated on private data sets that are not accessible for other researchers to evaluate the model.

    “Proprietary models present an additional layer of complexity because they are usually trained and validated on private data sets that are not accessible for other researchers to evaluate the model.”

  • The paper relies on this without support.

    Evaluation of proprietary algorithms, when performed, involves simulations to try and replicate model performance.

    “Evaluation of proprietary algorithms, when performed, involves simulations to try and replicate model performance.”

  • The paper relies on this without support.

    Vendors must navigate the convoluted process of integrating with current hospital IT systems, and engaging clinicians and patients.

    “Vendors must navigate the convoluted process of integrating with current hospital IT systems, and engaging clinicians and patients.”

  • The paper relies on this without support.

    In many cases, models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.

    “Moreover, in many cases, the models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.”

  • The paper relies on this without support.

    Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.

    “Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.”

  • The paper relies on this without support.

    Human–machine collaboration is now at the forefront of improving model performance, and to prevent unintended consequences including automation bias.

    “Human–machine collaboration is now at the forefront of improving model performance, and to prevent unintended consequences including automation bias.”

  • The paper relies on this without support.

    Broad grouping of underrepresented populations into the “Other” category is common when researchers have small sample sizes.

    “We also caution against broad grouping of underrepresented populations into the “Other” category which is common when researchers have small sample sizes.”

  • The paper relies on this without support.

    Mathematical approaches of fairness evaluation (e.g., fairness through unawareness, demographic parity, equalized odds or opportunity) may overlook nuanced biases and fail to address systemic issues.

    “We urge caution on relying on only mathematical approaches of fairness evaluation (e.g. relying solely on fairness through unawareness, demographic parity, or equalized odds or opportunity), as these approaches may overlook nuanced biases and fail to address systemic issues.”
22 of 22

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

10 of the 66 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

  1. Gebru  T , Morgenstern  J , Vecchione  B , Vaughan  JW , Wallach  H , Daumé III  H . Datasheets for Datasets . arXiv  2018 .
  2. Garbin  C , Rajpurkar  P , Irvin  J , Lungren  MP , Marques  O . Structured dataset documentation: a datasheet for CheXpert . arXiv  2021 .
  3. Chambon  P  et al . RoentGen: Vision-Language Foundation Model for Chest X-ray Generation . arXiv  2022 .
  4. Tsipras  D , Santurkar  S , Engstrom  L , Ilyas  A , Madry  A . From ImageNet to Image Classification: Contextualizing Progress on Benchmarks . arXiv  2020 .
  5. Yang  K , Yau  J , Fei-Fei  L , Deng  J , Russakovsky  O . A Study of Face Obfuscation in ImageNet . arXiv  2021 .

Cited by

Sorted by citations

Related

Authors
Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S.
Affiliations
  • Emory University (US)
  • Beth Israel Deaconess Medical Center (US)
  • Arizona State University (US)
  • York University (CA)
  • Indiana University – Purdue University Indianapolis (US)
Corresponding author
  • Gichoya, Judy Wawira
  • Purkayastha, Saptarshi
ORCID
Venue
The British Journal of Radiology
Publisher
Oxford University Press (OUP)
Published
2023-09-25
Volume
96
Issue
1150
Pages
20230023-20230023
ISSN
  • 0007-1285 (ISSN-L)
  • 1748-880X
DOI
10.1259/bjr.20230023
PMC
PMC10546443
OpenAlex
W4386624058
Semantic Scholar
ec14f2eda005b47d84f2be953a70ed5981fcc1aa
Cited by
277
References
66
Citations by year
2026: 88 · 2025: 118 · 2024: 67 · 2023: 4
FWCI
3.25
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • British Journal of Radiology (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Science Foundation (1928481)
  • Radiological Society of North America (#EIHD2204)
  • National Institutes of Health (75N92020C00021, 5r01eb017205-07, 75N92020C00008, R01-EB017205, EB017205)
  • National Heart, Lung, and Blood Institute (75N92020C00008, 75N92020C00021)
  • National Institute of Biomedical Imaging and Bioengineering (75N92020C00021, R01-EB017205, EB017205, 75N92020C00008)
  • National Institute on Minority Health and Health Disparities
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
algorithmic bias, bias mitigation, artificial intelligence in radiology, AI lifecycle, model training, model deployment, algorithmic fairness
MeSH terms
Artificial Intelligence, Humans, Learning, Radiology, Bias, Disease Progression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review