The DOI matches its Crossref record; 56 of the 66 references it cites were found in a registry or an index; none of the 86 sources its claims cite is checked for support yet.
AI pitfalls and what not to do: mitigating bias in AI
Open accessAbstract
Various forms of artificial intelligence (AI) applications are being deployed and used in many healthcare systems. As the use of these applications increases, we are learning the failures of these models and how they can perpetuate bias. With these new lessons, we need to prioritize bias evaluation and mitigation for radiology applications; all the while not ignoring the impact of changes in the larger enterprise AI deployment which may have downstream impact on performance of AI models. In this paper, we provide an updated review of known pitfalls causing AI bias and discuss strategies for mitigating these biases within the context of AI deployment in the larger healthcare enterprise. We describe these pitfalls by framing them in the larger AI lifecycle from problem definition, data set selection and curation, model training and deployment emphasizing that bias exists across a spectrum and is a sequela of a combination of both human and machine factors.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This review argues that bias in radiology AI arises across the entire lifecycle—problem definition, data acquisition, model development, validation, and deployment—from human, machine, and systems factors. It recommends mitigation at every stage, including diverse teams and data, rigorous validation, ongoing monitoring, and caution with purely mathematical fairness approaches.
Key findings
- Bias can persist even when race is excluded from model inputs: a commercial algorithm referred fewer Black patients with similar disease burden than White patients because healthcare costs were used as a proxy for need, learning from a system where Black patients are less likely to seek or receive care.
- Imaging AI data are geographically and demographically narrow: a review of published AI articles from 2015–2019 found data largely from California, Massachusetts, and New York, with none from 34 states; among 23 chest X-ray datasets, only 8.7% reported race or ethnicity and 4.3% reported insurance status; the UK Biobank has 500,000 patients but only 6% of non-European ancestry, and only 8% has been used for research.
- Validation gaps are common: among 151 FDA-cleared imaging AI products by November 2021, only 64% used clinical data for validation, 4% had patient demographics, 5% reported machine specifications, and 34% had multi-institutional validation; a systematic review of external validation found 86 studies with a median of 240 cases, about 47% positive.
- Deployment failures and human–machine interactions matter: the Epic sepsis model at Michigan Medicine identified only 7% of patients whose sepsis diagnosis was missed by a clinician, failed to flag 67% of patients with sepsis, and generated alerts on 18% of all hospitalized patients; in another example, pathologist accuracy worsened when a model’s prediction was incorrect.
Methods
- Narrative/updated review, not a systematic review; it frames AI bias pitfalls across the AI lifecycle: problem definition, dataset selection and curation, model training and deployment, and post-deployment evaluation and monitoring.
- Uses cited literature and examples—such as Obermeyer et al., the Epic sepsis model, COVID-19 imaging prediction models, RadImageNet, and federated learning—to illustrate bias sources and mitigation strategies.
- No new primary data, sample, or statistical analysis are reported; a formal methods section is not stated in the paper.
Limitations
- The paper does not include a dedicated limitations section; as a narrative review, it lacks systematic search or selection criteria and does not quantitatively synthesize evidence (evident from the text).
- It relies on selected published examples and author experience, so its coverage may not be exhaustive or generalizable (evident from the text).
- It states field-level limitations: lack of a unified definition of bias, limited model explainability, proprietary “black box inside a black box” models, and difficulty detecting harmful hidden signals in imaging datasets.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 86 cited sources checked · 0 supported · 0 flagged
In the paper
“In our own experience, 53 we observed that pre-processing of images is important and affects the federated model performance.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our preliminary work found that signals that identify race are also contained on synthetic data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Despite the promise and hope of artificial intelligence (AI) to improve patient care, several real-world failures of AI systems have been documented.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
- Not checked yetcites [2] Seyyed-Kalantari, 2021, Underdiagnosis bias of artificial intelligence Algorithms applied to chest Radiographs in under-served patient populations, Nat Med source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Before the subsequent model overhaul as a result of the publication (rather than some planned post-deployment evaluation and monitoring), it was observed that one input variable to the model was antibiotic orders by a provider—a type of data leakage since an infection is already being considered at that point.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source
- Not checked yetcites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“However, by using healthcare costs as proxy for the need for complex care, the model learned from a system where Black patients are less likely to seek and/or receive care compared to matched White patients. 1”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“A review of published AI articles between 2015 and 2019 in six disciplines found that data used for imaging AI models largely come from three states—California, Massachusetts, and New York—with none from 34 states. 22”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [22] Kaushal, 2020, Geographic distribution of US cohorts used to train deep learning Algorithms, JAMA source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“Zech et al demonstrated a similar finding where models predicted pneumonia by learning where the CXRs was obtained from. 20”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [20] Zech, 2018, Variable generalization performance of a deep learning model to detect pneumonia in chest Radiographs: A cross-sectional study, PLoS Med source
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIn the paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- It may state two separate claims as one. 29
- Probably not a substantial claim. 21
In the paper
“Moreover, FedAvg is biased to research sites contributing larger data sets with poor performance reported for smaller data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“In our experimental evaluation of the reading race paper where we demonstrated superhuman ability for AI to predict self-reported race across multiple modalities, we found that some transfer learning tasks are better in race detection. 28”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [28] Gichoya, 2022, AI recognition of patient race in medical imaging: a Modelling study, Lancet Digit Health source
Strength:boosted· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“When we used the FedAvg algorithm, the model improved its generalizability on external data sets but at the expense of some degradation in internal validation.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Existing AI data sets are acquired from limited geographical regions; most lack information necessary to evaluate subgroup performance of models.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [22] Kaushal, 2020, Geographic distribution of US cohorts used to train deep learning Algorithms, JAMA source
- Not checked yetcites [23] Yi, 2022, Demographic reporting in publicly available chest radiograph data SETS: opportunities for mitigating sex and racial disparities in deep learning models, J Am Coll Radiol source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition to observing overall performance decline secondary to differences in data distribution after model deployment, subgroup evaluations have shown varying levels of performance with poor performance especially for historically underserved patients. 1,2”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
- Not checked yetcites [2] Seyyed-Kalantari, 2021, Underdiagnosis bias of artificial intelligence Algorithms applied to chest Radiographs in under-served patient populations, Nat Med source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“The algorithm (which was abandoned before deployment) learnt that patients from less affluent zip codes are likely to have longer hospital stays and may not benefit from case management facilitation of early discharge.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [16] Nordling, 2019, A fairer way forward for AI in health care, Nature source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Problems in data set development include limited patient diversity, limited resolution and missing clinical confounders, limited or suboptimal quality annotations, and lack of data standards and best practice to ensure quality and reproducibility.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The failure and subsequent withdrawal of the Epic (TM) sepsis model used by 180 customers representing hundreds of hospitals provides a template to study AI failure across a spectrum. 13,14”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source
- Not checked yetcites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that bias permeates every step of the lifecycle and is a sequela of human, machine, and systems factors.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“A review of 23 CXR data sets found that although the majority reported age and sex, only 8.7% reported race or ethnicity and 4.3% reported insurance status. 23”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [23] Yi, 2022, Demographic reporting in publicly available chest radiograph data SETS: opportunities for mitigating sex and racial disparities in deep learning models, J Am Coll Radiol source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Efforts to mitigate these biases are challenging due to—lack of a unified definition of bias 10 ; focus on a statistical definition of bias and a technocentric definition of bias without involving patients and communities; and post-hoc reflection on bias rather than as a deliverable by design. 11”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [10] Narayanan A . Tutorial: 21 fairness definitions and their politics . 2018 . Available from : https://www.youtube.com/watch?v=jIXIuYdnyyk
- Not checked yetcites [11] Wawira Gichoya, 2021, Equity in essence: a call for Operationalising fairness in machine learning for Healthcare, BMJ Health Care Inform source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Upon testing at Michigan Medicine, the algorithm only identified 7% of patients whose sepsis diagnosis was missed by a clinician, failed to flag 67% of patients with sepsis despite generating alerts on 18% of all hospitalized patients resulting in alert fatigue.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source
- Not checked yetcites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It should be noted that model auditing and bias detection, which benefit from diverse perspectives during training and validation, are more difficult in federated learning where only the weights are shared across the groups of modelers.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Some groups have suggested datasheets 39,40 —a checklist to guide model developers on how to use data—to address this problem.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [39] Gebru T , Morgenstern J , Vecchione B , Vaughan JW , Wallach H , Daumé III H . Datasheets for Datasets . arXiv 2018 .
- Not checked yetcites [40] Garbin C , Rajpurkar P , Irvin J , Lungren MP , Marques O . Structured dataset documentation: a datasheet for CheXpert . arXiv 2021 .
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The availability of a radiology specific data set to facilitate transfer learning is a step towards efficiency in model development.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“CXRs have also been used to predict biological age, cardiac disease outcomes and healthcare cost at 1, 3 and 5 years. 30”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [30] Sohn, 2022, Prediction of future Healthcare expenses of patients from chest Radiographs using deep learning: a pilot study, Sci Rep source
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“Overall, when local model performance matters—then personalized federated learning models (such as FedBN) should be prioritized, while for external performance, FedAvg should be used. 53”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Federated learning—whereby models are trained across multiple institutions without the data being shared except for the model weights—has been proposed as a solution to improve model performance across diverse patient groups.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such data set will also discourage the use of Frankenstein data sets—assembled from other data sets and redistributed under a new name—as pretraining can be done on RadImageNet and fine-tuned on a smaller data set.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Beyond individual images, class imbalance also occurs due to distribution of positive versus negative cases (since most studies are normal), and across various demographic distributions within data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Class imbalance occurs frequently in radiology.
In the paper
“In fact, only 8% of the UK Biobank has been used for research, with most of this research focused on patients of European ancestry.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [24] Privé, 2022, Using the UK Biobank as a global reference of worldwide populations: application to measuring ancestry diversity from GWAS summary Statistics, Bioinformatics source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These models have a larger feature space used for training, which require feature selection early in the training process.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Radiology AI performance is affected by other underlying practice patterns which can appear as hidden signals in AI data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Bias risk assessment tools can facilitate this process. 65”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
What the sources say
- Not checked yetcites [65] Moons, 2019, PROBAST: A tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration, Ann Intern Med source
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As highlighted in Figure 1, bias at each stage of model development can occur due to machine factors, human factors or a combination of both.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
In the paper
“This test data must be created by the local IT team and then curated carefully by the local clinicians who may not be familiar with the requisite subgroup model performance evaluation and other unintended consequences of AI bias. 61”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model implementation and deployment
What the sources say
- Not checked yetcites [61] Ahluwalia, 2023, The subgroup imperative: chest X-ray Classifier generalization gaps in patient, setting, and pathology subgroups, Radiology: Artificial Intelligence
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: In many cases, models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.
None of the sources cited for this claim have been checked yet.
In the paper
“Following guidelines for research documentation and complete reporting increases transparency. 63,64”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
What the sources say
- Not checked yetcites [63] Mongan, 2020, Checklist for artificial intelligence in medical imaging (CLAIM): A guide for authors and reviewers, Radiol Artif Intell source
- Not checked yetcites [64] Bossuyt, 2015, STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies, Radiology source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, bias at one stage can be propagated to other stages, hence efforts to detect and mitigate bias should be at each stage.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
In the paper
“For example, the use of a machine learning algorithm by pathologists to differentiate different liver cancer types found that the pathologist accuracy did not improve when the model’s prediction was correct; however, the pathologist performance worsened when the model’s prediction was incorrect. 62”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model implementation and deployment
What the sources say
- Not checked yetcites [62] Kiani, 2020, Impact of a deep learning assistant on the histopathologic classification of liver cancer, NPJ Digit Med source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Neglecting continents and ethnically diverse locations perpetuates health inequities and hinders the growth and applicability of AI models.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In fact, a review of the use of clinical trials for machine learning in healthcare found only 41 randomized clinical trials, and most did not adhere to accepted reporting guidelines and had limited participation from minority groups. 58”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [58] Plana, 2022, Randomized clinical trials of machine learning interventions in health care: A systematic review, JAMA Netw Open source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“A review of 151 imaging AI products cleared by the FDA by November 2021 showed that only 64% of these products used clinical data for their validation; and these clinical data had limited information on demographic and technical confounders (only 4% had patient demographics and 5% reported machine specifications). 56”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [56] Khunte, 2023, Trends in clinical validation and usage of US food and Drug Administration-cleared artificial intelligence Algorithms for medical imaging, Clin Radiol source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, relying on data sets with narrow geographic and other dimensions of diversity can further biases and limits generalizability of AI technologies.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“The regulatory landscape is changing as exemplified by the actions of four federal US agencies who recently issued a joint statement against biased models and their use including non-medical use like employment discrimination. 59”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [59] Chopra R , Khan LM . Joint statement on enforcement efforts against discrimination and bias in automated systems . Available from : https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CFPB-AI-Joint-Statement%28final%29.pdf
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Failure to provide transparent documentation and detailed methodologies especially for commercial algorithms has potential to perpetuate biases and prevent critical analysis.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ensuring that AI works for all will require diverse teams in addition to diverse data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Despite infrequent to even non-existent external validation for both FDA approved and peer reviewed articles, when external validation is performed, it is typically limited in scope.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most challenging issue for bias in medical imaging arises from techniques to address class imbalance and how to evaluate and mitigate harmful “hidden signals” in imaging data sets, as they can lead to shortcut learning and reinforce existing biases.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bias occurs across the AI life cycle, and hence must be tackled at each step from problem definition all the way through post-deployment monitoring.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“For example, these organizations issue statements like “consider evaluation of bias”, without clear instructions on how the evaluation should be performed, and what consequences result from not performing such evaluation. 11”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [11] Wawira Gichoya, 2021, Equity in essence: a call for Operationalising fairness in machine learning for Healthcare, BMJ Health Care Inform source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to limited AI explainability, these hidden signals can be difficult to detect and have led to algorithmic bias that encrypts health inequities into practice.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Even on what would be considered a small sample size for external validation, nearly half of the 86 studies reported a modest increase in external performance with nearly a quarter reporting a substantial decrease, reiterating on the challenge of validation. 57”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“A recent systematic review found that 86 studies have a median of 240 cases used for external validation, with about 47% of which are positive cases. 57”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source
Strength:plain· not yet calibratedn = 240Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“because the approval process require resubmission when models are updated, there are no incentives to address algorithmic bias during the initial application.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
Bias principles are largely considered too late, if at all, when designing AI.
“Bias principles are largely considered too late, if at all, when designing AI.”
The paper relies on this without support.
Comprehensive AI data sets are limited and expensive to curate.
“Comprehensive AI data sets are limited and expensive to curate.”
The paper relies on this without support.
Presently, AI models are trained on data collected from tertiary care academic institutions.
“While there have been initiatives to build data sets from “data deserts”, presently, AI models are trained on data collected from tertiary care academic institutions.”
The paper relies on this without support.
Class imbalance occurs frequently in radiology.
“Class imbalance occurs frequently in radiology.”
The paper relies on this without support.
Radiology images tend to have high pixel dimensions and resolutions, yet pathology is represented on a small percentage of the pixels.
“Radiology images tend to have high pixel dimensions and resolutions, yet pathology is represented on a small percentage of the pixels.”
The paper relies on this without support.
Model validation remains a barrier that limits incorporation of AI into clinical care.
“Model validation remains a barrier that limits incorporation of AI into clinical care.”
The paper relies on this without support.
Many factors affect AI validation, including high costs and lack of consensus on proper validation protocol.
“Many factors affect AI validation—including high costs of validation (technical pipeline for data set preparation and curation, diverse team of machine learning and clinical experts, regulatory approval burden), and lack of consensus on what is the proper protocol of validation.”
The paper relies on this without support.
AI models 'fail silently' i.e. they do not provide an output to the user when they do not recognize the model input.
“In these cases, the AI models “fail silently” i.e. they do not provide an output to the user when they do not recognize the model input.”
The paper relies on this without support.
AI models will make a guess estimate of the prediction for wrong input data without providing the level of uncertainty.
“These AI models will make a guess estimate of the prediction for wrong input data without providing the level of uncertainty.”
The paper relies on this without support.
The greatest challenge of bias detection and mitigation remains the lack of a consensus definition of algorithmic bias despite many toolkits developed for this purpose.
“The greatest challenge of bias detection and mitigation remains the lack of a consensus definition of algorithmic bias despite many toolkits developed for this purpose.”
The paper relies on this without support.
External validation can ensure models generalize well across populations.
“External validation can ensure models generalize well across populations.”
The paper relies on this without support.
Bias evaluation and mitigation frequently occurs post-hoc.
“Bias evaluation and mitigation frequently occurs post-hoc”
The paper relies on this without support.
A review of AI reporting guidelines noted that references to bias are either absent, made in passing or included in the guideline supplement.
“A review of AI reporting guidelines noted that references to bias are either absent, made in passing or included in the guideline supplement.”
The paper relies on this without support.
Clinical trials have limitations including high costs, most trials never meet their enrollment proportions, and bias still exists in clinical trials despite their rigor.
“While some people have called for more clinical trials in validating AI, we must acknowledge the limitations of such trials including high costs, the fact that most trials never meet their enrollment proportions, and that bias still exists in clinical trials despite their rigor.”
The paper relies on this without support.
Proprietary models are usually trained and validated on private data sets that are not accessible for other researchers to evaluate the model.
“Proprietary models present an additional layer of complexity because they are usually trained and validated on private data sets that are not accessible for other researchers to evaluate the model.”
The paper relies on this without support.
Evaluation of proprietary algorithms, when performed, involves simulations to try and replicate model performance.
“Evaluation of proprietary algorithms, when performed, involves simulations to try and replicate model performance.”
The paper relies on this without support.
Vendors must navigate the convoluted process of integrating with current hospital IT systems, and engaging clinicians and patients.
“Vendors must navigate the convoluted process of integrating with current hospital IT systems, and engaging clinicians and patients.”
The paper relies on this without support.
In many cases, models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.
“Moreover, in many cases, the models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.”
The paper relies on this without support.
Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.
“Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.”
The paper relies on this without support.
Human–machine collaboration is now at the forefront of improving model performance, and to prevent unintended consequences including automation bias.
“Human–machine collaboration is now at the forefront of improving model performance, and to prevent unintended consequences including automation bias.”
The paper relies on this without support.
Broad grouping of underrepresented populations into the “Other” category is common when researchers have small sample sizes.
“We also caution against broad grouping of underrepresented populations into the “Other” category which is common when researchers have small sample sizes.”
The paper relies on this without support.
Mathematical approaches of fairness evaluation (e.g., fairness through unawareness, demographic parity, equalized odds or opportunity) may overlook nuanced biases and fail to address systemic issues.
“We urge caution on relying on only mathematical approaches of fairness evaluation (e.g. relying solely on fairness through unawareness, demographic parity, or equalized odds or opportunity), as these approaches may overlook nuanced biases and fail to address systemic issues.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“AI systems may also produce outputs that are skewed by biases related to race, ethnicity, sex, religion, politics, and other factors. [5, 23, 53–56]”
From Autonomous artificial intelligence, scientific research, and human values (2026) · cites this paper as [53]
In the citing paper
“One of the first strategies that can be used to identify bias related to the dataset is dataset evaluation against a set of predefined criteria (searching for exclusion bias, selection bias, recall bias, observer bias, and prejudice bias) and comprehensive data analysis.47”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [47]
In the citing paper
“Statistical comparison of model output according to different patient groups or confounders that may exist in the training dataset, such as the gender or age of patients. Potential discrepancies in group results could indicate a source of bias that can affect the final results.48”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [48]
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“BMJ Health Care Inform 2021; 28(1): e100289.”
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 1×
“18,19 Comprehensive AI data sets are limited and expensive to curate.”
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“For example, a developer will have to decide between aiming for group fairness (Black vs white patients, patients on government healthcare programs, or a specific age group) vs individual fairness 49 (where two individuals with similar characteristics are treated in the same manner).”
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 2×
“50 Attempts to be race agnostic or race neutral are challenging because it remains difficult to remove race and race proxies like pattern of health service utilization because of patient and/or provider factors.”
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · cited 1×
“20 To overcome this confounding, progress must be made to improve model training, evaluation, and explainability.”
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“63,64 We encourage researchers to ensure that their work is reproducible, allowing for independent validation and bias detection.”
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C., Glasziou, P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,360
Racial bias in pulse oximetry measurement
Sjoding, M. W., Dickson, R. P., Iwashyna, T. J., Gay, S. E., & Valley, T. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmc2029240 · Cited by 1,114 · cited 1×
“…in clinical parameters such as glomerular filtration rate, pulse oximetry (with undiagnosed hidden hypoxemia for historically underserved patients), 35 differences in ICU severity scoring 36 shape radiology data sets for machine learning through a cascade effect as they affect ordering patterns…”
Identifying and Avoiding Bias in Research
Pannucci, C. J., & Wilkins, E. G. (2010) · Plastic and Reconstructive Surgery · doi:10.1097/prs.0b013e3181de24bc · Cited by 1,079 · cited 1×
“Radiol Artif Intell 2022; 4(3): e210064.”
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · cited 2×
“13,14 A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration…”
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Self-citationSeyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928
Self-citationRoberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 902
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685 · cited 1×
“53 54 The assumption that inclusion of more minority groups in the training data set is sufficient to fix bias is incorrect for the reasons above and since patients exist at the intersection of multiple group representation e.g. Black, Black Males/Females, Black Males/Females (young or old), Old…”
AI recognition of patient race in medical imaging: a modelling study
Gichoya, J. W., Banerjee, I., Bhimireddy, A. R., Burns, J. L., Celi, L. A., Chen, L.-C., Correa, R., Dullerud, N., Ghassemi, M., Huang, S.-C., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00063-2 · Cited by 611 · cited 2×
“28 The availability of a radiology specific data set to facilitate transfer learning is a step towards efficiency in model development.”
Self-citationTowards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 477 · cited 2×
“57 Even on what would be considered a small sample size for external validation, nearly half of the 86 studies reported a modest increase in external performance with nearly a quarter reporting a substantial decrease, reiterating on the challenge of validation.”
The European Union Agency for Fundamental Rights
Dimitrakopoulos, I. (2015) · Policy Press eBooks · doi:10.1332/policypress/9781447314899.003.0006 · Cited by 473 · cited 1×
“,26 Recent studies have shown that AI models can learn features that are “invisible” to a human expert.”
Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N., Peixoto, R. M., Guimarães, G. A. D. S., Cruz, G. O. R., Araujo, M. M., dos Santos, L. L., Cruz, M. A. S., Oliveira, E. L. S., et al. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7010015 · Cited by 421
Brain Imaging Generation with Latent Diffusion Models
Pinaya, W. H. L., Tudosiu, P.-D., Dafflon, J., Da Costa, P. F., Fernandez, V., Nachev, P., Ourselin, S., & Cardoso, M. J. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-18576-2_12 · Cited by 395
Radimagenet: an open Radiologic deep learning research Dataset for effective transfer learning
Mei, X., Liu, Z., Robson, P. M., Marinelli, B., Huang, M., Doshi, A., Jacobi, A., Cao, C., Link, K. E., Yang, T., et al. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210315 · Cited by 341 · cited 1×
“A milestone in medical imaging data was achieved in 2022 with the publication of RadImageNet, 43 an open radiologic deep learning data set that consists of 1.35 million radiologic images for efficient transfer learning.”
InfluentialImpact of a deep learning assistant on the histopathologic classification of liver cancer
Kiani, A., Uyumazturk, B., Rajpurkar, P., Wang, A., Gao, R., Jones, E., Yu, Y., Langlotz, C. P., Ball, R. L., Montine, T. J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0232-8 · Cited by 325 · cited 1×
“62 Human–machine collaboration is now at the forefront of improving model performance, and to prevent unintended consequences including automation bias.”
Gender bias in artificial intelligence
Leavy, S. (2018) · doi:10.1145/3195570.3195580 · Cited by 285
Mitigating bias in machine learning for medicine
Vokinger, K. N., Feuerriegel, S., & Kesselheim, A. S. (2021) · Communications Medicine · doi:10.1038/s43856-021-00028-w · Cited by 269
Detecting racial bias in algorithms and machine learning
Lee, N. T. (2018) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-06-2018-0056 · Cited by 263 · cited 1×
“Radiol Artif Intell 2022; 4(5): e220061.”
Geographic Distribution of US Cohorts Used to Train Deep Learning Algorithms
Kaushal, A., Altman, R., & Langlotz, C. (2020) · JAMA · doi:10.1001/jama.2020.12067 · Cited by 239 · cited 1×
“22 While there have been initiatives to build data sets from “data deserts”, presently, AI models are trained on data collected from tertiary care academic institutions.”
Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review
Plana, D., Shung, D. L., Grimshaw, A. A., Saraf, A., Sung, J. J. Y., & Kann, B. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.33946 · Cited by 236
Unveiling COVID-19 from CHEST X-ray with deep learning: A hurdles race with small data
Tartaglione, E., Barbano, C. A., Berzovini, C., Calandri, M., & Grangetto, M. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17186933 · Cited by 211
Racial and ethnic differences in bystander CPR for witnessed cardiac arrest
Garcia, R. A., Spertus, J. A., Girotra, S., Nallamothu, B. K., Kennedy, K. F., McNally, B. F., Breathett, K., Del Rios, M., Sasson, C., & Chan, P. S. (2022) · New England Journal of Medicine · doi:10.1056/nejmoa2200798 · Cited by 163 · cited 1×
“34 This means that these patients would be excluded from an imaging cohort evaluating cardiac disease outcomes.”
Drukker, K., Chen, W., Gichoya, J. W., Gruszauskas, N. P., Kalpathy-Cramer, J., Koyejo, S., Myers, K. J., Sá, R. C., Sahiner, B., Whitney, H. M., et al. (2023) · Journal of Medical Imaging · doi:10.1117/1.jmi.10.6.061104 · Cited by 149 · cited 1×
“18,19 Comprehensive AI data sets are limited and expensive to curate.”
Self-citationMitigating Bias in Radiology Machine Learning: 1. Data Handling
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Vera Garcia, D. V., Singh, Y., Zhang, K., Conte, G. M., & Erickson, B. J. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.210290 · Cited by 145
Geis, J. R., Brady, A. P., Wu, C. C., Spencer, J., Ranschaert, E., Jaremko, J. L., Langer, S. G., Kitts, A. B., Birch, J., Shields, W. F., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.08.010 · Cited by 140
Self-citation“Shortcuts” Causing Bias in Radiology Artificial Intelligence: Causes, Evaluation, and Mitigation
Banerjee, I., Bhattacharjee, K., Burns, J. L., Trivedi, H., Purkayastha, S., Seyyed-Kalantari, L., Patel, B. N., Shiradkar, R., & Gichoya, J. W. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.06.025 · Cited by 131 · cited 1×
“21 A review of published AI articles between 2015 and 2019 in six disciplines found that data used for imaging AI models largely come from three states—California, Massachusetts, and New York—with none from 34 states.”
Self-citationAI and Algorithmic Bias: Source, Detection, Mitigation and Implications
Fu, R., Huang, Y. H., & Singh, P. V. (2020) · doi:10.1287/educ.2020.0215 · Cited by 130
Racial Disparities in ICU Outcomes: A Systematic Review*
McGowan, S. K., Sarigiannis, K. A., Fox, S. C., Gottlieb, M. A., & Chen, E. (2022) · Critical Care Medicine · doi:10.1097/ccm.0000000000005269 · Cited by 108 · cited 1×
“Examples of such metrics in addition to healthcare cost include outputs of decision-making such readmission 17 or patient no-show.”
Equity in essence: a call for operationalising fairness in machine learning for healthcare
Wawira Gichoya, J., McCoy, L. G., Celi, L. A., & Ghassemi, M. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100289 · Cited by 104
Self-citationMitigating Bias in Radiology Machine Learning: 3. Performance Metrics
Faghani, S., Khosravi, B., Zhang, K., Moassefi, M., Jagtap, J. M. M., Nugen, F., Vahdati, S., Kuanar, S. P., Rassoulinejad-Mousavi, S. M., Singh, Y., et al. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220061 · Cited by 104
The epic sepsis model falls short-the importance of external validation
Habib, A. R., Lin, A. L., & Grant, R. W. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.3333 · Cited by 101 · cited 2×
“13,14 A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration…”
A fairer way forward for AI in health care
Nordling, L. (2019) · Nature · doi:10.1038/d41586-019-02872-2 · Cited by 90 · cited 1×
“16 The algorithm (which was abandoned before deployment) learnt that patients from less affluent zip codes are likely to have longer hospital stays and may not benefit from case management facilitation of early discharge.”
Sarkar, R., Martin, C., Mattie, H., Gichoya, J. W., Stone, D. J., & Celi, L. A. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00022-4 · Cited by 85 · cited 1×
“…such as glomerular filtration rate, pulse oximetry (with undiagnosed hidden hypoxemia for historically underserved patients), 35 differences in ICU severity scoring 36 shape radiology data sets for machine learning through a cascade effect as they affect ordering patterns for medical images.”
Wang, H. E., Landers, M., Adams, R., Subbaswamy, A., Kharrazi, H., Gaskin, D. J., & Saria, S. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac065 · Cited by 63
Tariq, A., Purkayastha, S., Padmanaban, G. P., Krupinski, E., Trivedi, H., Banerjee, I., & Gichoya, J. W. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.08.018 · Cited by 63 · cited 1×
“60 Evaluation of proprietary algorithms, when performed, involves simulations to try and replicate model performance.”
Self-citationPeng, L., Luo, G., Walker, A., Zaiman, Z., Jones, E. K., Gupta, H., Kersten, K., Burns, J. L., Harle, C. A., Magoc, T., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac188 · Cited by 36
Self-citationPrivé, F. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac348 · Cited by 33 · cited 1×
“24 Presently, it would be difficult to further evaluate model bias on these data sets.”
Yi, P. H., Kim, T. K., Siegel, E. L., & Yahyavi‐Firouz‐Abadi, N. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.08.018 · Cited by 33 · cited 1×
“23 Even large repositories like the UK Biobank that represents a prospective cohort of 500,000 patients are limited for some patient groups: only 6% of the cohort are of non-European ancestry.”
Khunte, M., Chae, A., Wang, R., Jain, R., Sun, Y., Sollee, J. R., Jiao, Z., & Bai, H. X. (2023) · Clinical Radiology · doi:10.1016/j.crad.2022.09.122 · Cited by 33 · cited 2×
“56 Moreover, only 34% had multi-institutional validation and reported which institution(s) was(were) used for validation.”
Shan, A., Baumann, G., & Gholamrezanezhad, A. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.12.016 · Cited by 32 · cited 1×
“32 Inappro-priate use among historically underserved patients has been demonstrated in other areas of diagnostic imaging.”
Predicting patient demographics from chest Radiographs with deep learning
Adleberg, J., Wardeh, A., Doo, F. X., Marinelli, B., Cook, T. S., Mendelson, D. S., & Kagen, A. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2022.06.008 · Cited by 32 · cited 1×
“28,29 CXRs have also been used to predict biological age, cardiac disease outcomes and healthcare cost at 1, 3 and 5 years.”
Implications of predicting race variables from medical images
Zou, J., Gichoya, J. W., Ho, D. E., & Obermeyer, Z. (2023) · Science · doi:10.1126/science.adh4260 · Cited by 30
Self-citationPatient race or Ethnicity and the use of diagnostic imaging: A systematic review
Colwell, R. L., Narayan, A. K., & Ross, A. B. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2022.01.008 · Cited by 28 · cited 1×
“33 An investigation of bystander CPR response for witnessed cardiac arrest found that Black and Hispanic patients or females are less likely to get CPR regardless of income level or neighborhood where the cardiac arrest occurred.”
Trivedi, A., Robinson, C., Blazes, M., Ortiz, A., Desbiens, J., Gupta, S., Dodhia, R., Bhatraju, P. K., Liles, W. C., Kalpathy-Cramer, J., et al. (2022) · PLOS ONE · doi:10.1371/journal.pone.0274098 · Cited by 28 · cited 2×
“…characteristics and penalize the model when it learns features that can discriminate between the original data sets where the images are obtained) 51 ; applying pre-processing techniques like histogram equalization to correct for differences in the images and lung masking to exclude areas where…”
Ahluwalia, M., Abdalla, M., Sanayei, J., Seyyed-Kalantari, L., Hussain, M., Ali, A., & Fine, B. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220270 · Cited by 26 · cited 1×
“61 Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.”
Self-citationDetecting racial/ethnic health disparities using deep learning from frontal chest radiography
Pyrros, A., Rodríguez-Fernández, J. M., Borstelmann, S. M., Gichoya, J. W., Horowitz, J. M., Fornelli, B., Siddiqui, N., Velichko, Y., Koyejo, O., & Galanter, W. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.09.010 · Cited by 15 · cited 1×
“27 More concerning about these models is that information on social determinants of health like area deprivation index are learned and encoded on the CXRs. CXRs can also be used to predict demographic information—including insurance, self-reported race, and age.”
Sohn, J. H., Chen, Y., Lituiev, D., Yang, J., Ordovas, K. G., Hadley, D., Vu, T.-K. H., Franc, B. L., & Seo, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-12551-4 · Cited by 11 · cited 1×
“30 Despite the surprising performance of AI models to detect these signals, there are limitations in model explainability.”
Flagged first, then by citations
10 of the 66 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Gebru T , Morgenstern J , Vecchione B , Vaughan JW , Wallach H , Daumé III H . Datasheets for Datasets . arXiv 2018 .
- Garbin C , Rajpurkar P , Irvin J , Lungren MP , Marques O . Structured dataset documentation: a datasheet for CheXpert . arXiv 2021 .
- Chambon P et al . RoentGen: Vision-Language Foundation Model for Chest X-ray Generation . arXiv 2022 .
- Tsipras D , Santurkar S , Engstrom L , Ilyas A , Madry A . From ImageNet to Image Classification: Contextualizing Progress on Benchmarks . arXiv 2020 .
- Yang K , Yau J , Fei-Fei L , Deng J , Russakovsky O . A Study of Face Obfuscation in ImageNet . arXiv 2021 .
Cited by
Sorted by citations
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567
“Gichoya et al. (2023) argue that AI systems, trained on extensive data-sets sourced from the internet, inherently mirror societal biases.”
BackgroundBias recognition and mitigation strategies in artificial intelligence healthcare applications
Hasanzadeh, F., Josephson, C. B., Waters, G., Adedinsewo, D., Azizi, Z., & White, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01503-7 · Cited by 320
Generative AI for Customizable Learning Experiences
Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024) · Sustainability · doi:10.3390/su16073034 · Cited by 274
“Mitigation strategies include the resampling and pre-processing of data used for training the models [68], using diverse and representative datasets [69], feedback loops [70], etc.”
MethodWeiner, E., Dankwa‐Mullan, I., Nelson, W. A., & Hassanpour, S. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000810 · Cited by 306
AI agents under threat: A survey of key security challenges and future pathways
Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3716628 · Cited by 210
Metadata
- Authors
- Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S.
- Affiliations
- Emory University (US)
- Beth Israel Deaconess Medical Center (US)
- Arizona State University (US)
- York University (CA)
- Indiana University – Purdue University Indianapolis (US)
- Corresponding author
- Gichoya, Judy Wawira
- Purkayastha, Saptarshi
- ORCID
- Venue
- The British Journal of Radiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-09-25
- Volume
- 96
- Issue
- 1150
- Pages
- 20230023-20230023
- ISSN
- 0007-1285 (ISSN-L)
- 1748-880X
- PMC
- PMC10546443
- OpenAlex
- W4386624058
- Semantic Scholar
- ec14f2eda005b47d84f2be953a70ed5981fcc1aa
- Cited by
- 277
- References
- 66
- Citations by year
- 2026: 88 · 2025: 118 · 2024: 67 · 2023: 4
- FWCI
- 3.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- British Journal of Radiology (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (1928481)
- Radiological Society of North America (#EIHD2204)
- National Institutes of Health (75N92020C00021, 5r01eb017205-07, 75N92020C00008, R01-EB017205, EB017205)
- National Heart, Lung, and Blood Institute (75N92020C00008, 75N92020C00021)
- National Institute of Biomedical Imaging and Bioengineering (75N92020C00021, R01-EB017205, EB017205, 75N92020C00008)
- National Institute on Minority Health and Health Disparities
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic bias, bias mitigation, artificial intelligence in radiology, AI lifecycle, model training, model deployment, algorithmic fairness
- MeSH terms
- Artificial Intelligence, Humans, Learning, Radiology, Bias, Disease Progression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
