The DOI matches its Crossref record; 56 of the 66 references it cites were found in a registry or an index; none of the 86 sources its claims cite is checked for support yet.
AI pitfalls and what not to do: mitigating bias in AI
Open accessAbstract
Various forms of artificial intelligence (AI) applications are being deployed and used in many healthcare systems. As the use of these applications increases, we are learning the failures of these models and how they can perpetuate bias. With these new lessons, we need to prioritize bias evaluation and mitigation for radiology applications; all the while not ignoring the impact of changes in the larger enterprise AI deployment which may have downstream impact on performance of AI models. In this paper, we provide an updated review of known pitfalls causing AI bias and discuss strategies for mitigating these biases within the context of AI deployment in the larger healthcare enterprise. We describe these pitfalls by framing them in the larger AI lifecycle from problem definition, data set selection and curation, model training and deployment emphasizing that bias exists across a spectrum and is a sequela of a combination of both human and machine factors.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This review argues that bias in radiology AI arises across the entire lifecycle—problem definition, data acquisition, model development, validation, and deployment—from human, machine, and systems factors. It recommends mitigation at every stage, including diverse teams and data, rigorous validation, ongoing monitoring, and caution with purely mathematical fairness approaches.
Key findings
- Bias can persist even when race is excluded from model inputs: a commercial algorithm referred fewer Black patients with similar disease burden than White patients because healthcare costs were used as a proxy for need, learning from a system where Black patients are less likely to seek or receive care.
- Imaging AI data are geographically and demographically narrow: a review of published AI articles from 2015–2019 found data largely from California, Massachusetts, and New York, with none from 34 states; among 23 chest X-ray datasets, only 8.7% reported race or ethnicity and 4.3% reported insurance status; the UK Biobank has 500,000 patients but only 6% of non-European ancestry, and only 8% has been used for research.
- Validation gaps are common: among 151 FDA-cleared imaging AI products by November 2021, only 64% used clinical data for validation, 4% had patient demographics, 5% reported machine specifications, and 34% had multi-institutional validation; a systematic review of external validation found 86 studies with a median of 240 cases, about 47% positive.
- Deployment failures and human–machine interactions matter: the Epic sepsis model at Michigan Medicine identified only 7% of patients whose sepsis diagnosis was missed by a clinician, failed to flag 67% of patients with sepsis, and generated alerts on 18% of all hospitalized patients; in another example, pathologist accuracy worsened when a model’s prediction was incorrect.
Methods
- Narrative/updated review, not a systematic review; it frames AI bias pitfalls across the AI lifecycle: problem definition, dataset selection and curation, model training and deployment, and post-deployment evaluation and monitoring.
- Uses cited literature and examples—such as Obermeyer et al., the Epic sepsis model, COVID-19 imaging prediction models, RadImageNet, and federated learning—to illustrate bias sources and mitigation strategies.
- No new primary data, sample, or statistical analysis are reported; a formal methods section is not stated in the paper.
Limitations
- The paper does not include a dedicated limitations section; as a narrative review, it lacks systematic search or selection criteria and does not quantitatively synthesize evidence (evident from the text).
- It relies on selected published examples and author experience, so its coverage may not be exhaustive or generalizable (evident from the text).
- It states field-level limitations: lack of a unified definition of bias, limited model explainability, proprietary “black box inside a black box” models, and difficulty detecting harmful hidden signals in imaging datasets.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
68 Not checked yet · 32 Own interpretation · 5 No basis given · 1 Own result
In the paper
“In our own experience, 53 we observed that pre-processing of images is important and affects the federated model performance.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our preliminary work found that signals that identify race are also contained on synthetic data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Despite the promise and hope of artificial intelligence (AI) to improve patient care, several real-world failures of AI systems have been documented.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
- Not checked yetcites [2] Seyyed-Kalantari, 2021, Underdiagnosis bias of artificial intelligence Algorithms applied to chest Radiographs in under-served patient populations, Nat Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Before the subsequent model overhaul as a result of the publication (rather than some planned post-deployment evaluation and monitoring), it was observed that one input variable to the model was antibiotic orders by a provider—a type of data leakage since an infection is already being considered at that point.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source
- Not checked yetcites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, by using healthcare costs as proxy for the need for complex care, the model learned from a system where Black patients are less likely to seek and/or receive care compared to matched White patients. 1”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A review of published AI articles between 2015 and 2019 in six disciplines found that data used for imaging AI models largely come from three states—California, Massachusetts, and New York—with none from 34 states. 22”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [22] Kaushal, 2020, Geographic distribution of US cohorts used to train deep learning Algorithms, JAMA source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Zech et al demonstrated a similar finding where models predicted pneumonia by learning where the CXRs was obtained from. 20”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [20] Zech, 2018, Variable generalization performance of a deep learning model to detect pneumonia in chest Radiographs: A cross-sectional study, PLoS Med source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI developers must always consider whether the proxy of a target outcome they are predicting or optimizing may be reflective of a systemic bias.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Even large repositories like the UK Biobank that represents a prospective cohort of 500,000 patients are limited for some patient groups: only 6% of the cohort are of non-European ancestry. 24”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [24] Privé, 2022, Using the UK Biobank as a global reference of worldwide populations: application to measuring ancestry diversity from GWAS summary Statistics, Bioinformatics source
plainn = 500,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Bias in data is one of the major source of biased algorithmic outcome. 18,19”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [18] Mehrabi, 2022, A survey on bias and fairness in machine learning, ACM Comput Surv source
- Not checked yetcites [19] Drukker, 2023, Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment, J Med Imaging (Bellingham)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The algorithm referred less Black patients with a similar disease burden compared to White patients. 1”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A similar example is an algorithm used to predict hospital length of stay to consult case managers for patients with shorter predicted length of stay to maximize throughput. 16 The algorithm (which was abandoned before deployment) learnt that patients from less affluent zip codes are likely to have longer hospital stays and may not benefit from case management facilitation of early discharge.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [16] Nordling, 2019, A fairer way forward for AI in health care, Nature source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For example, Obermeyer et al demonstrated racial bias in a commercial prediction algorithm already in use for hundreds of million ambulatory patients.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Bias audits for models tend to be piecemeal—typically performed only during model development and validation; health equity considerations should commence during data collection and curation through post-deployment monitoring. 12”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [12] Wang, 2022, A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models, J Am Med Inform Assoc source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [15] Roberts, 2021, Common pitfalls and recommendations for using machine learning to detect and Prognosticate for COVID-19 using chest Radiographs and CT scans, Nat Mach Intell source
- Not checked yetcites [20] Zech, 2018, Variable generalization performance of a deep learning model to detect pneumonia in chest Radiographs: A cross-sectional study, PLoS Med source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The algorithm developers in this case did not use a race variable in their model input to avoid bias.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In cases where remote clinicians were deployed to monitor the algorithm outputs, the constant interruption (up to 18% of hospitalized patients generated an alert) led the floor nurses to cover the video camera to limit workflow disruption. 13,14”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source
- Not checked yetcites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Health outcome prediction may be biased even when attributes that are associated with suboptimal care are hidden from a neural network agent.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The assumption that inclusion of more minority groups in the training data set is sufficient to fix bias is incorrect for the reasons above and since patients exist at the intersection of multiple group representation e.g. Black, Black Males/Females, Black Males/Females (young or old), Old Black Males/Females (with/without insurance).”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Presently, it would be difficult to further evaluate model bias on these data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
hedgedn = 500,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“An investigation of bystander CPR response for witnessed cardiac arrest found that Black and Hispanic patients or females are less likely to get CPR regardless of income level or neighborhood where the cardiac arrest occurred. 34”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [34] Garcia, 2022, Racial and ethnic differences in bystander CPR for witnessed cardiac arrest, N Engl J Med source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Attempts to be race agnostic or race neutral are challenging because it remains difficult to remove race and race proxies like pattern of health service utilization because of patient and/or provider factors.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The “race signal” (or societal standing in general) in images may be used to inform clinical predictions and optimizations especially in this era of multimodal modeling. 31”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [31] Zou, 2023, Implications of predicting race variables from medical images, Science source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Recent NIH funding through AIM-AHEAD 37 and Bridge2AI 38 programs identifies this as the most challenging issue for mitigating bias in available data sets; in addition to a lack of diversity in the research terms and the absence of experts to provide guidance on limitations of the data.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [37] AIM-AHEAD . AIM-AHEAD . Available from : https://www.aim-ahead.net/
- Not checked yetcites [38] BRIDGE2AI . BRIDGE2AI – Propelling Biomedical Research with Artificial Intelligence . Available from : https://bridge2ai.org/
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“More concerning about these models is that information on social determinants of health like area deprivation index are learned and encoded on the CXRs.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Signals to detect race and other sensitive attributes may become even easier to learn during training.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“Inappropriate use among historically underserved patients has been demonstrated in other areas of diagnostic imaging. 33”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [33] Colwell, 2022, Patient race or Ethnicity and the use of diagnostic imaging: A systematic review, J Am Coll Radiol source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Recent studies have shown that AI models can learn features that are “invisible” to a human expert.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [27] Pyrros, 2022, Detecting racial/ethnic health disparities using deep learning from frontal chest radiography, J Am Coll Radiol source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Generative adversarial networks and latent diffusion models have been used to create synthetic data sets like brain MRI based on age, sex, and brain structure volumes. 41”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [41] Pinaya, 2022 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For example, bias exist in who gets specific types of imaging, and when.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [32] Shan, 2021, Patient race/Ethnicity and diagnostic imaging utilization in the emergency Department: A systematic review, J Am Coll Radiol source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In fact, these models have been shown to demonstrate the same bias patterns even when protective attributes are removed from the model variables. 1”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Bias in clinical parameters such as glomerular filtration rate, pulse oximetry (with undiagnosed hidden hypoxemia for historically underserved patients), 35 differences in ICU severity scoring 36 shape radiology data sets for machine learning through a cascade effect as they affect ordering patterns for medical images.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [35] Sjoding, 2020, Racial bias in pulse Oximetry measurement, N Engl J Med source
- Not checked yetcites [36] Sarkar, 2021, Performance of intensive care unit severity scoring systems across different Ethnicities in the USA: a retrospective observational study, Lancet Digit Health source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Blacks and Hispanic patients who survive an out-of-hospital cardiac arrest and are included in a data set may be different from those who died before reaching the hospital and are included in the database.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“It is important to note that in some countries collection of information such as race and ethnicity are prohibited. 25,26”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [25] Bleich E . Race Policy in France . Available from : https://www.brookings.edu/articles/race-policy-in-france/
- Not checked yetcites [26] European Union Agency for Fundamental Rights . Second European Union Minorities and Discrimination Survey - Main results . 2017 . Available from : https://fra.europa.eu/en/publication/2017/second-european-union-minorities-and-discrimination-survey-main-results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is further complicated when models can pick up “hidden signals”—clinical, demographic or technology-related confounders—in the data sets that may not be obvious to the model developer.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“CXRs can also be used to predict demographic information—including insurance, self-reported race, and age. 28,29”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Up- and downsampling may worsen bias.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“There is increased development of fusion models that combine other data sources to medical imaging data (pathology, clinical tabular data, genomics among others). 55”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [55] Huang, 2020, Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines, NPJ Digit Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“AI models learn the underlying data set distribution instead of disease characteristics leading to model failure. 51”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [51] Trivedi, 2022, Deep learning models for COVID-19 chest X-ray classification: preventing shortcut learning using feature Disentanglement, PLoS One source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The need for visual inspection of the images represents obvious limitations of large-scale data set generation, but since these models are increasingly available to the public, guidelines will be required to emphasize their limitations and direct their appropriate use.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A milestone in medical imaging data was achieved in 2022 with the publication of RadImageNet, 43 an open radiologic deep learning data set that consists of 1.35 million radiologic images for efficient transfer learning.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [43] Mei, 2022, Radimagenet: an open Radiologic deep learning research Dataset for effective transfer learning, Radiol Artif Intell source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The fact that these models demonstrate this superior ability even when the images are modified in such a way that they are barely perceptible as medical images implies that mitigating bias will be difficult at best.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These “hidden signals” in the context of ongoing demonstrations of bias beyond imaging presents a challenge because biases exist along the AI lifecycle even before the modeling step.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the surprising performance of AI models to detect these signals, there are limitations in model explainability.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These techniques do not address sampling selection bias such as what in the example described earlier on Black and Hispanic patients less likely to survive an out-of-hospital cardiac arrest.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ascertaining that these signals do not influence predictions and optimizations by AI becomes even more problematic.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For example, CXRs can be used to predict ICD-10 codes related to cardiac disease (including conduction abnormalities) with accuracies of above 0.76. 27”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [27] Pyrros, 2022, Detecting racial/ethnic health disparities using deep learning from frontal chest radiography, J Am Coll Radiol source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Several strategies are applied to mitigate this imbalance—including downsampling the majority class, upsampling the minority class, use of image augmentation techniques—and can change the training data set composition or cause model overfitting in small data sets with tightly curated features causing poor performance for the minority class.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that these patients would be excluded from an imaging cohort evaluating cardiac disease outcomes.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Black, Hispanic, and non-White patients are less likely to undergo diagnostic imaging in the emergency department. 32”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [32] Shan, 2021, Patient race/Ethnicity and diagnostic imaging utilization in the emergency Department: A systematic review, J Am Coll Radiol source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“This phenomenon, also called shortcut learning, can be mitigated using various techniques including feature disentanglement (where you reward a model when it learns disease characteristics and penalize the model when it learns features that can discriminate between the original data sets where the images are obtained) 51 ; applying pre-processing techniques like histogram equalization to correct”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [51] Trivedi, 2022, Deep learning models for COVID-19 chest X-ray classification: preventing shortcut learning using feature Disentanglement, PLoS One source
- Not checked yetcites [52] Tartaglione, 2020, Unveiling COVID-19 from CHEST X-ray with deep learning: A hurdles race with small data, Int J Environ Res Public Health source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite obtaining the relevant regulatory approval, safely translating models from bench to bedside is even more challenging than algorithm development.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model implementation and deployment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Vendors must navigate the convoluted process of integrating with current hospital IT systems, and engaging clinicians and patients.
- Rests on an assumption: In many cases, models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.
- Rests on an assumption: Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is critical to note that regulatory approval at its current state does not guarantee model fairness.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Bias evaluation and mitigation frequently occurs post-hoc.
In the paper
“The FDA, Canadian Agency for Drugs and Technology, European Union, and the UK code of conduct for data-driven healthcare technology only provide high-level guidance to promote fairness.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [11] Wawira Gichoya, 2021, Equity in essence: a call for Operationalising fairness in machine learning for Healthcare, BMJ Health Care Inform source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“An interesting observation is that the type of explanation matters when it comes to biasing the end user.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model implementation and deployment
What the sources say
- Not checked yetcites [62] Kiani, 2020, Impact of a deep learning assistant on the histopathologic classification of liver cancer, NPJ Digit Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Bias mitigation is especially important as we understand human–machine partnership, with early findings showing worsening performance for experts when presented with biased model outputs.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
What the sources say
- Not checked yetcites [62] Kiani, 2020, Impact of a deep learning assistant on the histopathologic classification of liver cancer, NPJ Digit Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, only 34% had multi-institutional validation and reported which institution(s) was(were) used for validation. 56”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [56] Khunte, 2023, Trends in clinical validation and usage of US food and Drug Administration-cleared artificial intelligence Algorithms for medical imaging, Clin Radiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“It is thus not surprising that most of these algorithms, despite regulatory approval, demonstrate bias when deployed to various clinical settings especially when the target population is different from the population the model was trained on.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [56] Khunte, 2023, Trends in clinical validation and usage of US food and Drug Administration-cleared artificial intelligence Algorithms for medical imaging, Clin Radiol source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Model validation remains a barrier that limits incorporation of AI into clinical care.
- Rests on an assumption: Many factors affect AI validation, including high costs and lack of consensus on proper validation protocol.
In the paper
“This “black box inside a black box” nature of commercial AI models and concerns for protecting intellectual is recipe for AI perpetuating or even scaling health inequities. 60”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [60] Tariq, 2020, Current clinical applications of artificial intelligence in Radiology and their best supporting evidence, J Am Coll Radiol source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Proprietary models are usually trained and validated on private data sets that are not accessible for other researchers to evaluate the model.
- It may state two separate claims as one. 29
- Probably not a substantial claim. 21
In the paper
“Moreover, FedAvg is biased to research sites contributing larger data sets with poor performance reported for smaller data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In our experimental evaluation of the reading race paper where we demonstrated superhuman ability for AI to predict self-reported race across multiple modalities, we found that some transfer learning tasks are better in race detection. 28”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [28] Gichoya, 2022, AI recognition of patient race in medical imaging: a Modelling study, Lancet Digit Health source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“When we used the FedAvg algorithm, the model improved its generalizability on external data sets but at the expense of some degradation in internal validation.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Existing AI data sets are acquired from limited geographical regions; most lack information necessary to evaluate subgroup performance of models.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [22] Kaushal, 2020, Geographic distribution of US cohorts used to train deep learning Algorithms, JAMA source
- Not checked yetcites [23] Yi, 2022, Demographic reporting in publicly available chest radiograph data SETS: opportunities for mitigating sex and racial disparities in deep learning models, J Am Coll Radiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition to observing overall performance decline secondary to differences in data distribution after model deployment, subgroup evaluations have shown varying levels of performance with poor performance especially for historically underserved patients. 1,2”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [1] Obermeyer, 2019, Dissecting racial bias in an algorithm used to manage the health of populations, Science source
- Not checked yetcites [2] Seyyed-Kalantari, 2021, Underdiagnosis bias of artificial intelligence Algorithms applied to chest Radiographs in under-served patient populations, Nat Med source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The algorithm (which was abandoned before deployment) learnt that patients from less affluent zip codes are likely to have longer hospital stays and may not benefit from case management facilitation of early discharge.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > AI pitfalls when defining a task
What the sources say
- Not checked yetcites [16] Nordling, 2019, A fairer way forward for AI in health care, Nature source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Problems in data set development include limited patient diversity, limited resolution and missing clinical confounders, limited or suboptimal quality annotations, and lack of data standards and best practice to ensure quality and reproducibility.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The failure and subsequent withdrawal of the Epic (TM) sepsis model used by 180 customers representing hundreds of hospitals provides a template to study AI failure across a spectrum. 13,14”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source
- Not checked yetcites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that bias permeates every step of the lifecycle and is a sequela of human, machine, and systems factors.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“A review of 23 CXR data sets found that although the majority reported age and sex, only 8.7% reported race or ethnicity and 4.3% reported insurance status. 23”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [23] Yi, 2022, Demographic reporting in publicly available chest radiograph data SETS: opportunities for mitigating sex and racial disparities in deep learning models, J Am Coll Radiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Efforts to mitigate these biases are challenging due to—lack of a unified definition of bias 10 ; focus on a statistical definition of bias and a technocentric definition of bias without involving patients and communities; and post-hoc reflection on bias rather than as a deliverable by design. 11”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [10] Narayanan A . Tutorial: 21 fairness definitions and their politics . 2018 . Available from : https://www.youtube.com/watch?v=jIXIuYdnyyk
- Not checked yetcites [11] Wawira Gichoya, 2021, Equity in essence: a call for Operationalising fairness in machine learning for Healthcare, BMJ Health Care Inform source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Upon testing at Michigan Medicine, the algorithm only identified 7% of patients whose sepsis diagnosis was missed by a clinician, failed to flag 67% of patients with sepsis despite generating alerts on 18% of all hospitalized patients resulting in alert fatigue.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction
What the sources say
- Not checked yetcites [13] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Intern Med source
- Not checked yetcites [14] Habib, 2021, The epic sepsis model falls short-the importance of external validation, JAMA Intern Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It should be noted that model auditing and bias detection, which benefit from diverse perspectives during training and validation, are more difficult in federated learning where only the weights are shared across the groups of modelers.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Some groups have suggested datasheets 39,40 —a checklist to guide model developers on how to use data—to address this problem.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [39] Gebru T , Morgenstern J , Vecchione B , Vaughan JW , Wallach H , Daumé III H . Datasheets for Datasets . arXiv 2018 .
- Not checked yetcites [40] Garbin C , Rajpurkar P , Irvin J , Lungren MP , Marques O . Structured dataset documentation: a datasheet for CheXpert . arXiv 2021 .
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The availability of a radiology specific data set to facilitate transfer learning is a step towards efficiency in model development.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“CXRs have also been used to predict biological age, cardiac disease outcomes and healthcare cost at 1, 3 and 5 years. 30”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [30] Sohn, 2022, Prediction of future Healthcare expenses of patients from chest Radiographs using deep learning: a pilot study, Sci Rep source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Overall, when local model performance matters—then personalized federated learning models (such as FedBN) should be prioritized, while for external performance, FedAvg should be used. 53”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
What the sources say
- Not checked yetcites [53] Peng, 2022, Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals, J Am Med Inform Assoc source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Federated learning—whereby models are trained across multiple institutions without the data being shared except for the model weights—has been proposed as a solution to improve model performance across diverse patient groups.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such data set will also discourage the use of Frankenstein data sets—assembled from other data sets and redistributed under a new name—as pretraining can be done on RadImageNet and fine-tuned on a smaller data set.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Beyond individual images, class imbalance also occurs due to distribution of positive versus negative cases (since most studies are normal), and across various demographic distributions within data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Class imbalance occurs frequently in radiology.
In the paper
“In fact, only 8% of the UK Biobank has been used for research, with most of this research focused on patients of European ancestry.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in data acquisition and collection
What the sources say
- Not checked yetcites [24] Privé, 2022, Using the UK Biobank as a global reference of worldwide populations: application to measuring ancestry diversity from GWAS summary Statistics, Bioinformatics source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These models have a larger feature space used for training, which require feature selection early in the training process.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model development
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Radiology AI performance is affected by other underlying practice patterns which can appear as hidden signals in AI data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Bias risk assessment tools can facilitate this process. 65”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
What the sources say
- Not checked yetcites [65] Moons, 2019, PROBAST: A tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration, Ann Intern Med source
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As highlighted in Figure 1, bias at each stage of model development can occur due to machine factors, human factors or a combination of both.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“This test data must be created by the local IT team and then curated carefully by the local clinicians who may not be familiar with the requisite subgroup model performance evaluation and other unintended consequences of AI bias. 61”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model implementation and deployment
What the sources say
- Not checked yetcites [61] Ahluwalia, 2023, The subgroup imperative: chest X-ray Classifier generalization gaps in patient, setting, and pathology subgroups, Radiology: Artificial Intelligence
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: In many cases, models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.
None of the sources cited for this claim have been checked yet.
In the paper
“Following guidelines for research documentation and complete reporting increases transparency. 63,64”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
What the sources say
- Not checked yetcites [63] Mongan, 2020, Checklist for artificial intelligence in medical imaging (CLAIM): A guide for authors and reviewers, Radiol Artif Intell source
- Not checked yetcites [64] Bossuyt, 2015, STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies, Radiology source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, bias at one stage can be propagated to other stages, hence efforts to detect and mitigate bias should be at each stage.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“For example, the use of a machine learning algorithm by pathologists to differentiate different liver cancer types found that the pathologist accuracy did not improve when the model’s prediction was correct; however, the pathologist performance worsened when the model’s prediction was incorrect. 62”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model implementation and deployment
What the sources say
- Not checked yetcites [62] Kiani, 2020, Impact of a deep learning assistant on the histopathologic classification of liver cancer, NPJ Digit Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Neglecting continents and ethnically diverse locations perpetuates health inequities and hinders the growth and applicability of AI models.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In fact, a review of the use of clinical trials for machine learning in healthcare found only 41 randomized clinical trials, and most did not adhere to accepted reporting guidelines and had limited participation from minority groups. 58”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [58] Plana, 2022, Randomized clinical trials of machine learning interventions in health care: A systematic review, JAMA Netw Open source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“A review of 151 imaging AI products cleared by the FDA by November 2021 showed that only 64% of these products used clinical data for their validation; and these clinical data had limited information on demographic and technical confounders (only 4% had patient demographics and 5% reported machine specifications). 56”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [56] Khunte, 2023, Trends in clinical validation and usage of US food and Drug Administration-cleared artificial intelligence Algorithms for medical imaging, Clin Radiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, relying on data sets with narrow geographic and other dimensions of diversity can further biases and limits generalizability of AI technologies.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“The regulatory landscape is changing as exemplified by the actions of four federal US agencies who recently issued a joint statement against biased models and their use including non-medical use like employment discrimination. 59”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [59] Chopra R , Khan LM . Joint statement on enforcement efforts against discrimination and bias in automated systems . Available from : https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CFPB-AI-Joint-Statement%28final%29.pdf
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Failure to provide transparent documentation and detailed methodologies especially for commercial algorithms has potential to perpetuate biases and prevent critical analysis.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ensuring that AI works for all will require diverse teams in addition to diverse data sets.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Despite infrequent to even non-existent external validation for both FDA approved and peer reviewed articles, when external validation is performed, it is typically limited in scope.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source
plainn = 240- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most challenging issue for bias in medical imaging arises from techniques to address class imbalance and how to evaluate and mitigate harmful “hidden signals” in imaging data sets, as they can lead to shortcut learning and reinforce existing biases.”
§ AI pitfalls and what not to do: mitigating bias in AI > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bias occurs across the AI life cycle, and hence must be tackled at each step from problem definition all the way through post-deployment monitoring.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“For example, these organizations issue statements like “consider evaluation of bias”, without clear instructions on how the evaluation should be performed, and what consequences result from not performing such evaluation. 11”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [11] Wawira Gichoya, 2021, Equity in essence: a call for Operationalising fairness in machine learning for Healthcare, BMJ Health Care Inform source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to limited AI explainability, these hidden signals can be difficult to detect and have led to algorithmic bias that encrypts health inequities into practice.”
§ AI pitfalls and what not to do: mitigating bias in AI > Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Even on what would be considered a small sample size for external validation, nearly half of the 86 studies reported a modest increase in external performance with nearly a quarter reporting a substantial decrease, reiterating on the challenge of validation. 57”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“A recent systematic review found that 86 studies have a median of 240 cases used for external validation, with about 47% of which are positive cases. 57”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
What the sources say
- Not checked yetcites [57] Yu, 2022, External validation of deep learning Algorithms for Radiologic diagnosis: A systematic review, Radiol Artif Intell source
plainn = 240- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“because the approval process require resubmission when models are updated, there are no incentives to address algorithmic bias during the initial application.”
§ AI pitfalls and what not to do: mitigating bias in AI > Introduction > Pitfalls in model evaluation and validation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
Bias principles are largely considered too late, if at all, when designing AI.
“Bias principles are largely considered too late, if at all, when designing AI.”
The paper relies on this without support.
Comprehensive AI data sets are limited and expensive to curate.
“Comprehensive AI data sets are limited and expensive to curate.”
The paper relies on this without support.
Presently, AI models are trained on data collected from tertiary care academic institutions.
“While there have been initiatives to build data sets from “data deserts”, presently, AI models are trained on data collected from tertiary care academic institutions.”
The paper relies on this without support.
Class imbalance occurs frequently in radiology.
“Class imbalance occurs frequently in radiology.”
The paper relies on this without support.
Radiology images tend to have high pixel dimensions and resolutions, yet pathology is represented on a small percentage of the pixels.
“Radiology images tend to have high pixel dimensions and resolutions, yet pathology is represented on a small percentage of the pixels.”
The paper relies on this without support.
Model validation remains a barrier that limits incorporation of AI into clinical care.
“Model validation remains a barrier that limits incorporation of AI into clinical care.”
The paper relies on this without support.
Many factors affect AI validation, including high costs and lack of consensus on proper validation protocol.
“Many factors affect AI validation—including high costs of validation (technical pipeline for data set preparation and curation, diverse team of machine learning and clinical experts, regulatory approval burden), and lack of consensus on what is the proper protocol of validation.”
The paper relies on this without support.
AI models 'fail silently' i.e. they do not provide an output to the user when they do not recognize the model input.
“In these cases, the AI models “fail silently” i.e. they do not provide an output to the user when they do not recognize the model input.”
The paper relies on this without support.
AI models will make a guess estimate of the prediction for wrong input data without providing the level of uncertainty.
“These AI models will make a guess estimate of the prediction for wrong input data without providing the level of uncertainty.”
The paper relies on this without support.
The greatest challenge of bias detection and mitigation remains the lack of a consensus definition of algorithmic bias despite many toolkits developed for this purpose.
“The greatest challenge of bias detection and mitigation remains the lack of a consensus definition of algorithmic bias despite many toolkits developed for this purpose.”
The paper relies on this without support.
External validation can ensure models generalize well across populations.
“External validation can ensure models generalize well across populations.”
The paper relies on this without support.
Bias evaluation and mitigation frequently occurs post-hoc.
“Bias evaluation and mitigation frequently occurs post-hoc”
The paper relies on this without support.
A review of AI reporting guidelines noted that references to bias are either absent, made in passing or included in the guideline supplement.
“A review of AI reporting guidelines noted that references to bias are either absent, made in passing or included in the guideline supplement.”
The paper relies on this without support.
Clinical trials have limitations including high costs, most trials never meet their enrollment proportions, and bias still exists in clinical trials despite their rigor.
“While some people have called for more clinical trials in validating AI, we must acknowledge the limitations of such trials including high costs, the fact that most trials never meet their enrollment proportions, and that bias still exists in clinical trials despite their rigor.”
The paper relies on this without support.
Proprietary models are usually trained and validated on private data sets that are not accessible for other researchers to evaluate the model.
“Proprietary models present an additional layer of complexity because they are usually trained and validated on private data sets that are not accessible for other researchers to evaluate the model.”
The paper relies on this without support.
Evaluation of proprietary algorithms, when performed, involves simulations to try and replicate model performance.
“Evaluation of proprietary algorithms, when performed, involves simulations to try and replicate model performance.”
The paper relies on this without support.
Vendors must navigate the convoluted process of integrating with current hospital IT systems, and engaging clinicians and patients.
“Vendors must navigate the convoluted process of integrating with current hospital IT systems, and engaging clinicians and patients.”
The paper relies on this without support.
In many cases, models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.
“Moreover, in many cases, the models do not automatically perform well out of the box and require some fine tuning and calibration to the local site data.”
The paper relies on this without support.
Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.
“Due to lack of IT infrastructure and process for continuous model monitoring, changes in software or image acquisition equipment or protocols can result in model failure that may not be immediately detected, if at all.”
The paper relies on this without support.
Human–machine collaboration is now at the forefront of improving model performance, and to prevent unintended consequences including automation bias.
“Human–machine collaboration is now at the forefront of improving model performance, and to prevent unintended consequences including automation bias.”
The paper relies on this without support.
Broad grouping of underrepresented populations into the “Other” category is common when researchers have small sample sizes.
“We also caution against broad grouping of underrepresented populations into the “Other” category which is common when researchers have small sample sizes.”
The paper relies on this without support.
Mathematical approaches of fairness evaluation (e.g., fairness through unawareness, demographic parity, equalized odds or opportunity) may overlook nuanced biases and fail to address systemic issues.
“We urge caution on relying on only mathematical approaches of fairness evaluation (e.g. relying solely on fairness through unawareness, demographic parity, or equalized odds or opportunity), as these approaches may overlook nuanced biases and fail to address systemic issues.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“AI systems may also produce outputs that are skewed by biases related to race, ethnicity, sex, religion, politics, and other factors. [5, 23, 53–56]”
From Autonomous artificial intelligence, scientific research, and human values (2026) · cites this paper as [53]
In the citing paper
“One of the first strategies that can be used to identify bias related to the dataset is dataset evaluation against a set of predefined criteria (searching for exclusion bias, selection bias, recall bias, observer bias, and prejudice bias) and comprehensive data analysis.47”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [47]
In the citing paper
“Statistical comparison of model output according to different patient groups or confounders that may exist in the training dataset, such as the gender or age of patients. Potential discrepancies in group results could indicate a source of bias that can affect the final results.48”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [48]
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“BMJ Health Care Inform 2021; 28(1): e100289.”
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 1×
“18,19 Comprehensive AI data sets are limited and expensive to curate.”
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“For example, a developer will have to decide between aiming for group fairness (Black vs white patients, patients on government healthcare programs, or a specific age group) vs individual fairness 49 (where two individuals with similar characteristics are treated in the same manner).”
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 2×
“50 Attempts to be race agnostic or race neutral are challenging because it remains difficult to remove race and race proxies like pattern of health service utilization because of patient and/or provider factors.”
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · cited 1×
“20 To overcome this confounding, progress must be made to improve model training, evaluation, and explainability.”
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“63,64 We encourage researchers to ensure that their work is reproducible, allowing for independent validation and bias detection.”
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C., Glasziou, P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,360
Racial bias in pulse oximetry measurement
Sjoding, M. W., Dickson, R. P., Iwashyna, T. J., Gay, S. E., & Valley, T. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmc2029240 · Cited by 1,114 · cited 1×
“…in clinical parameters such as glomerular filtration rate, pulse oximetry (with undiagnosed hidden hypoxemia for historically underserved patients), 35 differences in ICU severity scoring 36 shape radiology data sets for machine learning through a cascade effect as they affect ordering patterns…”
Identifying and Avoiding Bias in Research
Pannucci, C. J., & Wilkins, E. G. (2010) · Plastic and Reconstructive Surgery · doi:10.1097/prs.0b013e3181de24bc · Cited by 1,079 · cited 1×
“Radiol Artif Intell 2022; 4(3): e210064.”
Research works printed but not linked: 5
- Gebru T , Morgenstern J , Vecchione B , Vaughan JW , Wallach H , Daumé III H . Datasheets for Datasets . arXiv 2018 .
- Garbin C , Rajpurkar P , Irvin J , Lungren MP , Marques O . Structured dataset documentation: a datasheet for CheXpert . arXiv 2021 .
- Chambon P et al . RoentGen: Vision-Language Foundation Model for Chest X-ray Generation . arXiv 2022 .
- Tsipras D , Santurkar S , Engstrom L , Ilyas A , Madry A . From ImageNet to Image Classification: Contextualizing Progress on Benchmarks . arXiv 2020 .
- Yang K , Yau J , Fei-Fei L , Deng J , Russakovsky O . A Study of Face Obfuscation in ImageNet . arXiv 2021 .
Cited by
Sorted by citations
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567
“Gichoya et al. (2023) argue that AI systems, trained on extensive data-sets sourced from the internet, inherently mirror societal biases.”
BackgroundBias recognition and mitigation strategies in artificial intelligence healthcare applications
Hasanzadeh, F., Josephson, C. B., Waters, G., Adedinsewo, D., Azizi, Z., & White, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01503-7 · Cited by 320
Generative AI for Customizable Learning Experiences
Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024) · Sustainability · doi:10.3390/su16073034 · Cited by 274
“Mitigation strategies include the resampling and pre-processing of data used for training the models [68], using diverse and representative datasets [69], feedback loops [70], etc.”
MethodWeiner, E., Dankwa‐Mullan, I., Nelson, W. A., & Hassanpour, S. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000810 · Cited by 306
AI agents under threat: A survey of key security challenges and future pathways
Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3716628 · Cited by 210
Metadata
- Authors
- Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S.
- Affiliations
- Emory University (US)
- Beth Israel Deaconess Medical Center (US)
- Arizona State University (US)
- York University (CA)
- Indiana University – Purdue University Indianapolis (US)
- Corresponding author
- Gichoya, Judy Wawira
- Purkayastha, Saptarshi
- ORCID
- Venue
- The British Journal of Radiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-09-25
- Volume
- 96
- Issue
- 1150
- Pages
- 20230023-20230023
- ISSN
- 0007-1285 (ISSN-L)
- 1748-880X
- PMC
- PMC10546443
- OpenAlex
- W4386624058
- Semantic Scholar
- ec14f2eda005b47d84f2be953a70ed5981fcc1aa
- Cited by
- 277
- References
- 66
- Citations by year
- 2026: 88 · 2025: 118 · 2024: 67 · 2023: 4
- FWCI
- 3.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- British Journal of Radiology (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (1928481)
- Radiological Society of North America (#EIHD2204)
- National Institutes of Health (75N92020C00021, 5r01eb017205-07, 75N92020C00008, R01-EB017205, EB017205)
- National Heart, Lung, and Blood Institute (75N92020C00008, 75N92020C00021)
- National Institute of Biomedical Imaging and Bioengineering (75N92020C00021, R01-EB017205, EB017205, 75N92020C00008)
- National Institute on Minority Health and Health Disparities
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic bias, bias mitigation, artificial intelligence in radiology, AI lifecycle, model training, model deployment, algorithmic fairness
- MeSH terms
- Artificial Intelligence, Humans, Learning, Radiology, Bias, Disease Progression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
