CoChat
Sign Up
Cited by 145Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIRadiomics and Machine Learning in Medical Imaging

Mitigating Bias in Radiology Machine Learning: 1. Data Handling

Open access
Rouzrokh, Pouria, Khosravi, Bardia, Faghani, Shahriar, Moassefi, Mana, Vera Garcia, Diana V., Singh, Yashbir, Zhang, Kuan, Conte, Gian Marco +1 more·2022·Radiology: Artificial Intelligence
DOI 10.1148/ryai.210290PMC PMC9533091
Open this paper in CoChat — free account

Abstract

Minimizing bias is critical to adoption and implementation of machine learning (ML) in clinical practice. Systematic mathematical biases produce consistent and reproducible differences between the observed and expected performance of ML systems, resulting in suboptimal performance. Such biases can be traced back to various phases of ML development: data handling, model development, and performance evaluation. This report presents 12 suboptimal practices during data handling of an ML study, explains how those practices can lead to biases, and describes what may be done to mitigate them. Authors employ an arbitrary and simplified framework that splits ML data handling into four steps: data collection, data investigation, data splitting, and feature engineering. Examples from the available research literature are provided. A Google Colaboratory Jupyter notebook includes code examples to demonstrate the suboptimal practices and steps to prevent them. Keywords: Data Handling, Bias, Machine Learning, Deep Learning, Convolutional Neural Network (CNN), Computer-aided Diagnosis (CAD) © RSNA, 2022

AI snapshot

Written from the paper’s full text.

TL;DR

This report describes 12 suboptimal data-handling practices that can introduce systematic mathematical bias into radiology machine learning studies and offers mitigation strategies. It organizes data handling into four steps—data collection, data investigation, data splitting, and feature engineering—and provides a Google Colaboratory notebook with code examples.

Key findings

  • The authors identify 12 major practices during data handling that predispose ML systems to mathematical bias, spanning data collection, data investigation, data splitting, and feature engineering.
  • Systematic biases produce consistent and reproducible differences between observed and expected performance, limiting real-world application; the report notes more than 8000 radiology AI publications worldwide from 2000 to 2018.
  • Specific risks include single-source data, inadequate exploratory data analysis, leakage between training and test sets, imbalanced datasets, overfitting to hyperparameters, improper feature removal or scaling, and mismanagement of missing data.
  • In an illustrative hip radiograph example, a 2% complication incidence among 10 000 patients yields 9800 patients without complications versus 200 with complications; with a conventional 10%–20% test holdout, only 20–40 patients with complications would be in the test set.

Methods

  • Narrative methodological report/review intended for radiologists interested in ML who are not expert data scientists, using computer vision examples but noting most principles apply to other ML applications.
  • The authors employ an arbitrary and simplified framework that splits ML data handling into four steps: data collection, data investigation, data splitting, and feature engineering.
  • Examples are drawn from available research literature, and a Google Colaboratory Jupyter notebook includes code examples to demonstrate suboptimal practices and prevention steps.

Limitations

  • The paper states that its four-step data-handling framework is arbitrary and oversimplified, and that ML systems are not developed as a linear process.
  • The report states it considers only mathematical forms of bias, not social and ethical forms, and is not an exhaustive list of problems or solutions; bias mitigation strategies are problem specific and may be unnecessary or inappropriate in some situations.
  • The paper also states that not all bias-prone practices can be easily replaced or improved, and some biases may be tolerated or even helpful depending on data and task; as evident from the text, this is a conceptual review without an empirical sample or systematic review methodology.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1148/ryai.210290

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 27 cited sources checked · 0 supported · 0 flagged

  • Premisecitation
    § Introduction

    Models may not generalize well if training data are collected from a single source.

    “Models may not generalize well if training data are collected from a single source (15).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] 15. Doshi-Velez F , Kim B . Considerations for Evaluation and Generalization in Interpretable Machine Learning . In: Escalante HJ , Escalera S , Guyon I , et al. , eds. Explainable and Interpretable Models in Computer Vision and Machine Learning. The Springer Series on Challenges in Machine Learning . Cham, Switzerland: : Springer; , 2018. ; 3 – 17 . [Google Scholar] source

      Not checked in the free run.

  • Interpretationnone given
    § Abstract

    Twelve major practices during data handling predispose machine learning systems to mathematical bias.

    “This report describes 12 major practices during data handling that predispose machine learning systems to mathematical bias.”
    Quote in the paperSubstantial claim
  • Premisecitation
    § Introduction

    Discussion of proper data handling in existing guidelines for machine learning system development is often insufficient.

    “Although several guidelines exist for the proper development of ML systems, discussion of proper data handling is often insufficient (8,9).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [8] 8. Kumar A , Boehm M , Yang J . Data Management in Machine Learning: Challenges, Techniques, and Systems . In: Proceedings of the 2017 ACM International Conference on Management of Data . New York, NY: : Association for Computing Machinery; , 2017. ; 1717 – 1722 . [Google Scholar] source

      Not checked in the free run.

    • cites [9] 9. Polyzotis N , Roy S , Whang SE , Zinkevich M . Data Management Challenges in Production Machine Learning . In: Proceedings of the 2017 ACM International Conference on Management of Data . New York, NY: : Association for Computing Machinery; , 2017. ; 1723 – 1726 . [Google Scholar]

      Not checked in the free run.

  • Premisecitation
    § Introduction

    Training a model on multiple data types may improve the performance of machine learning systems.

    “Training a model on multiple data types may improve the performance of ML systems (14).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] 14. Huang S-C , Pareek A , Seyyedi S , Banerjee I , Lungren MP . Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines . NPJ Digit Med 2020. ; 3 ( 1 ): 136 . [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Premisecitation
    § Abstract

    Systematic mathematical biases produce consistent and reproducible differences between the observed and expected performance of machine learning systems, resulting in suboptimal performance.

    “Systematic mathematical biases produce consistent and reproducible differences between the observed and expected performance of ML systems, resulting in suboptimal performance.”
    • Also stated in: § Introduction · plain
      “systematic bias produces consistent and reproducible differences between observed and expected performance (4).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4] 4. Krishna R , Maithreyi R , Surapaneni KM . Research bias: a review for medical students . J Clin Diagn Res 2010. ; 4 ( 2 ): 2320 – 2324 . https://www.jcdr.net/article_abstract.aspx?issn=0973-709x&year=2010&volume=4&issue=2&page=2320%20-%202324&issn=0973-709x&id=677 . [Google Scholar]

      Not checked in the free run.

  • Classified as probably not a substantial claim. 21
  • May combine two claims. 17

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § Abstract

    Biases in machine learning systems can be traced back to various phases of machine learning development: data handling, model development, and performance evaluation.

    “Such biases can be traced back to various phases of ML development: data handling, model development, and performance evaluation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § Introduction

    Human-labeled annotations require a clear protocol and measurement of intra- and interannotator reliability to reduce recall, observer, or measurement biases.

    “Human-labeled annotations (eg, segmentation masks or bounding boxes) require a clear protocol and measurement of intra- and interannotator reliability to reduce recall, observer, or measurement biases (24).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] 24. Mongan J , Moy L , Kahn CE Jr . Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers . Radiol Artif Intell 2020. ; 2 ( 2 ): e200029 . [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § Introduction

    Incorporation of domain expertise can enrich exploratory data analysis.

    “Incorporation of domain expertise can enrich EDA (34).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] 34. Langer T , Meisen T . Towards Utilizing Domain Expertise for Exploratory Data Analysis . In: Proceedings of the 12th International Symposium on Visual Information Communication and Interaction . New York, NY: : Association for Computing Machinery; ; 2019. ; 1 – 5 . [Google Scholar]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § Introduction

    Public datasets may be prone to biases and may not represent or report the age, race, or sex of the original population.

    “Public datasets may be prone to biases and may not represent or report the age, race, or sex of the original population (25).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [25] 25. Garcia Santa Cruz B , Bossa MN , Sölter J , Husch AD . Public Covid-19 X-ray datasets and their impact on model bias - A systematic review of a significant problem . Med Image Anal 2021. ; 74 : 102225 . [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Introduction

    Chest radiograph classifiers trained on publicly available datasets may have multiple racial, sex, and socioeconomic biases that reflect the patient distribution in their training data, resulting in poor generalizability.

    “For example, chest radiograph classifiers trained on publicly available datasets may have multiple racial, sex, and socioeconomic biases that reflect the patient distribution in their training data (16), resulting in poor generalizability.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [16] 16. Seyyed-Kalantari L , Liu G , McDermott M , Chen IY , Ghassemi M . CheXclusion: Fairness gaps in deep chest X-ray classifiers . Pac Symp Biocomput 2021. ; 26 : 232 – 243 . [PubMed] [Google Scholar] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § Introduction

    Medical data for underrepresented groups may be less available and ignored or misinterpreted by ML systems that have not addressed missing values properly during training, resulting in prejudice and selection biases.

    “Therefore, the medical data for such underrepresented groups may be less available and ignored (or interpreted mistakenly) by ML systems that have not addressed missing values properly during training, resulting in prejudice and selection biases.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: All real-world data are likely to have missing data.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § Introduction

    Data split at levels higher than the patient level, such as institution level, may be beneficial if there are consistent differences in how a hospital handles patient scans.

    “Finally, there are instances where data split at levels higher than the patient level may be beneficial. For example, if there are consistent differences in how a hospital handles patient scans, it may be valuable to separate data at the institution level.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Introduction

    Unrecognized bias may contribute to suboptimal results.

    “Unrecognized bias may contribute to suboptimal results.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § Introduction

    Normalization may amplify noise in features with almost constant values, and standardization can significantly distort data values in the presence of outliers.

    “As with other examples, normalization may amplify the noise in features with almost constant values, and standardization can significantly distort data values in the presence of outliers (48).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] 48. Cendrero SM . Warning About Normalizing Data . business.blogthinkbig.com. https://business.blogthinkbig.com/warning-about-normalizing-data/. Published 2018. Accessed October 2021 .

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Introduction

    Removing all patients with missing data can result in bias and fairness issues, including exacerbating small or imbalanced datasets and worse model performance on underrepresented populations.

    “This approach can result in bias and fairness issues, as (a) removal of cases may exacerbate the problem of small or imbalanced datasets, and (b) the missing data are not random and more commonly affects a specific population of patients; thus, the final model’s performance may be significantly worse on underrepresented populations (53,54).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [53] 53. Schafer JL , Graham JW . Missing data: our view of the state of the art . Psychol Methods 2002. ; 7 ( 2 ): 147 – 177 . [PubMed] [Google Scholar]

      Not checked in the free run.

    • cites [54] 54. Johnson DR , Young R . Toward best practices in analyzing datasets with missing data: Comparisons and recommendations . J Marriage Fam 2011. ; 73 ( 5 ): 926 – 945 . [Google Scholar]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Introduction

    Some ML repositories may have more than 40% of data missing.

    “In fact, some ML repositories may have more than 40% of data missing (49).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49] 49. Dua D , Graff C . UCI Machine Learning Repository . http://archive.ics.uci.edu/ml. Published 2017. Accessed October 2021.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § Conclusion

    ML systems may be biased in different ways.

    “Limitations do exist in ML studies, and ML systems may be biased in different ways.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § Introduction

    Observing the data itself, rather than only its statistical properties, often provides new insights.

    “While we already described several statistical measures to evaluate input data, observing the data itself (and not its statistical properties) often provides new insights.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Introduction

    Feature scaling techniques can result in poorer fit and even bias if misapplied.

    “Although these techniques are applied to improve model fit and generalizability, they can result in poorer fit, and even bias, if misapplied.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given
    § Conclusion

    Bias mitigation strategies are problem specific, and certain strategies may be unnecessary or inappropriate in specific situations.

    “Additionally, bias mitigation strategies are problem specific, and certain strategies described above may be unnecessary or even inappropriate in specific situations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisecitation
    § Conclusion

    Depending on the data and task, biases in ML systems could be tolerated or even considered helpful.

    “Depending on the data and task at hand, such biases could be tolerated or even considered helpful (58).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [58] 58. Pot M , Kieusseyan N , Prainsack B . Not all biases are bad: equitable and inequitable biases in machine learning and radiology . Insights Imaging 2021. ; 12 ( 1 ): 13 . [DOI] [PMC free article] [PubMed] [Google Scholar] Articles from Radiology: Artificial Intelligence are provided here courtesy of Radiological Society of North America

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Introduction

    Missing data may not be evenly distributed, leading to unwanted effects on the fairness of the data and the resulting models.

    “Previous research has shown that missing data may not be evenly distributed, leading to unwanted effects on the fairness of the data and the resulting models (50).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [50] 50. Martínez-Plumed F , Ferri C , Nieves D , Hernández-Orallo J . Missing the missing values: The ugly duckling of fairness in machine learning . Int J Intell Syst 2021. ; 36 ( 7 ): 3217 – 3258 . [Google Scholar]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Introduction

    Both k-fold cross-validation and nested cross-validation are prone to methodologic biases and may not always be feasible.

    “It should be noted that both the k-fold cross-validation and nested cross-validation approaches are prone to methodologic biases and may not be always feasible; therefore, specific guidelines must be followed to obtain their most reliable performance (46).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46] 46. Zheng A , Casari A . Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists . Sebastopol, Calif: : O’Reilly Media; , 2018. . [Google Scholar]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Introduction

    Underrepresented groups may be reluctant to provide sensitive information on medical or demographic questionnaires and are more likely to miss follow-up medical visits.

    “For example, underrepresented groups may be reluctant to provide sensitive information on medical or demographic questionnaires and are more likely to miss follow-up medical visits (51).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [51] 51. Gilbert DT , Fiske ST , Lindzey G . The Handbook of Social Psychology . Boston, Mass: : McGraw-Hill; , 1998. . [Google Scholar]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § Introduction

    Tree-based ML algorithms do not consider the scale of the data and often do not need feature scaling.

    “For example, ML algorithms such as tree-based models do not consider the scale of the data and often do not need feature scaling.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § Introduction

    Using k-fold cross-validation ensures that data from each patient with complications have been used once to evaluate the model's performance.

    “Using k-fold cross-validation in our example, developers can ensure that the data from each of the 200 patients with complications have been used once to evaluate the model’s performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Introduction

    In a conventional train-test split with 10%-20% held out, a highly imbalanced dataset can yield a test set with too few patients with the outcome to be representative, so reported model performance may not generalize to real-world data.

    “there is a substantial risk that the few patients with complications from the test set are not representative of the population. Therefore, developers cannot rely on the reported model performance applied to their test set, as it may not generalize to the real-world data.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Medical datasets are substantially smaller than most nonmedical datasets.
    • Rests on an assumption: Available medical data may have imbalanced distribution based on several variables.
    • Rests on an assumption: The causes of imbalance in medical datasets go beyond machine learning and may pertain to population-level frequencies or patient recruitment protocols.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Conclusion

    Addressing social and ethical biases often needs political decisions and time-consuming changes in stakeholders' behaviors.

    “Addressing such biases often needs political decisions and time-consuming changes in stakeholders’ behaviors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § Introduction

    As k models have been trained on different data, it is more likely that their mean performance is generalizable to real-world data, particularly if the variance is low.

    “Also, as the k models have been trained on different data, it is more likely that their mean performance is generalizable to real-world data, particularly if the variance is low.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Introduction

    Nested cross-validation is a variant of k-fold cross-validation that aims to address overfitting to hyperparameters.

    “Nested cross-validation is a variant of the k-fold cross-validation technique that tries to address the issue of overfitting to the hyperparameters (45).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] 45. Varma S , Simon R . Bias in error estimation when using cross-validation for model selection . BMC Bioinformatics 2006. ; 7 ( 1 ): 91 . [DOI] [PMC free article] [PubMed] [Google Scholar]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given
    § Conclusion

    Not all bias-prone practices in ML can be easily replaced or improved.

    “Finally, not all bias-prone practices in ML can be easily replaced or improved.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § Conclusion

    Social and ethical forms of bias can harm underrepresented groups or benefit those with more resources.

    “Social and ethical forms of bias can harm underrepresented groups or benefit those with more resources.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § Introduction

    Some imputation techniques, especially those using predictive ML, may amplify algorithmic bias, but imputation generally improves model performance while maintaining fairness.

    “While some imputation techniques (especially those that use predictive ML) may amplify algorithmic bias in the data, they generally improve the model’s performance while maintaining its fairness (50).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] 50. Martínez-Plumed F , Ferri C , Nieves D , Hernández-Orallo J . Missing the missing values: The ugly duckling of fairness in machine learning . Int J Intell Syst 2021. ; 36 ( 7 ): 3217 – 3258 . [Google Scholar]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § Introduction

    Standardization is more effective when feature values have an almost Gaussian distribution and the ML algorithm assumes normal distribution, whereas normalization does not have this limitation.

    “For example, standardization is more effective if the feature values have an almost Gaussian distribution, and the ML algorithm assumes the distribution of data to be normal. Normalization, on the other hand, does not have this limitation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § Introduction

    Normalization is more appropriate for algorithms like k-nearest neighbors and artificial neural networks, while standardization works better for regression algorithms.

    “Therefore, while normalization is more appropriate for algorithms like k-nearest neighbors and artificial neural networks, standardization works better for regression algorithms.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § Introduction

    The quality of an exploratory data analysis depends on the ability of developers to ask the right questions and detect patterns based on cognitive ability, domain expertise, and experience in data interpretation.

    “In fact, the quality of an EDA depends on the ability of the developers to ask the right questions and detect patterns within the data based on their cognitive ability, domain expertise, and experience in data interpretation (35).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] 35. Grolemund G , Wickham H . A Cognitive Interpretation of Data Analysis . Int Stat Rev 2014. ; 82 ( 2 ): 184 – 204 . [Google Scholar]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Introduction

    Most principles of data handling for computer vision machine learning are applicable to other machine learning applications such as natural language processing or tabular data analysis.

    “however, most of the principles are applicable to other ML applications such as natural language processing or tabular data analysis.”
    Quote in the paperNot a substantial claim
32 of 32

  • The paper relies on this without support.

    Medical datasets are substantially smaller than most nonmedical datasets.

    “the size of medical datasets is substantially smaller than most nonmedical datasets (eg, ImageNet)”

  • The paper relies on this without support.

    Available medical data may have imbalanced distribution based on several variables.

    “even the available data may have imbalanced distribution based on several variables”

  • The paper relies on this without support.

    The causes of imbalance in medical datasets go beyond machine learning and may pertain to population-level frequencies or patient recruitment protocols.

    “The causes of this imbalance go beyond ML and may pertain to population-level frequencies or patient recruitment protocols.”

  • The paper relies on this without support.

    All real-world data are likely to have missing data.

    “All real-world data are likely to have missing data.”

  • The paper relies on this without support.

    ML models do not necessarily learn as humans do.

    “Another problem with that assumption is that ML models do not necessarily learn as humans do.”

  • The paper relies on this without support.

    What is considered noise to developers may be a source of valuable signal data for models.

    “What is considered noise to developers may be a source of valuable signal data for models.”
6 of 6

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Krishna R, 2010, J Clin Diagn Res, 4, 2320
  2. Ohno-Machado L, 1998, Proc AMIA Symp, 523
  3. Seyyed-Kalantari L, 2021, Pac Symp Biocomput, 26, 232
  4. Pedregosa F, 2011, J Mach Learn Res, 12, 2825
  5. Cawley GC, 2010, J Mach Learn Res, 11, 2079
  6. Masoudi S, 2021, J Med Imaging (Bellingham), 8, 010901
  7. Gilbert DT, Fiske ST, Lindzey G. The Handbook of Social Psychology. Boston, Mass: McGraw-Hill, 1998.

Cited by

Sorted by citations

Related

Authors
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Vera Garcia, D. V., Singh, Y., Zhang, K., Conte, G. M., & Erickson, B. J.
Affiliations
  • Mayo Clinic (US)
  • From the Radiology Informatics Laboratory, Department of Radiology, Mayo Clinic, 200 1st St SW, Rochester, MN 55905.
ORCID
Venue
Radiology: Artificial Intelligence
Publisher
Radiological Society of North America (RSNA)
Published
2022-09-01
Volume
4
Issue
5
Pages
e210290-e210290
ISSN
2638-6100 (ISSN-L)
DOI
10.1148/ryai.210290
PMC
PMC9533091
OpenAlex
W4292958735
Semantic Scholar
caaf3277faea39df66b92d950b3a5b406f1d2310
Cited by
145
References
40
Citations by year
2026: 33 · 2025: 35 · 2024: 43 · 2023: 27 · 2022: 7
FWCI
8.43
Type
journal article
Language
en
Open access
Locations
  • Radiology Artificial Intelligence (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
algorithmic bias, bias mitigation, data handling, data collection, data partitioning, feature engineering, computer-aided diagnosis, convolutional neural networks, deep learning, performance evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review