One Algorithm May Not Fit All: How Selection Bias Affects Machine Learning Performance
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Machine learning (ML) algorithms have demonstrated high diagnostic accuracy in identifying and categorizing disease on radiologic images. Despite the results of initial research studies that report ML algorithm diagnostic accuracy similar to or exceeding that of radiologists, the results are less impressive when the algorithms are installed at new hospitals and are presented with new images. This phenomenon is potentially the result of selection bias in the data that were used to develop the ML algorithm. Selection bias has long been described by clinical epidemiologists as a key consideration when designing a clinical research study, but this concept has largely been unaddressed in the medical imaging ML literature. The authors discuss the importance of selection bias and its relevance to ML algorithm development to prepare the radiologist to critically evaluate ML literature for potential selection bias and understand how it might affect the applicability of ML algorithms in real clinical environments. ©RSNA, 2020
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Kim, D. W., Jang, H. Y., Kim, K. W., Shin, Y., & Park, S. H. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0025 · Cited by 459
Bluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Biases in Randomized Trials: A Conversation Between Trialists and Epidemiologists
Mansournia, M. A., Higgins, J. P. T., Sterne, J. A. C., & Hernán, M. A. (2017) · Epidemiology · doi:10.1097/ede.0000000000000564 · Cited by 285
Deep learning in chest radiography: Detection of findings and presence of change
Singh, R., Kalra, M. K., Nitiwarangkul, C., Patti, J. A., Homayounieh, F., Padole, A., Rao, P., Putha, P., Muse, V. V., Sharma, A., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204155 · Cited by 195
Cited by
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 408
Self-citationThe Sociodemographic Biases in Machine Learning Algorithms: A Biomedical Informatics Perspective
Franklin, G., Stephens, R., Piracha, M., Tiosano, S., Lehouillier, F., Koppel, R., & Elkin, P. (2024) · Life · doi:10.3390/life14060652 · Cited by 76
“Review and keep in mind basic clinical epidemiological principles (e.g., selection bias , which affects “real-world” study interpretation and generalizability to a representative population) [13,27].”
BackgroundTranslating AI to Clinical Practice: Overcoming Data Shift with Explainability
Choi, Y., Yu, W., Nagarajan, M. B., Teng, P., Goldin, J. G., Raman, S. S., Enzmann, D. R., Kim, G. H. J., & Brown, M. S. (2023) · RadioGraphics · doi:10.1148/rg.220105 · Cited by 65
US Regulatory Considerations for Low Field Magnetic Resonance Imaging Systems
Krainak, D. M., Zeng, R., Li, N., Woods, T. O., & Delfino, J. G. (2023) · Magnetic Resonance Materials in Physics, Biology and Medicine · doi:10.1007/s10334-023-01083-1 · Cited by 5
Crafting Disability Fairness Learning in Data Science: A Student-Centric Pedagogical Approach
Newman, P., Opdahl, T., Liu, Y., Wehrwein, S., & Elglaly, Y. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630815 · Cited by 3
Metadata
- Authors
- Yu, A. C., & Eng, J.
- Venue
- RadioGraphics
- Published
- 2020-11
- Volume
- 40
- Issue
- 7
- Pages
- 1932-1937
- ISSN
- 0271-5333 (ISSN-L)
- 1527-1323
- OpenAlex
- W3088943327
- Cited by
- 59
- References
- 7
- FWCI
- 1.55
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.