CoChat
Sign Up
Cited by 61Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

Twelve key challenges in medical machine learning and solutions

Open access
Ellis, Randall J., Sander, Ryan M., Limon, Alfonso·2022·Intelligence-Based Medicine
DOI 10.1016/j.ibmed.2022.100068
Open this paper in CoChat — free accountOpen PDF

Abstract

The utility of machine learning in biomedicine is being investigated in various contexts, including for diagnostic and interpretive purposes for imaging modalities, quantifying disease risk, and processing text from physician and patient reports. To best facilitate the potential of machine learning, clinicians and computational scientists must inform one another about the nature of their clinical challenges and available methods for solving them, respectively. To this end, clinicians need to critically evaluate machine learning studies conducted to solve relevant problems in medicine. This article serves as a checklist for clinicians to understand and appraise machine learning studies and help facilitate productive conversations between the clinical and data science communities to improve human health.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ibmed.2022.100068

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Ongoing challenges in medical diagnostics with generative AI include reducing false positives and negatives.

    “Ongoing challenges include reducing false positives and negatives [21].”

    From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [21]

    Section 3.3 (page not available) treats false positives and false negatives as consequential errors in medical diagnostic classifiers and discusses the trade-off between reducing them. The paper addresses medical machine learning generally, however; it does not establish this as a challenge specific to generative AI diagnostics. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. Alexander Amini, et al. Uncovering and mitigating algorithmic bias through learned latent structure. In: Proceedings of the 2019 AAAI/ACM Conference on AI, ethics, and ¡society; 2019. p. 289-95.
  2. Beam Andrew L, Manrai Arjun K, Ghassemi Marzyeh. Challenges to the repro- ducibility of machine learning models in health care. In: Jama 323. vol. 4; 2020. p. 305-6.
  3. Bojar Ondrej, et al. Findings of the 2014 workshop on statistical machine trans- lation. In: Proceedings of the ninth workshop on statistical machine translation; 2014. p. 12-58.
  4. Breiman Leo. Random forests. Mach Learn 2001;45(1):5-32.
  5. Bzdok Danilo, Ioannidis John PA. Exploration, inference, and prediction in neuroscience and biomedicine. Trends Neurosci 2019;42(4):251-62.
  6. Chawla NV, et al. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321-57. https://doi.org/10.1613/jair.953. ISSN: 1076-9757. · DOI · Paper page
  7. Deng Li. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process Mag 2012;29(6):141-2.
  8. Dietterich Thomas G. Approximate statistical tests for comparing supervised clas- sification learning algorithms. In: Neural computation. vol. 10; 1998. p. 1895-923. 7, DOI: 1 0.1162/089976698300017197. · DOI · Paper page
  9. Dietterich Tom. Overfitting and undercomputing in machine learning. In: ACM computing surveys (CSUR). vol. 27; 1995. p. 326-7. 3.
  10. Fan Feng-Lei, et al. On interpretability of artificial neural networks: a survey. In: IEEE transactions on radiation and plasma medical. vol. 5. Sciences; 2021. p. 741-60. 6, DOI: 10 . 1109/TRPMS . 2021.3066428. · DOI
  11. Galdran Adrian, Carneiro Gustavo, Ballester Miguel A González. Balanced-MixUp for highly imbalanced medical image classification. In: International Conference on medical image Computing and computer-assisted intervention. Springer; 2021. p. 323-33.
  12. Goncalves Andre, et al. Generation and evaluation of synthetic patient data. BMC Med Res Methodol 2020;20(1):1-40.
  13. Guyon Isabelle, et al. What size test set gives good error rate estimates?. In: IEEE transactions on pattern analysis and machine intelligence. vol. 20; 1998. p. 52-64. 1.
  14. Heller Nicholas, et al. In: The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes; 2019, 00445. arXiv preprint arXiv:1904.
  15. Arthur Jacot, Gabriel Franck, Hongler Clément. Neural tangent kernel: conver- gence and generalization in neural networks. In: Proceedings of the 32nd inter- national conference on neural information processing systems; 2018. p. 8580-9.
  16. Kidwell Mallory C, et al. Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLoS Biol 2016;14(5):e1002456.
  17. Larrazabal Agostina J, et al. Gender imbalance in medical imaging datasets pro- duces biased classifiers for computer-aided diagnosis. In: Proceedings of the na- tional academy of,Sciences. vol. 117; 2020. p. 12592-4. 23.
  18. Lin Tsung-Yi, et al. Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision; 2017. p. 2980-8.
  19. Lin Tsung-Yi, et al. Microsoft coco: common objects in context. In: European conference on computer vision. Springer; 2014. p. 740-55.
  20. Merkel Dirk. Docker: lightweight linux containers for consistent development and deployment. Linux J 2014 2014;239:2.
  21. Nowok Beata, Raab Gillian M, Dibben Chris. synthpop: bespoke creation of syn- thetic data in R. J Statist Software 2016;74(1):1-26.
  22. Panch Trishan, et al. Yes, but will it work for my pa-tients? Driving clinically relevant research with benchmark datasets. NPJ Digital Med 2020;3(1):1-4.
  23. Rahman M Mostafizur, Davis Darryl N. Addressing the class imbalance problem in medical datasets. Int J Mach Learn Comput 2013;3(2):224.
  24. Rajkomar Alvin, Dean Jeffrey, Kohane Isaac. Machine learning in medicine. New England J Med 2019;380(14):1347-58.
  25. Rezaei Mina, et al. Generative synthetic adversarial network for internal bias correction and handling class imbalance problem in medical image diagnosis. In: Medical imaging 2020: computer-aided diagnosis. vol. 11314. International Soci- ety for Optics and Photonics; 2020, 113140E.
  26. Roberts Daniel A, Yaida Sho, Hanin Boris. In: The principles of deep learning theory; 2021. arXiv preprint arXiv:2106.10165. · arXiv
  27. Sambasivan Nithya, et al. Everyone wants to do the model work, not the data : data Cascades in High-Stakes AI. In: Proceedings of the 2021 CHI conference on human factors in computing systems; 2021. p. 1-15.
  28. Ramprasaath R. Selvaraju et al. "Grad-CAM: visual Explanations from Deep Net- works via Gradient-Based Localization. Int J Comput Vis Oct. 2019;128(2):336-59. https://doi.org/10.1007/s11263-019-01228-7. 1573-1405. · DOI · Paper page
  29. Selvaraju Ramprasaath R, et al. Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international confer- ence on computer vision; 2017. p. 618-26.
  30. Sokolova Marina, Japkowicz Nathalie, Szpakowicz Stan. Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. In: Aus- tralasian joint conference on artificial intelligence. Springer; 2006. p. 1015-21.
  31. Strobl Carolin, et al. Conditional variable importance for random forests. BMC Bioinf 2008;9(1):1-11.
  32. Wang Xiaosong, et al. Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common tho- rax diseases. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017. p. 2097-106.
  33. Wu Eric, et al. How medical AI devices are evaluated: limitations and recom- mendations from an analysis of FDA approvals. Nat Med 2021;27(4):582-4.
  34. Yang Greg, Edward J, Hu. Tensor programs IV: feature learning in infinite-width neural networks. In: Proceedings of the 38th international conference on machine learning. Marina meila and tong zhang. Vol. 139. Proceedings of machine learning research. PMLR; July 2021. p. 11727-37.
  35. Zhang BH, Lemoine B, Mitchell Margaret. Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM conference on AI. Ethics, and Society; 2018.
  36. Seo, Arık, Yoon, Zhang, Sohn, Pfister. NeurIPS 2021.

Cited by

Sorted by citations

Authors
Ellis, R. J., Sander, R. M., & Limon, A.
Affiliations
  • Icahn School of Medicine at Mount Sinai (US)
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
Corresponding author
  • Ellis, Randall J.
  • Sander, Ryan M.
  • Limon, Alfonso
ORCID
Venue
Intelligence-Based Medicine
Publisher
Elsevier BV
Published
2022
Volume
6
Pages
100068
ISSN
2666-5212 (ISSN-L)
DOI
10.1016/j.ibmed.2022.100068
OpenAlex
W4283528092
Cited by
61
References
6
Citations by year
2026: 9 · 2025: 20 · 2024: 18 · 2023: 13 · 2022: 1
FWCI
3.91
Type
journal article
Language
en
Open access
Licence
Locations
Intelligence-Based Medicine (publishedVersion, cc-by-nc-nd)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
machine learning in medicine, medical image interpretation, clinical text processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review