CoChat
Sign Up
Cited by 42AI in cancer detectionRadiomics and Machine Learning in Medical ImagingArtificial Intelligence in Healthcare

Hazards of data leakage in machine learning: a study on classification of breast cancer using deep neural networks

Samala, Ravi K., Chan, Heang-Ping, Hadjiiski, Lubomir, Koneru, Sathvik·2020·Medical Imaging 2020: Computer-Aided Diagnosis
DOI 10.1117/12.2549313
Open this paper in CoChat — free account

Abstract

With the renewed interest in developing machine learning methods for medical imaging using deep-learning approaches, it is essential to reexamine data leakage. In this study, we simulated data leakage in the form of feature leakage, where a classifier was trained on the training set, but the feature selection was influenced by the performance on the validation set. A pre-trained deep-learning convolutional neural network (DCNN) without fine-tuning was used as a feature extractor for malignant and benign mass classification in mammography. A feature selection algorithm was trained in the wrapper mode with a cost function tuned to follow the performance metric on the validation set. Linear discriminant analysis (LDA) classifier was trained to classify masses on mammographic patches. Mammograms from 1,882 patient cases with 4,577 unique patches were partitioned by patient into 3,222 for training and 508 for validation, while 847 were sequestered as unseen independent test set to evaluate the generalization error. The effects of the finite sample size on data leakage were studied by varying the training and validation set sizes from 10% to 100% of the available sets. The area under the receiver operating characteristic curve (AUC) was used as the performance metric. The results show that the performance on the validation set could be overestimated, having AUCs of 0.75 to 0.99 for various sample sizes, whereas the independent test performance could realistically only reach an AUC of 0.72. The analysis indicates that deep learning can risk a high inflation in performance and proper housekeeping rules should be followed when designing and developing deep learning methods in medical imaging.

Integrity signals

Registry

Crossref · 10.1117/12.2549313

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In the context of the ATG-based ML model, the inclusion of gene ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) terms can introduce information directly related to the research question.

    “In the context of the ATG-based ML model, the inclusion of gene ontology (GO) (64) and Kyoto Encyclopedia of Genes and Genomes (KEGG) (75) terms can introduce information directly related to the research question.”

    From Artificial Intelligence for Drug Discovery: Are We There Yet? (2024) · cites this paper as [75]

  • The nonstationary nature of policy data introduces strong temporal dependencies that should be considered throughout the modeling pipeline and makes these models susceptible to errors such as data leakage.

    “This nonstationary nature of the data introduces strong temporal dependencies that should be considered throughout the modeling pipeline and makes these models susceptible to errors such as data leakage (Kaufman et al., 2011; Samala et al., 2020).”

    From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Samala et al., 2020

Cited by

Sorted by citations

Related

Authors
Samala, R. K., Chan, H.-P., Hadjiiski, L., & Koneru, S.
Affiliations
University of Michigan (US)
ORCID
Venue
Medical Imaging 2020: Computer-Aided Diagnosis
Publisher
SPIE
Published
2020-03-16
Pages
39
DOI
10.1117/12.2549313
OpenAlex
W3011190810
Cited by
42
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 18 · 2023: 7 · 2022: 5 · 2021: 5 · 2020: 1
FWCI
1.72
Type
conference paper
Language
en
Open access
closed
Locations
SPIE Medical Imaging: Computer-Aided Diagnosis (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
data leakage, feature leakage, breast cancer classification, mammography, convolutional neural networks, feature selection, sample size effects, linear discriminant analysis, wrapper-based feature selection, diagnostic imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review