CoChat
Sign Up
Cited by 18Gene expression and cancer classificationBreast Cancer Treatment StudiesBioinformatics and Genomic Networks

Assessing reproducibility and veracity across machine learning techniques in biomedicine: A case study using TCGA data

Open access
Kim, Ahyoung Amy, Rachid Zaim, Samir, Subbian, Vignesh·2020·International Journal of Medical Informatics
DOI 10.1016/j.ijmedinf.2020.104148
Open this paper in CoChat — free account

Abstract

Background Many studies that aim to identify gene biomarkers using statistical methods and translate them into FDA-approved drugs have faced challenges due to lack of clinical validity and methodological reproducibility. Since genomic data analysis relies heavily on these statistical learning tools more than before, it is vital to address the limitations of these computational techniques.Methods Our study demonstrates these methodological gaps among most common statistical learning techniques used in gene expression analysis. To assess the classification ability and reproducibility of statistical learning tools for gene biomarker detection, six state-of-the-art machine learning models were trained on four different cancer data retrieved from The Cancer Genome Atlas (TCGA). Standard performance metrics including specificity, sensitivity, precision, and F1 score were evaluated to investigate the classification ability. For analysis of reproducibility, the identifiability of gene classifiers was examined by quantifying the consistency of the chosen classifier genes.Results Among the six state-of-the-art machine learning methods, the random forest had the best classification ability overall. Very few genes were selected by multiple methods, which suggests poor identifiability and reproducibility of statistical learning methods for gene expression data. Our results demonstrated the challenges of reproducing discoveries from gene expression analysis due to the inherent differences that exist in statistical machine learning methods.Conclusion Since statistical machine learning models can have large variations in high-dimensional settings such as analysis of gene expression data, transparent analysis procedures including data preprocessing, model parameterization, and evaluation and choice of interpretable models are required for clinical validity and utility.

Integrity signals

Registry

Crossref · 10.1016/j.ijmedinf.2020.104148

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For some single omics, such as gene data, machine learning performance can be worse because of intrinsic properties.

    “However, ML may have deteriorated performance for certain single omics such as gene data due to inherent characteristics (Kim et al., 2020).”

    From Using machine learning approaches for multi-omics data analysis: A review (2021) · cites this paper as Kim et al., 2020

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Kim, A. A., Rachid Zaim, S., & Subbian, V.
Affiliations
University of Arizona (US)
Corresponding author
Kim, Ahyoung Amy
ORCID
Venue
International Journal of Medical Informatics
Publisher
Elsevier BV
Published
2020-09
Volume
141
Pages
104148
ISSN
1386-5056 (ISSN-L)
DOI
10.1016/j.ijmedinf.2020.104148
OpenAlex
W3025501838
Cited by
18
References
42
Citations by year
2026: 2 · 2025: 5 · 2024: 3 · 2023: 3 · 2022: 2 · 2021: 2 · 2020: 1
FWCI
0.65
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Medical Informatics (acceptedVersion)
  • PubMed (publishedVersion)
Funders
National Science Foundation
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Breast Cancer Treatment Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
reproducibility, genetic biomarkers, gene expression analysis, machine learning classifiers, random forest, cancer genomics, TCGA, clinical validity, high-dimensional data
MeSH terms
Machine Learning, Algorithms, Humans, Neoplasms, Reproducibility of Results, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review