CoChat

Experimenting with reproducibility: a case study of robustness in bioinformatics

Open access
Kim, Yang-Min, Poline, Jean-Baptiste, Dumas, Guillaume · 2018 · GigaScience

Published version of a preprint

Cited by 76

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Abstract

Reproducibility has been shown to be limited in many scientific fields. This question is a fundamental tenet of scientific activity, but the related issues of reusability of scientific data are poorly documented. Here, we present a case study of our difficulties in reproducing a published bioinformatics method even though code and data were available. First, we tried to re-run the analysis with the code and data provided by the authors. Second, we reimplemented the whole method in a Python package to avoid dependency on a MATLAB license and ease the execution of the code on a high-performance computing cluster. Third, we assessed reusability of our reimplementation and the quality of our documentation, testing how easy it would be to start from our implementation to reproduce the results. In a second section, we propose solutions from this case study and other observations to improve reproducibility and research efficiency at the individual and collective levels.While finalizing our code, we created case-specific documentation and tutorials for the associated Python package StratiPy. Readers are invited to experiment with our reproducibility case study by generating the two confusion matrices (see more in section "Robustness: from MATLAB to Python, language and organization"). Here, we propose two options: a step-by-step process to follow in a Jupyter/IPython notebook or a Docker container ready to be built and run.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/gigascience/giy077

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 19

  1. Hothorn, 2011, Case studies in reproducibility, Brief Bioinform, 12, 288, 10.1093/bib/bbq084 · DOI · Paper page
  2. Shapin, 2011, Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (New in Paper)
  3. Whitaker, 2017, Showing your working: a how to guide to reproducible research, Figshare
  4. Nekrutenko, 2012, Next-generation sequencing data interpretation: enhancing reproducibility and accessibility, Nat Rev Genet, 13, 667, 10.1038/nrg3305 · DOI · Paper page
  5. Hutson, 2018, Missing data hinder replication of artificial intelligence studies, Science
  6. Herndon, 2014, Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff, Camb J Econ, 38, 257, 10.1093/cje/bet075 · DOI · Paper page
  7. Hofree, 2013, Network-based stratification of tumor mutations, Nat Methods, 10, 1108, 10.1038/nmeth.2651 · DOI · Paper page
  8. G., 2017, Stratipy: Graph regularized nonnegative matrix factorization (GNMF) in Python, GHFC
  9. Glatard, 2015, Reproducibility of neuroimaging analyses across operating systems, Front Neuroinformatics, 9, 12, 10.3389/fninf.2015.00012 · DOI · Paper page
  10. TCGA, Cancer Genome Atlas - Natl, Cancer Inst
  11. Blischak, 2016, A quick introduction to version control with Git and GitHub, PLoS Comput Biol, 12, e1004668, 10.1371/journal.pcbi.1004668 · DOI · Paper page
  12. Buckheit, 1995, WaveLab and Reproducible Research. Wavelets Stat, 55, 10.1007/978-1-4612-2544-7_5 · DOI · Paper page
  13. Stodden, 2016, Enhancing reproducibility for computational methods, Science, 354, 1240, 10.1126/science.aah6168 · DOI · Paper page
  14. Hill, 2016, How do we know what we know? Discovering neuroscience data sets through minimal metadata, Nat Rev Neurosci, 17, 735, 10.1038/nrn.2016.134 · doi:10.1038/nrn.2016.134
  15. Eglen, 2017, Toward standard practices for sharing computer code and programs in neuroscience, Nat Neurosci, 20, 770, 10.1038/nn.4550 · DOI · Paper page
  16. Bourne, 2017, Ten simple rules to consider regarding preprint submission, PLoS Comput Biol, 13, e1005473, 10.1371/journal.pcbi.1005473 · DOI · Paper page
  17. Preprints in biology, 2016, Nat Methods, 13, 277, 10.1038/nmeth.3831 · doi:10.1038/nmeth.3831
  18. Day, 2011, Historical Perspectives, 6
  19. Yang-Min, 2017, StratiPy

Cited by

See all 76

Metadata

Authors
Kim, Y.-M., Poline, J.-B., & Dumas, G.
Venue
GigaScience
Published
2018-06-28
Volume
7
Issue
7
ISSN
2047-217X (ISSN-L)
OpenAlex
W2810419619
Cited by
76
References
11
FWCI
12.67
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.