Experimenting with reproducibility: a case study of robustness in bioinformatics
Open accessPublished version of a preprint
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Reproducibility has been shown to be limited in many scientific fields. This question is a fundamental tenet of scientific activity, but the related issues of reusability of scientific data are poorly documented. Here, we present a case study of our difficulties in reproducing a published bioinformatics method even though code and data were available. First, we tried to re-run the analysis with the code and data provided by the authors. Second, we reimplemented the whole method in a Python package to avoid dependency on a MATLAB license and ease the execution of the code on a high-performance computing cluster. Third, we assessed reusability of our reimplementation and the quality of our documentation, testing how easy it would be to start from our implementation to reproduce the results. In a second section, we propose solutions from this case study and other observations to improve reproducibility and research efficiency at the individual and collective levels.While finalizing our code, we created case-specific documentation and tutorials for the associated Python package StratiPy. Readers are invited to experiment with our reproducibility case study by generating the two confusion matrices (see more in section "Robustness: from MATLAB to Python, language and organization"). Here, we propose two options: a step-by-step process to follow in a Jupyter/IPython notebook or a Docker container ready to be built and run.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Singularity: Scientific containers for mobility of compute
Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177459 · Cited by 2,740 · cited 1×
“Con-tainer and virtual machine technologies such as Docker [30], Vagrant [31], and Singularity [32, 33] (easily works in cluster environments) are becoming a standard solution to mitigate installation issues.”
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 1×
“Several concepts for reproducibility in computational science are closely associated [5, 6].”
An introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059
From the genetic architecture to synaptic plasticity in autism spectrum disorder
Bourgeron, T. (2015) · Nature Reviews Neuroscience · doi:10.1038/nrn3992 · Cited by 975 · cited 1×
“One of the current challenges of our research is to uncover homogeneous subgroups of patients (i.e., stratification) with more precise clinical outcomes, improving their prognosis and treat-ment [10, 11].”
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · cited 1×
“…by suggesting tools and best practices following the programming best practices of Wilson et al., such as modularizing and re-using code, unit testing, document design, data management, and project organization [2, 27], as well as keeping data provenance and recording all intermediate results [28].”
Best Practices for Scientific Computing
Wilson, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., et al. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001745 · Cited by 723 · cited 1×
“…by suggesting tools and best practices following the programming best practices of Wilson et al., such as modularizing and re-using code, unit testing, document design, data management, and project organization [2, 27], as well as keeping data provenance and recording all intermediate results [28].”
Foster, E. D., & Deardorff, A. (2017) · Journal of the Medical Library Association · doi:10.5195/jmla.2017.88 · Cited by 715 · cited 1×
“The Open Science Framework also hosts data and software for a given project [49].”
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
Identification and validation of biomarkers for autism spectrum disorders
Loth, E., Spooren, W., Ham, L. M., Isaac, M. B., Auriche-Benichou, C., Banaschewski, T., Baron-Cohen, S., Broich, K., Bölte, S., Bourgeron, T., et al. (2016) · Nature Reviews Drug Discovery · doi:10.1038/nrd.2015.7 · Cited by 140 · cited 1×
“One of the current challenges of our research is to uncover homogeneous subgroups of patients (i.e., stratification) with more precise clinical outcomes, improving their prognosis and treat-ment [10, 11].”
Research works printed but not linked: 19
- Hothorn, 2011, Case studies in reproducibility, Brief Bioinform, 12, 288, 10.1093/bib/bbq084 · DOI · Paper page
- Shapin, 2011, Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (New in Paper)
- Whitaker, 2017, Showing your working: a how to guide to reproducible research, Figshare
- Nekrutenko, 2012, Next-generation sequencing data interpretation: enhancing reproducibility and accessibility, Nat Rev Genet, 13, 667, 10.1038/nrg3305 · DOI · Paper page
- Hutson, 2018, Missing data hinder replication of artificial intelligence studies, Science
- Herndon, 2014, Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff, Camb J Econ, 38, 257, 10.1093/cje/bet075 · DOI · Paper page
- Hofree, 2013, Network-based stratification of tumor mutations, Nat Methods, 10, 1108, 10.1038/nmeth.2651 · DOI · Paper page
- G., 2017, Stratipy: Graph regularized nonnegative matrix factorization (GNMF) in Python, GHFC
- Glatard, 2015, Reproducibility of neuroimaging analyses across operating systems, Front Neuroinformatics, 9, 12, 10.3389/fninf.2015.00012 · DOI · Paper page
- TCGA, Cancer Genome Atlas - Natl, Cancer Inst
- Blischak, 2016, A quick introduction to version control with Git and GitHub, PLoS Comput Biol, 12, e1004668, 10.1371/journal.pcbi.1004668 · DOI · Paper page
- Buckheit, 1995, WaveLab and Reproducible Research. Wavelets Stat, 55, 10.1007/978-1-4612-2544-7_5 · DOI · Paper page
- Stodden, 2016, Enhancing reproducibility for computational methods, Science, 354, 1240, 10.1126/science.aah6168 · DOI · Paper page
- Hill, 2016, How do we know what we know? Discovering neuroscience data sets through minimal metadata, Nat Rev Neurosci, 17, 735, 10.1038/nrn.2016.134 · doi:10.1038/nrn.2016.134
- Eglen, 2017, Toward standard practices for sharing computer code and programs in neuroscience, Nat Neurosci, 20, 770, 10.1038/nn.4550 · DOI · Paper page
- Bourne, 2017, Ten simple rules to consider regarding preprint submission, PLoS Comput Biol, 13, e1005473, 10.1371/journal.pcbi.1005473 · DOI · Paper page
- Preprints in biology, 2016, Nat Methods, 13, 277, 10.1038/nmeth.3831 · doi:10.1038/nmeth.3831
- Day, 2011, Historical Perspectives, 6
- Yang-Min, 2017, StratiPy
Cited by
Zhou, G., Soufan, O., Ewald, J., Hancock, R. E. W., Basu, N., & Xia, J. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz240 · Cited by 1,899
RESCRIPt: Reproducible sequence taxonomy reference database management
(2021) · doi:10.1371/journal.pcbi.1009581 · Cited by 983
Towards reproducible computational drug discovery
Schaduangrat, N., Lampa, S., Simeon, S., Gleeson, M. P., Spjuth, O., & Nantasenamat, C. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-0408-x · Cited by 208
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131
“Several suggestions have been published to improve reproducibility in bioinformatics (Kim et al , 2018).”
Lefebvre, A., Delorme, R., Delanoë, C., Amsellem, F., Beggiato, A., Germanaud, D., Bourgeron, T., Toro, R., & Dumas, G. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00662 · Cited by 58
Self-citation
Metadata
- Authors
- Kim, Y.-M., Poline, J.-B., & Dumas, G.
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-06-28
- Volume
- 7
- Issue
- 7
- ISSN
- 2047-217X (ISSN-L)
- OpenAlex
- W2810419619
- Cited by
- 76
- References
- 11
- FWCI
- 12.67
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Experimenting with reproducibility: a case study of robustness in bioinformatics. https://app.cochat.ai/papers/10.1093/gigascience/giy077
- The paper
- Kim, Y.-M., Poline, J.-B., & Dumas, G. (2018). Experimenting with reproducibility: a case study of robustness in bioinformatics. GigaScience. https://doi.org/10.1093/gigascience/giy077
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.