Where next for the reproducibility agenda in computational biology?
Open accessAbstract
BACKGROUND: The concept of reproducibility is a foundation of the scientific method. With the arrival of fast and powerful computers over the last few decades, there has been an explosion of results based on complex computational analyses and simulations. The reproducibility of these results has been addressed mainly in terms of exact replicability or numerical equivalence, ignoring the wider issue of the reproducibility of conclusions through equivalent, extended or alternative methods. RESULTS: We use case studies from our own research experience to illustrate how concepts of reproducibility might be applied in computational biology. Several fields have developed 'minimum information' checklists to support the full reporting of computational simulations, analyses and results, and standardised data formats and model description languages can facilitate the use of multiple systems to address the same research question. We note the importance of defining the key features of a result to be reproduced, and the expected agreement between original and subsequent results. Dynamic, updatable tools for publishing methods and results are becoming increasingly common, but sometimes come at the cost of clear communication. In general, the reproducibility of computational research is improving but would benefit from additional resources and incentives. CONCLUSIONS: We conclude with a series of linked recommendations for improving reproducibility in computational biology through communication, policy, education and research practice. More reproducible research will lead to higher quality conclusions, deeper understanding and more valuable knowledge.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bioinformatics sets the bar of being able to generate every figure in an article from a single script.
“This is the bar set by bioinformatics,32”
From Reproducibility of Quantitative Systems Pharmacology Models: Current Challenges and Future Opportunities (2019) · cites this paper as [32]
In Results > Software tools for bioinformatics (page unavailable), the paper describes publishing code and data needed to rerun an analysis and generate figures as an aspirational target for replicable bioinformatics research. It does not specify generating every figure from a single script or present that as a bar established across bioinformatics. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190 · see citing works
Ultrafast and memory-efficient alignment of short DNA sequences to the human genome.
Langmead, B., Trapnell, C., Pop, M., & Salzberg, S. L. (2009) · Genome Biology · doi:10.1186/gb-2009-10-3-r25 · Cited by 21,849 · see citing works
Bioconductor: open software development for computational biology and bioinformatics
Gentleman, R., Carey, V., Bates, D., Bolstad, B., Dettling, M., Dudoit, S., Ellis, B., Gautier, L., Ge, Y., Gentry, J., et al. (2004) · Genome Biology · doi:10.1186/gb-2004-5-10-r80 · Cited by 13,133 · see citing works
BEAST 2: a software platform for Bayesian evolutionary analysis
Bouckaert, R., Heled, J., Kühnert, D., Vaughan, T., Wu, C.-H., Xie, D., Suchard, M. A., Rambaut, A., & Drummond, A. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003537 · Cited by 6,527 · see citing works
IPython: A System for Interactive Scientific Computing
Pérez, F., & Granger, B. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.53 · Cited by 3,927 · see citing works
Goecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,437 · see citing works
Docker: lightweight Linux containers for consistent development and deployment
Merkel, D. (2014) · Linux journal · Cited by 3,277 · see citing works
Hucka, M., Finney, A., Sauro, H. M., Bolouri, H., Doyle, J. C., Kitano, H., Arkin, A. P., Bornstein, B. J., Bray, D., Cornish-Bowden, A., et al. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg015 · Cited by 3,060 · see citing works
Galaxy: a platform for interactive large-scale genome analysis.Preprint
Giardine, B., Riemer, C., Hardison, R., Burhans, R. C., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,217 · see citing works
Knuth, D. (1984) · Computer/law journal · doi:10.1093/comjnl/27.2.97 · Cited by 1,863 · see citing works
A comparative encyclopedia of DNA elements in the mouse genome
Yue, F., Cheng, Y., Breschi, A., Vierstra, J., Wu, W., Ryba, T., Sandstrom, R., Ma, Z., Davis, C., Pope, B. D., et al. (2014) · Nature · doi:10.1038/nature13992 · Cited by 1,781 · see citing works
Riemann manifold langevin and hamiltonian monte carlo methods
Girolami, M., & Calderhead, B. (2011) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2010.00765.x · Cited by 1,473 · see citing works
Galaxy: A Web‐Based Genome Analysis Tool for Experimentalists
Blankenberg, D. J., Kuster, G. V., Coraor, N., Ananda, G., Lazarus, R. M., Mangan, M. E., Nekrutenko, A., & Taylor, J. (2010) · Current Protocols in Molecular Biology · doi:10.1002/0471142727.mb1910s89 · Cited by 1,430 · see citing works
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological bulletin · doi:10.1037/h0026141 · Cited by 1,363 · see citing works
An introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059 · see citing works
Keeling, P. J., Burki, F., Wilcox, H. M., Allam, B., Allen, E. E., Amaral-Zettler, L. A., Armbrust, E. V., Archibald, J. M., Bharti, A. K., Bell, C. J., et al. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001889 · Cited by 1,042 · see citing works
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · see citing works
The BioPAX community standard for pathway data sharing
Demir, E., Cary, M. P., Paley, S., Fukuda, K., Lemer, C., Vastrik, I., Wu, G., D'Eustachio, P., Schaefer, C., Luciano, J., et al. (2010) · Nature Biotechnology · doi:10.1038/nbt.1666 · Cited by 745 · cited 1× · see citing works
“The wide variety of data formats used in bio-informatics software can also make replicability hard to achieve, and standards for data exchange and analysis such as the PSI-MI specification for proteomics [12, 13] and the BioPAX language for pathway data [14, 15] have been developed to improve this.”
BackgroundBest Practices for Scientific Computing
Wilson, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., et al. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001745 · Cited by 723 · see citing works
Hermjakob, H., Montecchi-Palazzi, L., Bader, G., Wojcik, J., Salwinski, L., Ceol, A., Moore, S., Orchard, S., Sarkans, U., von Mering, C., et al. (2004) · Nature Biotechnology · doi:10.1038/nbt926 · Cited by 630 · see citing works
Minimum information requested in the annotation of biochemical models (MIRIAM)
Novère, N. L., Finney, A., Hucka, M., Bhalla, U. S., Campagne, F., Collado-Vides, J., Crampin, E. J., Halstead, M., Klipp, E., Mendes, P., et al. (2005) · Nature Biotechnology · doi:10.1038/nbt1156 · Cited by 622 · see citing works
Taylor, C. F., Field, D., Sansone, S.-A., Aerts, J., Apweiler, R., Ashburner, M., Ball, C. A., Binz, P.-A., Bogue, M., Booth, T., et al. (2008) · Nature Biotechnology · doi:10.1038/nbt.1411 · Cited by 558 · cited 1× · see citing works
“Encouragingly, “ minimum information ” checklists are being developed, and brought together under the um-brella of the Minimum Information for Biological and Biomedical Investigations (MIBBI) project [11], to help researchers ensure they provide all relevant context for their data.”
BackgroundStatistical inference for noisy nonlinear ecological dynamic systems
Wood, S. N. (2010) · Nature · doi:10.1038/nature09319 · Cited by 542 · see citing works
Sweave: Dynamic Generation of Statistical Reports Using Literate Data Analysis
Leisch, F. (2002) · International Conference on Computational Statistics · doi:10.1007/978-3-642-57489-4_89 · Cited by 523 · see citing works
Chaste: An Open Source C++ Library for Computational Physiology and Biology
Mirams, G. R., Arthurs, C. J., Bernabéu, M. O., Bordas, R., Cooper, J., Corrias, A., Davit, Y., Dunn, S.-J., Fletcher, A. G., Harvey, D. G., et al. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002970 · Cited by 424 · see citing works
Self-citationComparison of the transcriptional landscapes between human and mouse tissues
Lin, S., Lin, Y., Nery, J. R., Urich, M. A., Breschi, A., Davis, C., Dobin, A., Zaleski, C., Beer, M., Chapman, W. C., et al. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1413624111 · Cited by 408 · see citing works
An introduction to Docker for reproducible research, with examples from the R environmentPreprint
Boettiger, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0846 · Cited by 372 · see citing works
knitr: A General-Purpose Package for Dynamic Report Generation in R
Xie, Y. (2012) · doi:10.32614/cran.package.knitr · Cited by 285 · see citing works
Golightly, A., & Wilkinson, D. J. (2011) · Interface Focus · doi:10.1098/rsfs.2011.0047 · Cited by 282 · see citing works
Reproducible Research in Computational Harmonic Analysis
Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2008) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274 · see citing works
Waltemath, D., Adams, R., Bergmann, F. T., Hucka, M., Kolpakov, F., Miller, A. K., Moraru, I., Nickerson, D., Sahle, S., Snoep, J., et al. (2011) · BMC Systems Biology · doi:10.1186/1752-0509-5-198 · Cited by 269 · see citing works
A Reference-Free Algorithm for Computational Normalization of Shotgun Sequencing DataPreprint
Brown, C. T., Howe, A., Zhang, Q., Pyrkosz, A. B., & Brom, T. H. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1203.4802 · Cited by 233 · see citing works
Chaste: A test-driven approach to software development for biological modelling
Pitt‐Francis, J., Pathmanathan, P., Bernabéu, M. O., Bordas, R., Cooper, J., Fletcher, A. G., Mirams, G. R., Murray, P. J., Osborne, J. M., Walter, A., et al. (2009) · Computer Physics Communications · doi:10.1016/j.cpc.2009.07.019 · Cited by 233 · see citing works
Self-citationReplicability is not Reproducibility:Nor is it Good Science
Drummond, C. (2009) · Cited by 229 · see citing works
Interactive notebooks: Sharing the code
Shen, H. (2014) · Nature · doi:10.1038/515151a · Cited by 225 · see citing works
Modeling Prevention Strategies for Gonorrhea and Chlamydia Using Stochastic Network Simulations
Kretzschmar, M., van Duynhoven, Y. T. H. P., & Severijnen, A. J. (1996) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a008926 · Cited by 207 · see citing works
A Quick Introduction to Version Control with Git and GitHub
Blischak, J., Davenport, E., & Wilson, G. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004668 · Cited by 188 · see citing works
Minimum Information About a Simulation Experiment (MIASE)
Waltemath, D., Adams, R., Beard, D. A., Bergmann, F. T., Bhalla, U. S., Britten, R., Chelliah, V., Cooling, M. T., Cooper, J., Crampin, E. J., et al. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001122 · Cited by 168 · see citing works
Self-citationStodden, V., & Miguez, S. (2014) · Journal of Open Research Software · doi:10.5334/jors.ay · Cited by 163 · see citing works
Software Carpentry: Getting Scientists to Write Better Code by Making Them More Productive
Wilson, G. (2006) · Computing in Science & Engineering · doi:10.1109/mcse.2006.122 · Cited by 147 · see citing works
Hucka, M., Nickerson, D. P., Bader, G. D., Bergmann, F., Cooper, J., Demir, E., Garny, A., Golebiewski, M., Myers, C. J., Schreiber, F., et al. (2015) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2015.00019 · Cited by 118 · see citing works
Self-citationData Carpentry: Workshops to Increase Data Literacy for Researchers
Teal, T. K., Cranston, K. A., Lapp, H., White, E., Wilson, G., Ram, K., & Pawlik, A. (2015) · International Journal of Digital Curation · doi:10.2218/ijdc.v10i1.351 · Cited by 106 · see citing works
Molecules for memory: modelling CaMKII
Stefan, M. I., & Novère, N. L. (2007) · BMC Systems Biology · doi:10.1186/1752-0509-1-s1-p40 · Cited by 100 · see citing works
A reanalysis of mouse ENCODE comparative gene expression data
Gilad, Y., & Mizrahi-Man, O. (2015) · F1000Research · doi:10.12688/f1000research.6536.1 · Cited by 100 · see citing works
Ten Simple Rules for a Successful Cross-Disciplinary Collaboration
Knapp, B., Bardenet, R., Bernabéu, M. O., Bordas, R., Bruna, M., Calderhead, B., Cooper, J., Fletcher, A. G., Groen, D., Kuijper, B., et al. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004214 · Cited by 92 · see citing works
Self-citationMALONE, J. M., Brown, A. J., Lister, A., Ison, J., Hull, D., Parkinson, H., & Stevens, R. (2014) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-5-25 · Cited by 90 · see citing works
COMPSTAT 1974 : proceedings in computational statistics
Bruckmann, G., Ferschl, F., & Schmetterer, L. F. (1974) · Physica-Verl. eBooks · Cited by 86 · see citing works
Virtual Appliances, Cloud Computing, and Reproducible Research
Howe, B. (2012) · Computing in Science & Engineering · doi:10.1109/mcse.2012.62 · Cited by 83 · see citing works
Althaus, C. L., Turner, K., Schmid, B. V., Heijne, J. C. M., Kretzschmar, M., & Low, N. N. (2011) · Journal of The Royal Society Interface · doi:10.1098/rsif.2011.0131 · Cited by 77 · see citing works
Validation in Genetic Association Studies
König, I. R. (2011) · Briefings in Bioinformatics · doi:10.1093/bib/bbq074 · Cited by 76 · see citing works
On Some Principles of Statistical Inference
Reid, N. M., & Cox, D. R. (2014) · International Statistical Review · doi:10.1111/insr.12067 · Cited by 67 · see citing works
A Stronger Post-Publication Culture Is Needed for Better Science
Bastian, H. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001772 · Cited by 60 · see citing works
Gardiner, B. S., Wong, K. K. L., Joldes, G. R., et al. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004544 · Cited by 58 · see citing works
Ten Simple Rules for Effective Computational Research
Osborne, J. M., Bernabéu, M. O., Bruna, M., et al. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003506 · Cited by 56 · see citing works
Is Replication the Gold Standard for Validating Genome-Wide Association Findings?
Liu, Y.-J., Papasian, C. J., Liu, J., Hamilton, J. J., & Deng, H. (2008) · PLoS ONE · doi:10.1371/journal.pone.0004037 · Cited by 51 · see citing works
High-throughput functional curation of cellular electrophysiology models
Cooper, J., Mirams, G. R., & Niederer, S. (2011) · Progress in Biophysics and Molecular Biology · doi:10.1016/j.pbiomolbio.2011.06.003 · Cited by 49 · see citing works
Self-citationA reappraisal of how to build modular, reusable models of biological systems.
Neal, M. L., Cooling, M. T., Smith, L. P., Thompson, C. T., Sauro, H. M., Carlson, B. E., Cook, D. L., & Gennari, J. H. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003849 · Cited by 49 · see citing works
Li, Y., Wu, G., Dawe, G. S., Zeng, L., Cui, S.-S., Loers, G., Tilling, T., Sun, L. J., Schachner, M., & Xiao, Z. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003841 · Cited by 47 · see citing works
Neuroanatomical domain of the foundational model of anatomy ontology
Nichols, B. N., Mejino, J. L. V., Detwiler, L. T., Nilsen, T. T., Martone, M. E., Turner, J. A., Rubin, D. L., & Brinkley, J. F. (2014) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-5-1 · Cited by 46 · see citing works
Laman, M., Moore, B. R., Benjamin, J. M., et al. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001773 · Cited by 43 · see citing works
Probabilistic Approach to Predicting Substrate Specificity of Methyltransferases
Szczepińska, T., Kutner, J., Kopczyński, M., et al. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003514 · Cited by 40 · see citing works
A call for virtual experiments: Accelerating the scientific process
Cooper, J., Vik, J. O., & Waltemath, D. (2014) · Progress in Biophysics and Molecular Biology · doi:10.1016/j.pbiomolbio.2014.10.001 · Cited by 38 · see citing works
Self-citationWorkflow-Centric Research Objects: A First Class Citizen in the Scholarly Discourse
Belhajjame, K., Corcho, Ó., Garijo, D., et al. (2012) · Research Explorer (The University of Manchester) · Cited by 37 · see citing works
Corpas, M., Jiménez, R. C., Bongcam‐Rudloff, E., Budd, A., Brazas, M. D., Fernandes, P., Gaëta, B. A., van Gelder, C. W. G., Korpelainen, E., Lewitter, F., et al. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu601 · Cited by 30 · see citing works
Inferring Visuomotor Priors for Sensorimotor Learning
Turnham, E. J. A., Braun, D. A., & Wolpert, D. M. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001112 · Cited by 28 · see citing works
A specialized ODE integrator for the efficient computation of parameter sensitivities
Gonnet, P., Dimopoulos, S., Widmer, L. A., & Stelling, J. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-46 · Cited by 15 · see citing works
Insights into the pathogenesis of axial spondyloarthropathy from network and pathway analysis
Zhao, J., Chen, J., Yang, T.-H., et al. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-s1-s4 · Cited by 14 · see citing works
Connecting Models to Data in Multiscale Multicellular Tissue Simulations
Cooper, J., & Osborne, J. M. (2013) · Procedia Computer Science · doi:10.1016/j.procs.2013.05.235 · Cited by 9 · see citing works
Self-citationA general concept for consistent documentation of computational analyses
Ebert, P., Müller, F., Nordström, K. J. V., et al. (2015) · Database · doi:10.1093/database/bav050 · Cited by 8 · see citing works
Bioinformatics tools extracted from a typical mammalian genome project
Mangan, M. E. (2014) · Figshare · doi:10.6084/m9.figshare.1194910 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [64]http://arxiv.org/. Accessed 21 Oct 2015
Cited by
Sorted by citations
Kirouac, D. C., Cicali, B., & Schmidt, S. (2019) · CPT: Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12390 · Cited by 46 · see citing works
- Authors
- Lewis, J., Breeze, C. E., Charlesworth, J., Maclaren, O. J., & Cooper, J.
- Affiliations
- National Institute for Health and Care Research (GB)
- University College London (GB)
- University of Cambridge (GB)
- University of Auckland (NZ)
- University of Oxford (GB)
- Corresponding author
- Lewis, Joanna
- Venue
- BMC Systems Biology
- Published
- 2016-07-15
- Volume
- 10
- Issue
- 1
- Pages
- 52-52
- ISSN
- 1752-0509 (ISSN-L)
- PMC
- PMC4946111
- OpenAlex
- W2474648852
- Cited by
- 23
- References
- 70
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 4 · 2022: 1 · 2021: 4 · 2020: 3 · 2019: 3 · 2018: 2 · 2017: 3
- FWCI
- 5.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Systems Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- ResearchSpace (University of Auckland) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (GB) (EP/I017909/1)
- Engineering and Physical Sciences Research Council (EP/I017909/1)
- National Institute for Health Research (HPRU-2012-10080)
- EU-FP7 (316758)
- Software Sustainability Institute
- Biotechnology and Biological Sciences Research Council (GB) (BB/K017578/1)
- 2020 Science Programme (EPSRC) (EP/I017909/1)
- Software Sustainability Institute (EPSRC)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- reproducibility, computational biology, computer simulation, model description language, research reporting, science policy, research training, research incentives, science communication
- MeSH terms
- Software, Reproducibility of Results, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
