Reproducible Computational Workflows with Continuous Analysis
Open accessPreprint — published as 10.1038/nbt.3780 · Paper page
The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Reproducing experiments is vital to science. Being able to replicate, validate and extend previous work also speeds new research projects. Reproducing computational biology experiments, which are scripted, should be straightforward. But reproducing such work remains challenging and time consuming. In the ideal world we would be able to quickly and easily rewind to the precise computing environment where results were generated. We would then be able to reproduce the original analysis or perform new analyses. We introduce a process termed “continuous analysis” which provides inherent reproducibility to computational research at a minimal cost to the researcher. Continuous analysis combines Docker, a container service similar to virtual machines, with continuous integration, a popular software development technique, to automatically re-run computational analysis whenever relevant changes are made to the source code. This allows results to be reproduced quickly, accurately and without needing to contact the original authors. Continuous analysis also provides an audit trail for analyses that use data with sharing restrictions. This allows reviewers, editors, and readers to verify reproducibility without manually downloading and rerunning any code. Example configurations are available at our online repository ( https://github.com/greenelab/continuous_analysis ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We note that this setup overlaps considerably with the setup suggested for “continuous analysis” by Beaulieu-Jones and Greene [32].”
From Salmon provides fast and bias-aware quantification of transcript expression (2017) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
limma powers differential expression analyses for RNA-sequencing and microarray studies
Ritchie, M. E., Phipson, B., Wu, D., Hu, Y., Law, C. W., Shi, W., & Smyth, G. K. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv007 · Cited by 40,458
MAFFT: a novel method for rapid multiple sequence alignment based on fast Fourier transform
Katoh, K. (2002) · Nucleic Acids Research · doi:10.1093/nar/gkf436 · Cited by 17,027
Smyth, G. K. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1027 · Cited by 11,829
Near-optimal probabilistic RNA-seq quantification
Bray, N. L., Pimentel, H., Melsted, P., & Pachter, L. (2016) · Nature Biotechnology · doi:10.1038/nbt.3519 · Cited by 9,723
EMBOSS: the European molecular biology open software suite.
Rice, P., Longden, I., & Bleasby, A. (2000) · Trends in Genetics · doi:10.1016/s0168-9525(00)02024-2 · Cited by 9,466
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
IPython: A System for Interactive Scientific Computing
Perez, F., & Granger, B. E. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.53 · Cited by 4,009
Organoid models of human and mouse ductal pancreatic cancer
Boj, S. F., Hwang, C.-I., Baker, L. A., Chio, I. I. C., Engle, D. D., Corbo, V., Jager, M., Ponz-Sarvise, M., Tiriac, H., Spector, M. S., et al. (2015) · Cell · doi:10.1016/j.cell.2014.12.021 · Cited by 2,150
Evolving gene/transcript definitions significantly alter the interpretation of GeneChip data
Dai, M. (2005) · Nucleic Acids Research · doi:10.1093/nar/gni179 · Cited by 1,797 · cited 2×
“These nine used versions 11, 15, 16, 17, 18, and 19 of the BrainArray CustomCDF.”
BackgroundMethodReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 2×
“(3) The researcher pushes code that includes”
Method
Cited by
Salmon provides fast and bias-aware quantification of transcript expression
Patro, R., Duggal, G., Love, M. I., Irizarry, R. A., & Kingsford, C. (2017) · Nature Methods · doi:10.1038/nmeth.4197 · Cited by 12,700
Näpflin, K., O’Connor, E., Becks, L., Bensch, S., Ellis, V. A., Hafer, N., Hårding, K. C., Lindén, S. K., Olsen, M. T., Roved, J., et al. (2019) · PeerJ · doi:10.7717/peerj.8013 · Cited by 55
Eleven quick tips for architecting biomedical informatics workflows with cloud computing
Cole, B. S., & Moore, J. H. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005994 · Cited by 17
“Additionally, containers can be leveraged for reproducible computational analysis [14].”
BackgroundOpen Publication, Digital Abundance, and Scarce Labour
Eve, M. P. (2017) · Journal of Scholarly Publishing · doi:10.3138/jsp.49.1.26 · Cited by 10
“One potential solution that Brown proposes, following Beaulieu-Jones and Greene, is a continuous integration-esque model in which software is run in perpetuity so that immediate failures are obvious and can be regularly corrected.(21) The level of expertise and degree of labour that would be required by research labs,…”
BackgroundSochat, V. (2018) · GigaScience · doi:10.1093/gigascience/giy023 · Cited by 6
“Each contribution is then assessed by the predefined metric of goodness or assessed as the analysis code changes over time [25].”
Method
Metadata
- Authors
- Beaulieu-Jones, B. K., & Greene, C. S.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Greene, Casey S.
- Venue
- Nature Biotechnology
- Publisher
- openRxiv
- Published
- 2016-06-01
- Volume
- 35
- Issue
- 4
- Pages
- 342-346
- ISSN
- 0733-222X (ISSN-L)
- PMC
- PMC6103790
- OpenAlex
- W2597357676
- Cited by
- 154
- References
- 53
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 19 · 2023: 11 · 2022: 12 · 2021: 21 · 2020: 15 · 2019: 27 · 2018: 20 · 2017: 13
- FWCI
- 33.02
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- Nature Biotechnology (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Gordon and Betty Moore Foundation (GBMF4552)
- Pennsylvania Department of Health
- National Institute of Allergy and Infectious Diseases (R01AI116794)
- U.S. National Library of Medicine (R01LM010098)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- reproducibility, computational workflows, continuous analysis, continuous integration, Docker, containerization, computational reproducibility, audit trail, numerical experiments
- MeSH terms
- Machine Learning, Sensitivity and Specificity, Software, User-Computer Interface, Reproducibility of Results, Computational Biology, Gene Expression Profiling, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.