Reproducibility of computational workflows is automated using continuous analysis
Open accessPublished version of a preprint
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Replication, validation and extension of experiments are crucial for scientific progress. Computational experiments are scriptable and should be easy to reproduce. However, computational analyses are designed and run in a specific computing environment, which may be difficult or impossible to match using written instructions. We report the development of continuous analysis, a workflow that enables reproducible computational analyses. Continuous analysis combines Docker, a container technology akin to virtual machines, with continuous integration, a software development technique, to automatically rerun a computational analysis whenever updates or improvements are made to source code or data. This enables researchers to reproduce results without contacting the study authors. Continuous analysis allows reviewers, editors or readers to verify reproducibility without manually downloading and rerunning code and can provide an audit trail for analyses of data that cannot be shared.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
limma powers differential expression analyses for RNA-sequencing and microarray studies
Ritchie, M. E., Phipson, B., Wu, D., Hu, Y., Law, C. W., Shi, W., & Smyth, G. K. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv007 · Cited by 40,458
MAFFT: a novel method for rapid multiple sequence alignment based on fast Fourier transform
Katoh, K. (2002) · Nucleic Acids Research · doi:10.1093/nar/gkf436 · Cited by 17,027 · cited 2×
“The sequences were aligned using MAFFT 23 and converted to PHYLIP interleaved format using EMBOSS Seqret 33 .”
MethodNear-optimal probabilistic RNA-seq quantification
Bray, N. L., Pimentel, H., Melsted, P., & Pachter, L. (2016) · Nature Biotechnology · doi:10.1038/nbt.3519 · Cited by 9,723 · cited 1×
“This analysis quantified transcript counts using kallisto 26 , limma 27,28 and sleuth 29 ; performed principal component analysis and ran a differential expression analysis.”
MethodEMBOSS: the European molecular biology open software suite.
Rice, P., Longden, I., & Bleasby, A. (2000) · Trends in Genetics · doi:10.1016/s0168-9525(00)02024-2 · Cited by 9,466
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Organoid models of human and mouse ductal pancreatic cancer
Boj, S. F., Hwang, C.-I., Baker, L. A., Chio, I. I. C., Engle, D. D., Corbo, V., Jager, M., Ponz-Sarvise, M., Tiriac, H., Spector, M. S., et al. (2015) · Cell · doi:10.1016/j.cell.2014.12.021 · Cited by 2,150
Differential analysis of RNA-seq incorporating quantification uncertainty
Pimentel, H., Bray, N. L., Puente, S., Melsted, P., & Pachter, L. (2017) · Nature Methods · doi:10.1038/nmeth.4324 · Cited by 1,534
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 4×
“It has been proposed that open science could aid reproducibility 3,11 .”
BackgroundInfluentialAn introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059
FoxA1 Translates Epigenetic Signatures into Enhancer-Driven Lineage-Specific Transcription
Lupien, M., Eeckhoute, J., Meyer, C. A., Wang, Q., Zhang, Y., Li, W., Carroll, J. S., Liu, X. S., & Brown, M. (2008) · Cell · doi:10.1016/j.cell.2008.01.018 · Cited by 983
Cited by
Salmon provides fast and bias-aware quantification of transcript expression
Patro, R., Duggal, G., Love, M. I., Irizarry, R. A., & Kingsford, C. (2017) · Nature Methods · doi:10.1038/nmeth.4197 · Cited by 12,700
fMRIPrep: A robust preprocessing pipeline for functional MRI
Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., et al. (2019) · Nature Methods · doi:10.1038/s41592-018-0235-4 · Cited by 3,905
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310
Self-citationA comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
Data-analysis strategies for image-based cell profiling
Caicedo, J. C., Cooper, S., Heigwer, F., Warchal, S., Qiu, P., Molnar, C., Vasilevich, A. S., Barry, J. D., Bansal, H. S., Kraus, O., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4397 · Cited by 739
Metadata
- Authors
- Beaulieu-Jones, B. K., & Greene, C. S.
- Venue
- Nature Biotechnology
- Published
- 2017-03-13
- Volume
- 35
- Issue
- 4
- Pages
- 342-346
- ISSN
- 0733-222X (ISSN-L)
- 1087-0156
- 1546-1696
- OpenAlex
- W2597357676
- Cited by
- 151
- References
- 15
- FWCI
- 31.51
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reproducibility of computational workflows is automated using continuous analysis. https://app.cochat.ai/papers/10.1038/nbt.3780
- The paper
- Beaulieu-Jones, B. K., & Greene, C. S. (2017). Reproducibility of computational workflows is automated using continuous analysis. Nature Biotechnology. https://doi.org/10.1038/nbt.3780
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.