Reproducible Computational Workflows with Continuous Analysis
Open accessPreprint — published as 10.1038/nbt.3780 · Paper page
The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Reproducing experiments is vital to science. Being able to replicate, validate and extend previous work also speeds new research projects. Reproducing computational biology experiments, which are scripted, should be straightforward. But reproducing such work remains challenging and time consuming. In the ideal world we would be able to quickly and easily rewind to the precise computing environment where results were generated. We would then be able to reproduce the original analysis or perform new analyses. We introduce a process termed “continuous analysis” which provides inherent reproducibility to computational research at a minimal cost to the researcher. Continuous analysis combines Docker, a container service similar to virtual machines, with continuous integration, a popular software development technique, to automatically re-run computational analysis whenever relevant changes are made to the source code. This allows results to be reproduced quickly, accurately and without needing to contact the original authors. Continuous analysis also provides an audit trail for analyses that use data with sharing restrictions. This allows reviewers, editors, and readers to verify reproducibility without manually downloading and rerunning any code. Example configurations are available at our online repository ( https://github.com/greenelab/continuous_analysis ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We note that this setup overlaps considerably with the setup suggested for “continuous analysis” by Beaulieu-Jones and Greene [32].”
From Salmon provides fast and bias-aware quantification of transcript expression (2017) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
limma powers differential expression analyses for RNA-sequencing and microarray studies
Ritchie, M. E., Phipson, B., Wu, D., Hu, Y., Law, C. W., Shi, W., & Smyth, G. K. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv007 · Cited by 40,458
MAFFT: a novel method for rapid multiple sequence alignment based on fast Fourier transform
Katoh, K. (2002) · Nucleic Acids Research · doi:10.1093/nar/gkf436 · Cited by 17,027
Smyth, G. K. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1027 · Cited by 11,829
Near-optimal probabilistic RNA-seq quantification
Bray, N. L., Pimentel, H., Melsted, P., & Pachter, L. (2016) · Nature Biotechnology · doi:10.1038/nbt.3519 · Cited by 9,723
EMBOSS: the European molecular biology open software suite.
Rice, P., Longden, I., & Bleasby, A. (2000) · Trends in Genetics · doi:10.1016/s0168-9525(00)02024-2 · Cited by 9,466
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
IPython: A System for Interactive Scientific Computing
Perez, F., & Granger, B. E. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.53 · Cited by 4,009
Organoid models of human and mouse ductal pancreatic cancer
Boj, S. F., Hwang, C.-I., Baker, L. A., Chio, I. I. C., Engle, D. D., Corbo, V., Jager, M., Ponz-Sarvise, M., Tiriac, H., Spector, M. S., et al. (2015) · Cell · doi:10.1016/j.cell.2014.12.021 · Cited by 2,150
Evolving gene/transcript definitions significantly alter the interpretation of GeneChip data
Dai, M. (2005) · Nucleic Acids Research · doi:10.1093/nar/gni179 · Cited by 1,797 · cited 2×
“These nine used versions 11, 15, 16, 17, 18, and 19 of the BrainArray CustomCDF.”
BackgroundMethodReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 2×
“(3) The researcher pushes code that includes”
MethodDifferential analysis of RNA-seq incorporating quantification uncertaintyPreprint
Pimentel, H., Bray, N. L., Puente, S., Melsted, P., & Pachter, L. (2016) · Nature Methods · doi:10.1101/058164 · Cited by 1,401
Genome-wide analysis of estrogen receptor binding sites
Carroll, J. S., Meyer, C. A., Song, J., Li, W., Geistlinger, T. R., Eeckhoute, J., Brodsky, A. S., Keeton, E. K., Fertuck, K. C., Hall, G. F., et al. (2006) · Nature Genetics · doi:10.1038/ng1901 · Cited by 1,377 · cited 1×
“To determine the extent to which this issue affects high impact papers, we performed a parallel evaluation for the ten most cited papers 32–41 that cite Dai et al. We determined the ten most cited papers using Web of Science on May 31, 2016.”
MethodAn introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059
FoxA1 Translates Epigenetic Signatures into Enhancer-Driven Lineage-Specific Transcription
Lupien, M., Eeckhoute, J., Meyer, C. A., Wang, Q., Zhang, Y., Li, W., Carroll, J. S., Liu, X. S., & Brown, M. (2008) · Cell · doi:10.1016/j.cell.2008.01.018 · Cited by 983
Androgen Receptor Regulates a Distinct Transcription Program in Androgen-Independent Prostate Cancer
Wang, Q., Li, W., Zhang, Y., Yuan, X., Xu, K., Yu, J., Chen, Z., Beroukhim, R., Wang, H., Lupien, M., et al. (2009) · Cell · doi:10.1016/j.cell.2009.04.056 · Cited by 902
PPARγ and C/EBP factors orchestrate adipocyte biology via adjacent binding on a genome-wide scale
Lefterova, M. I., Zhang, Y., Steger, D. J., Schupp, M., Schug, J., Cristancho, A., Feng, D., Zhuo, D., Stoeckert, C. J., Liu, X. S., et al. (2008) · Genes & Development · doi:10.1101/gad.1709008 · Cited by 824
Silencing of microRNA families by seed-targeting tiny LNAs
Obad, S., dos Santos, C. O., Petri, A., Heidenblad, M., Broom, O., Ruse, C., Fu, C., Lindow, M., Stenvang, J., Straarup, E. M., et al. (2011) · Nature Genetics · doi:10.1038/ng.786 · Cited by 627
A humanized version of Foxp2 affects cortico-basal ganglia circuits in mice
Enard, W., Gehre, S., Hammerschmidt, K., Hölter, S. M., Blass, T., Somel, M., Brückner, M. K., Schreiweis, C., Winter, C., Sohr, R., et al. (2009) · Cell · doi:10.1016/j.cell.2009.03.041 · Cited by 605
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551
Continuous Integration: Improving Software Quality and Reducing Risk
Duvall, P., Matyas, S., & Glover, A. (2007) · Cited by 510
Tuupanen, S., Turunen, M., Lehtonen, R., Hallikas, O., Vanharanta, S., Kivioja, T., Björklund, M., Wei, G., Yan, J., Niittymäki, I., et al. (2009) · Nature Genetics · doi:10.1038/ng.406 · Cited by 504
Nucleosome dynamics define transcriptional enhancers
He, H. H., Meyer, C. A., Shin, H., Bailey, S. T., Wei, G., Wang, Q., Zhang, Y., Xu, K., Ni, M., Lupien, M., et al. (2010) · Nature Genetics · doi:10.1038/ng.545 · Cited by 475
Sweave: Dynamic Generation of Statistical Reports Using Literate Data Analysis
Leisch, F. (2002) · Compstat · doi:10.1007/978-3-642-57489-4_89 · Cited by 422
Zuo, T., Wang, L., Morrison, C., Chang, X., Zhang, H., Li, W., Liu, Y., Wang, Y., Liu, X., Chan, M. W. Y., et al. (2007) · Cell · doi:10.1016/j.cell.2007.04.034 · Cited by 395
An introduction to Docker for reproducible research, with examples from the R environmentPreprint
Boettiger, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0846 · Cited by 372
A HUPO test sample study reveals common problems in mass spectrometry–based proteomics
HUPO Test Sample Working Group, Bell, A. W., Deutsch, E. W., Au, C. E., Kearney, R. E., Beavis, R., Sechi, S., Nilsson, T., & Bergeron, J. J. M. (2009) · Nature Methods · doi:10.1038/nmeth.1333 · Cited by 351
High-throughput mapping of the chromatin structure of human promoters
Ozsolak, F., Song, J. S., Liu, X. S., & Fisher, D. E. (2007) · Nature Biotechnology · doi:10.1038/nbt1279 · Cited by 335
McNutt, M. (2014) · Science · doi:10.1126/science.1250475 · Cited by 291
Dekkers, B. J. W., He, H., Hanson, J., Willems, L. A. J., Jamar, D. C. L., Cueff, G., Rajjou, L., Hilhorst, H. W. M., & Bentsink, L. (2016) · The Plant Journal · doi:10.1111/tpj.13118 · Cited by 201
R Markdown: Integrating A Reproducible Analysis Tool into Introductory Statistics
Baumer, B., Cetinkaya-Rundel, M., Bray, A., Loi, L., & Horton, N. J. (2014) · Technology Innovations in Statistics Education · doi:10.5070/t581020118 · Cited by 137
Perceptions of Emergency Department Crowding in the Commonwealth of Pennsylvania
Pines, J., Isserman, J., & Kelly, J. (2013) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2011.5.6700 · Cited by 118
Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome
Garijo, D., Kinnings, S., Xie, L., Xie, L., Zhang, Y., Bourne, P. E., & Gil, Y. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080278 · Cited by 116
The Mycobacterium tuberculosis Drugome and Its Polypharmacological Implications
Kinnings, S., Xie, L., Fung, K. H., Jackson, R. M., Xie, L., & Bourne, P. E. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000976 · Cited by 109
mTORC1 and CK2 coordinate ternary and eIF4F complex assembly
Gandin, V., Masvidal, L., Cargnello, M., Gyenis, L., McLaughlan, S., Cai, Y., Tenkerian, C., Morita, M., Balanathan, P., Jean-Jean, O., et al. (2016) · Nature Communications · doi:10.1038/ncomms11127 · Cited by 103 · cited 2×
“We performed a parallel analysis using each of the three versions that we found installed on machines that we could access (18, 19, and 20).”
BackgroundMethodGenomic hallmarks of homologous recombination deficiency in invasive breast carcinomas
Manié, É., Popova, T. G., Battistella, A., Tarabeux, J., Caux‐Moncoutier, V., Golmard, L., Smith, N. K., Mueller, C. R., Mariani, O., Sigal‐Zafrani, B., et al. (2015) · International Journal of Cancer · doi:10.1002/ijc.29829 · Cited by 91
Case studies in reproducibility
Hothorn, T., & Leisch, F. (2011) · Briefings in Bioinformatics · doi:10.1093/bib/bbq084 · Cited by 77 · cited 1×
“These nine used versions 11, 15, 16, 17, 18, and 19 of the BrainArray CustomCDF.”
MethodDrug-set enrichment analysis: a novel tool to investigate drug mode of action
Napolitano, F., Sirci, F., Carrella, D., & di Bernardo, D. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btv536 · Cited by 75 · cited 1×
“We analyzed the fifteen most recently published papers that cite Dai et al., a common source for custom chip description files (CustomCDF), that were accessible at our institution 16–31 .”
BackgroundDisease, population viability, and recovery of endangered Sierra Nevada bighorn sheep
Cahn, M. L., Conner, M. M., Schmitz, O. J., Stephenson, T. R., Wehausen, J. D., & Johnson, H. E. (2011) · Journal of Wildlife Management · doi:10.1002/jwmg.232 · Cited by 71
Distinguishing benign from pathologic TH2 immunity in atopic children
Holt, P. G., Strickland, D., Bosco, A., Belgrave, D., Hales, B., Simpson, A., Hollams, E., Holt, B., Kusel, M., Ahlstedt, S., et al. (2016) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2015.08.044 · Cited by 68
Kopljar, I., Gallacher, D. J., Bondt, A. D., Cougnaud, L., Vlaminckx, E., den Wyngaert, I. V., & Lu, H. R. (2016) · Stem Cells Translational Medicine · doi:10.5966/sctm.2015-0279 · Cited by 57
Bosco, A., Wiehler, S., & Proud, D. (2016) · BMC Genomics · doi:10.1186/s12864-016-2405-z · Cited by 56
Troy, N. M., Hollams, E. M., Holt, P. G., & Bosco, A. (2016) · BMC Medical Genomics · doi:10.1186/s12920-016-0171-z · Cited by 53
Semi-Supervised Learning of the Electronic Health Record for Phenotype StratificationPreprint
Beaulieu‐Jones, B. K., & Greene, C. S. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/039800 · Cited by 52
Self-citationReduced number of peripheral natural killer cells in schizophrenia but not in bipolar disorder
Karpiński, P., Frydecka, D., Sasiadek, M. M., & Misiak, B. (2016) · Brain Behavior and Immunity · doi:10.1016/j.bbi.2016.02.005 · Cited by 44
Open science and reproducible research
Groves, T., & Godlee, F. (2012) · BMJ · doi:10.1136/bmj.e4383 · Cited by 37 · cited 1×
“The practice of “open science” has been discussed as a means to aid reproducibility 3,12 .”
GMCM: Unsupervised Clustering and Meta-Analysis Using Gaussian Mixture Copula Models
Bilgrau, A. E., Eriksen, P. S., Rasmussen, J. G., Johnsen, H. E., Dybkær, K., & Boegsted, M. (2016) · Journal of Statistical Software · doi:10.18637/jss.v070.i02 · Cited by 37
TLR3 Signaling Promotes the Induction of Unique Human BDCA-3 Dendritic Cell Populations
Colletti, N. J., Liu, H., Gower, A. C., Alekseyev, Y. O., Arendt, C., & Shaw, M. H. (2016) · Frontiers in Immunology · doi:10.3389/fimmu.2016.00088 · Cited by 36
Circadian mRNA expression: insights from modeling and transcriptomics
Lück, S., & Westermark, P. O. (2015) · Cellular and Molecular Life Sciences · doi:10.1007/s00018-015-2072-2 · Cited by 34
Scalable and cost-effective NGS genotyping in the cloud
Souilmi, Y., Lancaster, A. K., Jung, J.-Y., Rizzo, E., Hawkins, J. B., Powles, R. L., Amzazi, S., Ghazal, H., Tonellato, P. J., & Wall, D. P. (2015) · BMC Medical Genomics · doi:10.1186/s12920-015-0134-9 · Cited by 30
(2014) · Nature Methods · doi:10.1038/nmeth.2880 · Cited by 27
Killeen, A. P., Diskin, M. G., Morris, D. G., Kenny, D. A., & Waters, S. M. (2016) · Physiological Genomics · doi:10.1152/physiolgenomics.00042.2015 · Cited by 25
Computational selection of antibody-drug conjugate targets for breast cancer
Fauteux, F., Hill, J. J., Jaramillo, M., Pan, Y., Phan, S., Famili, F. A., & O'Connor‐McCourt, M. D. (2015) · Oncotarget · doi:10.18632/oncotarget.6679 · Cited by 23
(2006) · Nature · doi:10.1038/442001a · Cited by 21
Brummelman, J., Raeven, R. H. M., Helm, K., Pennings, J. L. A., Metz, B., van Eden, W., van Els, C. A. C. M., & Han, W. G. H. (2016) · Scientific Reports · doi:10.1038/srep25064 · Cited by 19
(2015) · Nature Biotechnology · doi:10.1038/nbt.3202 · Cited by 14
Núñez, M., Sánchez-Jiménez, C., Alcalde, J., & Izquierdo, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0113141 · Cited by 6
Lee, M., Huang, R., & Tong, W. (2016) · Toxicological Sciences · doi:10.1093/toxsci/kfv261 · Cited by 4
Denoising Autoencoders for Phenotype Stratification (DAPS): Preprint Release
Beaulieu-Jones, B. K. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.46165 · Cited by 3
Self-citationContinuous Analysis Example Docker Images
Brett, B.-J. (2016) · Figshare · doi:10.6084/m9.figshare.3545156.v1 · Cited by 1
Continuous Analysis Example Docker Images
Beaulieu‐Jones, B. K. (2016) · Figshare · doi:10.6084/m9.figshare.3545156 · Cited by 1
Self-citation
Cited by
Salmon provides fast and bias-aware quantification of transcript expression
Patro, R., Duggal, G., Love, M. I., Irizarry, R. A., & Kingsford, C. (2017) · Nature Methods · doi:10.1038/nmeth.4197 · Cited by 12,700
Näpflin, K., O’Connor, E., Becks, L., Bensch, S., Ellis, V. A., Hafer, N., Hårding, K. C., Lindén, S. K., Olsen, M. T., Roved, J., et al. (2019) · PeerJ · doi:10.7717/peerj.8013 · Cited by 55
Eleven quick tips for architecting biomedical informatics workflows with cloud computing
Cole, B. S., & Moore, J. H. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005994 · Cited by 17
“Additionally, containers can be leveraged for reproducible computational analysis [14].”
BackgroundOpen Publication, Digital Abundance, and Scarce Labour
Eve, M. P. (2017) · Journal of Scholarly Publishing · doi:10.3138/jsp.49.1.26 · Cited by 10
“One potential solution that Brown proposes, following Beaulieu-Jones and Greene, is a continuous integration-esque model in which software is run in perpetuity so that immediate failures are obvious and can be regularly corrected.(21) The level of expertise and degree of labour that would be required by research labs,…”
BackgroundSochat, V. (2018) · GigaScience · doi:10.1093/gigascience/giy023 · Cited by 6
“Each contribution is then assessed by the predefined metric of goodness or assessed as the analysis code changes over time [25].”
Method
Metadata
- Authors
- Beaulieu-Jones, B. K., & Greene, C. S.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Greene, Casey S.
- Venue
- Nature Biotechnology
- Publisher
- openRxiv
- Published
- 2016-06-01
- Volume
- 35
- Issue
- 4
- Pages
- 342-346
- ISSN
- 0733-222X (ISSN-L)
- PMC
- PMC6103790
- OpenAlex
- W2597357676
- Cited by
- 154
- References
- 53
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 19 · 2023: 11 · 2022: 12 · 2021: 21 · 2020: 15 · 2019: 27 · 2018: 20 · 2017: 13
- FWCI
- 33.02
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- Nature Biotechnology (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Gordon and Betty Moore Foundation (GBMF4552)
- Pennsylvania Department of Health
- National Institute of Allergy and Infectious Diseases (R01AI116794)
- U.S. National Library of Medicine (R01LM010098)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- reproducibility, computational workflows, continuous analysis, continuous integration, Docker, containerization, computational reproducibility, audit trail, numerical experiments
- MeSH terms
- Machine Learning, Sensitivity and Specificity, Software, User-Computer Interface, Reproducibility of Results, Computational Biology, Gene Expression Profiling, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.