Ten Simple Rules for Reproducible Computational Research
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Replication is the cornerstone of a cumulative science [1]. However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2]. As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3]. This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4]. The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8], [9]. This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3]. We want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run. We further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities. In a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys. As a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible. With this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6], [14]. We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your researc
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival1,11,12, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
From The FAIR Guiding Principles for scientific data management and stewardship (2016) · cites this paper as [12]
In the citing paper
“In a recent publication reviewing the automated citation screening methods reported for systematic reviews, the number of studies that met the current standards for reproducible software engineering experiments was low [29, 30, 36]; of 33 studies reporting approaches to automated screening, no study provided all the criteria for reproducible reporting [36].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundMinimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundGoecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome Biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,453 · cited 1×
“For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19], the GenePattern Word add-in [4], and Galaxy Pages [20].”
MethodSelf-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundReich, M., Liefeld, T., Gould, J., Lerner, J., Tamayo, P., & Mesirov, J. P. (2006) · Nature Genetics · doi:10.1038/ng0506-500 · Cited by 2,032
Galaxy: A platform for interactive large-scale genome analysis
Giardine, B., Riemer, C., Hardison, R. C., Burhans, R., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,026 · cited 2×
“If you are instead performing your analyses through an integrated framework (such as Gene-Pattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules.”
BackgroundSelf-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“…through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8,9].”
ResultTaverna: a tool for the composition and enactment of bioinformatics workflows
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M. R., Wipat, A., et al. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth361 · Cited by 1,602 · cited 2×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 3×
“In order to foster such aspects, it has been suggested that the scientific community needs to develop a ‘‘culture of reproducibility’’ for computational science, and to require it for published claims [3].”
BackgroundPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222 · cited 1×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
Background
Cited by
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
“…of recent, often domain-focused publications advocating for speci fi c improvements in practices relating to data management and archival 1 , 11 , 12 , FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
BackgroundEnvironmental DNA metabarcoding: Transforming how we survey animal and plant communities
Deiner, K., Bik, H. M., Mächler, E., Seymour, M., Lacoursière‐Roussel, A., Altermatt, F., Creer, S., Bista, I., Lodge, D. M., de Vere, N., et al. (2017) · Molecular Ecology · doi:10.1111/mec.14350 · Cited by 1,918
“Sandve et al. (2013) provide 10 rules that can be followed to ensure such reproducibility, and we strongly encourage researchers using eDNA metabarcoding methods to uphold these practices and take advantage of archiving intermediate steps (Box 2) of their analysis for full transparency.”
MethodAn Introduction to Deep Reinforcement Learning
Vincent, F.-L., Peter, H., Riashat, I., Marc, G. B., & Joelle, P. (2018) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000071 · Cited by 1,286
“Across various fields, investigations in reproducibility have found problems in numerous publications, resulting in several works providing experimental guidelines in proper scientific practices (Sandve et al., 2013; Baker, 2016; Halsey et al., 2015; Casadevall and Fang, 2010).”
landscapemetrics : an open‐source R tool to calculate landscape metrics
Hesselbarth, M. H. K., Sciaini, M., With, K. A., Wiegand, K., & Nowosad, J. (2019) · Ecography · doi:10.1111/ecog.04617 · Cited by 1,280
Creation and analysis of biochemical constraint-based models using the COBRA Toolbox v.3.0
Heirendt, L., Arreckx, S., Pfau, T., Mendoza, S. N., Richelle, A., Heinken, A., Haraldsdóttir, H. S., Wachowiak, J., Keating, S. M., Vlasov, V., et al. (2019) · Nature Protocols · doi:10.1038/s41596-018-0098-2 · Cited by 1,242
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
“…relevant concentrations Smith & Houghton (2013)44 Consider litter effects Lazic & Essioux (2013)28 Recommendations to improve computational biology Sandve et al (2013)46 Focus on reproducibility in training, grants, journals LeVeque et al (2012)61 Pathology: Biospecimen quality control…”
Ten quick tips for machine learning in computational biology
Chicco, D. (2017) · BioData Mining · doi:10.1186/s13040-017-0155-3 · Cited by 878
“Even more, releasing your code openly in the internet also allows the computational reproducibility of your paper results [61].”
BackgroundO'Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G., Moher, D., et al. (2021) · Biological Reviews · doi:10.1111/brv.12721 · Cited by 697
Spatially distinct tumor immune microenvironments stratify triple-negative breast cancers
Gruosso, T., Gigoux, M., Manem, V. S. K., Bertos, N., Zuo, D., Perlitch, I., Saleh, S. M. I., Zhao, H., Souleimanova, M., Johnson, R. M., et al. (2019) · Journal of Clinical Investigation · doi:10.1172/jci96313 · Cited by 439
Text Preprocessing for Text Mining in Organizational Research: Review and Recommendations
Hickman, L., Thapa, S., Tay, L., Cao, M., & Srinivasan, P. (2022) · Organizational Research Methods · doi:10.1177/1094428120971683 · Cited by 362
“One of the best ways to ensure replicability would be to post the source code used to generate all analytic results (Sandve et al., 2013).”
BackgroundMachado, D., & Herrgård, M. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003580 · Cited by 411
“These types of method comparison studies also play an important role in ensuring the reproducibility of computational research as they verify that published method implementations function as intended with novel data sets [15].”
MethodITK: enabling reproducible research and open science
McCormick, M., Liu, X., Jomier, J., Marion, C., & Ibanez, L. (2014) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2014.00013 · Cited by 376
“Nor will they be subject to the veil of suspicion caused by many incidents of sci-entific fraud and data manufacturing that have been reported in recent months (Couzin, 2006; Begley and Ellis, 2012; Mobley et al., 2013; Sandve et al., 2013).”
Nygaard, V., Rødland, E. A., & Hovig, E. (2016) · Biostatistics · doi:10.1093/biostatistics/kxv027 · Cited by 376
Self-citationA brief history of bioinformatics
Gauthier, J., Vincent, A. T., Charette, S. J., & Derome, N. (2019) · Briefings in Bioinformatics · doi:10.1093/bib/bby063 · Cited by 375
“Furthermore, switching to newer and/or different bioinformatics tools requires additional training and testing, thereby making researchers reluctant to abandon software they are familiar with [114].”
BackgroundBlockchain technology for improving clinical research quality
Benchoufi, M., & Ravaud, P. (2017) · Trials · doi:10.1186/s13063-017-2035-z · Cited by 367
“Thus, the database opens a wide path to the data user ’ s control or differential privacy, data sharing and community-driven clinical study [23]: in a trusted environment, clinical research teams can “ crowd-recruit ” people to be enrolled in protocols with the help of community management…”
BackgroundSmith, A. M., Katz, D. S., Niemeyer, K. E., & FORCE11 Software Citation Working Group (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.86 · Cited by 276
“Research results, including data, increasingly depend on software (Hannay et al., 2009), and thus may depend on the specific version used (Sandve et al., 2013; Wilson et al., 2014).”
BackgroundInflammatory Bowel Disease as a Model for Translating the Microbiome
Huttenhower, C., Kostic, A. D., & Xavier, R. J. (2014) · Immunity · doi:10.1016/j.immuni.2014.05.013 · Cited by 332
“Microbiome assays are currently particularly sensitive to batch effects and interpopulation differences (Koren et al., 2013; Leek et al., 2010), in addition to the multiple hypothesis testing issues common to any high-dimensional analysis (Sandve et al., 2013).”
BackgroundPublic Data Archiving in Ecology and Evolution: How Well Are We Doing?
Roche, D. G., Kruuk, L. E. B., Lanfear, R., & Binning, S. A. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002295 · Cited by 303
“, [53,54]).”
Schloss, P. D. (2018) · mBio · doi:10.1128/mbio.00525-18 · Cited by 280
Opening the black box of energy modelling: Strategies and lessons learned
Pfenninger, S., Hirth, L., Schlecht, I., Schmid, E., Wiese, F., Brown, T., Davis, C., Gidden, M., Heinrichs, H., Heuberger, C., et al. (2018) · Energy Strategy Reviews · doi:10.1016/j.esr.2017.12.002 · Cited by 248
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293
Ten quick tips for effective dimensionality reduction
Nguyen, L. H., & Holmes, S. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006907 · Cited by 270
“Recording your choices is a crucial part of reproducible research [59]; it allows others to replicate the same results you obtained and speeds up your analysis process the next time you work with similar data.”
Our path to better science in less time using open data science tools
Lowndes, J. S. S., Best, B. D., Scarborough, C., Afflerbach, J. C., Frazier, M. R., O’Hara, C. C., Jiang, N., & Halpern, B. S. (2017) · Nature Ecology & Evolution · doi:10.1038/s41559-017-0160 · Cited by 286
Prospects and challenges of multi-omics data integration in toxicology
Canzler, S., Schor, J., Busch, W., Schubert, K., Rolle-Kampczyk, U. E., Seitz, H., Kamp, H., von Bergen, M., Buesen, R., & Hackermüller, J. (2020) · Archives of Toxicology · doi:10.1007/s00204-020-02656-y · Cited by 286
“Linking executable analysis code with intermediate and final resulting data is promoted as the gold standard of reproducibility by Peng (2011), while Sandve and colleagues suggest to follow the ”ten simple rules for reproducible computational research” (Sandve et al. 2013).”
MethodDOME: recommendations for supervised machine learning validation in biology
Walsh, I., Fishman, D., Garcia-Gasulla, D., Titma, T., Pollastri, G., ELIXIR Machine Learning Focus Group, Capriotti, E., Casadio, R., Capella-Gutierrez, S., Cirillo, D., et al. (2021) · Nature Methods · doi:10.1038/s41592-021-01205-4 · Cited by 282
Packaging research artefacts with RO-Crate
Soiland-Reyes, S., Sefton, P., Crosas, M., Castro, L. J., Coppens, F., Fernández, J. M., Garijo, D., Grüning, B., La Rosa, M., Leo, S., et al. (2022) · Data Science · doi:10.3233/ds-210053 · Cited by 200
“Important examples include data publication with rich metadata (e.g. Zenodo [40]), domain-specific data deposition (e.g. PDB [16]) and following practices for reproducible research software [101] (e.g. use of containers).”
InfluentialHuppmann, D., Gidden, M., Fricko, O., Kolp, P., Orthofer, C., Pimmer, M., Kushin, N., Vinca, A., Mastrucci, A., Riahi, K., et al. (2019) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2018.11.012 · Cited by 247
Reproducible Research and GIScience: An Evaluation Using GIScience Conference Papers
Ostermann, F. O., Nüst, D., Granell, C., Hofer, B., & Konkol, M. (2018) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2021.ii.2 · Cited by 135
Alam, K. K., Chang, J. L., & Burke, D. H. (2015) · Molecular Therapy - Nucleic Acids · doi:10.1038/mtna.2015.4 · Cited by 260
Addendum: The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Jan Aalbersberg, I., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2019) · Scientific Data · doi:10.1038/s41597-019-0009-6 · Cited by 258
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 129
“Similarly, it is important to report the exact version of code used for running the experiments and producing the results in a paper ( 105 ).”
Pick, J. L., Nakagawa, S., & Noble, D. W. A. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13118 · Cited by 257
pybliometrics: Scriptable bibliometrics using a Python interface to Scopus
Rose, M. E., & Kitchin, J. R. (2019) · SoftwareX · doi:10.1016/j.softx.2019.100263 · Cited by 228
Open science, reproducibility, and transparency in ecology
Powers, S. M., & Hampton, S. E. (2019) · Ecological Applications · doi:10.1002/eap.1822 · Cited by 224
Repeatability in computer systems research
Collberg, C., & Proebsting, T. A. (2016) · Communications of the ACM · doi:10.1145/2812803 · Cited by 242
“Previous work on repeatability describes the steps that must be taken in order to produce research that is truly repeatable 11,12 or describes tools or websites that support publication of repeatable research.”
BackgroundSharing Clinical Trial Data: Maximizing Benefits, Minimizing Risk
Lo, B. (2015) · JAMA · doi:10.1001/jama.2015.292 · Cited by 233
Chicco, D., Sichenze, A., & Jurman, G. (2025) · BioData Mining · doi:10.1186/s13040-025-00465-6 · Cited by 277
Performing Skin Microbiome Research: A Method to the Madness
Kong, H. H., Andersson, B., Clavel, T., Common, J. E., Jackson, S. A., Olson, N. D., Segre, J. A., & Traidl-Hoffmann, C. (2017) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2016.10.033 · Cited by 227
“Depositing primary data into public databases enables independent cross-evaluation and comparison with previous studies, but requires access to raw data, metadata, software, and code used to process the data (Sandve et al., 2013; Stodden et al., 2014).”
BackgroundMethodTen simple rules for writing and sharing computational analyses in Jupyter Notebooks
Rule, A., Birmingham, A., Zuniga, C., Altintas, I., Huang, S.-C., Knight, R., Moshiri, N., Nguyen, M. H., Rosenthal, S. B., Pérez, F., et al. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007007 · Cited by 168
NanoSIMS for biological applications: Current practices and analyses
Nuñez, J., Renslow, R., Cliff, J. B., & Anderton, C. R. (2018) · Biointerphases · doi:10.1116/1.4993628 · Cited by 211
Towards reproducible computational drug discovery
Schaduangrat, N., Lampa, S., Simeon, S., Gleeson, M. P., Spjuth, O., & Nantasenamat, C. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-0408-x · Cited by 208
Journal of Computational Neuroscience
(2013) · Journal of Computational Neuroscience · doi:10.1007/10827.1573-6873 · Cited by 190
A practical guide to DNA-based methods for biodiversity assessment
Bruce, K., Blackman, R., Bourlat, S. J., Hellström, A. M., Bakker, J., Bista, I., Bohmann, K., Bouchez, A., Brys, R., Clark, K., et al. (2021) · Pensoft Publishers eBooks · doi:10.3897/ab.e68634 · Cited by 175
“More broadly, the ten simple rules for reproducible computational research (Sandve et al. 2013) have been advocated as best practice standards in DNA based species monitoring (Deiner et al.”
BackgroundTools and techniques for computational reproducibility
Piccolo, S. R., & Frampton, M. B. (2016) · GigaScience · doi:10.1186/s13742-016-0135-4 · Cited by 169
Marwick, B. (2017) · Journal of Archaeological Method and Theory · doi:10.1007/s10816-015-9272-9 · Cited by 179
“2008; Kahle and Wickham 2013; Sarkar 2008; Wickham 2009).”
COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project
Bergmann, F. T., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., Hucka, M., Laibe, C., Miller, A. K., Nickerson, D. P., et al. (2014) · BMC Bioinformatics · doi:10.1186/s12859-014-0369-z · Cited by 173
“Reproducible science has even become a field of research in its own right [1,2].”
BackgroundAssessing data availability and research reproducibility in hydrology and water resources
Stagge, J. H., Rosenberg, D. E., Abdallah, A. M., Akbar, H., Attallah, N. A., & James, R. (2019) · Scientific Data · doi:10.1038/sdata.2019.30 · Cited by 165
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167
A large-scale study on research code quality and execution
Trisovic, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022) · Scientific Data · doi:10.1038/s41597-022-01143-6 · Cited by 135
“Extensive guides on code practices have been published [30, 31, 32, 33, 34] and are available online [35].”
BackgroundThe role of metadata in reproducible computational research
Leipzig, J., Nüst, D., Hoyt, C. T., Soiland-Reyes, S., Ram, K., & Greenberg, J. (2020) · Patterns · doi:10.1016/j.patter.2021.100322 · Cited by 130
“Software development practices, such as documentation and testing, are not a standard requirement of the doctoral curriculum, the peer-review process, or the funding structure, and as a result, the scientific community suffers from diminished reuse and reproducibility.31 Sandve et al.32 identified the most common sour…”
BackgroundInformation visualisation for science and policy: engaging users and avoiding bias
McInerny, G. J., Chen, M., Freeman, R., Gavaghan, D., Meyer, M., Rowland, F., Spiegelhalter, D. J., Stefaner, M., Tessarolo, G., & Hortal, J. (2014) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2014.01.003 · Cited by 147
Recommendations to enhance rigor and reproducibility in biomedical research
Brito, J. J., Li, J., Moore, J. H., Greene, C. S., Nogoy, N. A., Garmire, L. X., & Mangul, S. (2020) · GigaScience · doi:10.1093/gigascience/giaa056 · Cited by 153
“Prior studies [4] have addressed issues of computational reproducibility, including the need to automate all data manipulation tasks and version control of code.”
The quest for understanding phenotypic variation via integrated approaches in the field environment
Pauli, D., Chapman, S. C., Bart, R. S., Topp, C. N., Lawrence‐Dill, C. J., Poland, J., & Gore, M. A. (2016) · PLANT PHYSIOLOGY · doi:10.1104/pp.16.00592 · Cited by 172
“…it both discoverable and reusable, Minimal Information About a Plant Phenotyping Experiment standards (http://cropnet.pl/ phenotypes; Krajewski et al., 2015) should be followed, and methods used to measure phenotypes should be detailed at a level that enables reproducibility (Sandve et al., 2013).”
MethodIncorporating Machine Learning into Established Bioinformatics Frameworks
Auslander, N., Gussow, A. B., & Koonin, E. V. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms22062903 · Cited by 169
“…and miRNA functional information [151] Disease-phenotypevisualization t-SNE OMIM database and human disease networks [154] Biomarker Reproducibility Acceptance Documentation and deposition of the processed data [195] - Benchmarking of the processing pipeline and optimized parameters [196] -”
BackgroundMethodUAS-Based Plant Phenotyping for Research and Breeding Applications
Guo, W., Carroll, M. E., Singh, A., Swetnam, T. L., Merchant, N. C., Sarkar, S., Singh, A. K., & Ganapathysubramanian, B. (2021) · Plant Phenomics · doi:10.34133/2021/9840192 · Cited by 161
“WMS also afford the necessary reproducibility [154], by keeping track of input parameters used for applications and processed outputs for every step, with the ability to perform advanced analysis that requires parameters sweep, e.”
BackgroundHow blockchain technology can change medicine
Roman-Belmonte, J. M., De la Corte-Rodriguez, H., & Rodriguez-Merchan, E. C. (2018) · Postgraduate Medicine · doi:10.1080/00325481.2018.1472996 · Cited by 136
“Data analysis can be performed automatically, according to the statistical analysis plan designed beforehand [25].”
MethodSustainable computational science: the ReScience initiative
Rougier, N. P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L. A., Benureau, F. C. Y., Brown, C. T., de Buyl, P., Caglayan, O., Davison, A. P., et al. (2017) · PeerJ Computer Science · doi:10.7717/peerj-cs.142 · Cited by 120
“All ReScience activities are organized around a few such Open Source projects hosted by GitHub. is an unusual choice for a scientific journal, the only other journal hosted on GitHub being The Journal of Open Source Software(Smith et al., 2017).”
BackgroundA Multi-center Study on the Reproducibility of Drug-Response Assays in Mammalian Cell Lines
Niepel, M., Hafner, M., Mills, C. E., Subramanian, K., Williams, E. H., Chung, M., Gaudio, B., Barrette, A. M., Stern, A. D., Hu, B., et al. (2019) · Cell Systems · doi:10.1016/j.cels.2019.06.005 · Cited by 133
“Data-processing routines are important for reproducibility (Sandve et al., 2013).”
Research perspectives on animal health in the era of artificial intelligence
Ezanno, P., Picault, S., Beaunée, G., Bailly, X., Muñoz, F., Duboz, R., Monod, H., & Guégan, J. (2021) · Veterinary Research · doi:10.1186/s13567-021-00902-4 · Cited by 149
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131
Ten simple rules for writing Dockerfiles for reproducible data science
Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., & Evans, B. D. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008316 · Cited by 142
“Such containers have been demonstrated for capturing scientific notebooks [5] and reproducible workflows [6].”
BackgroundPackaging Data Analytical Work Reproducibly Using R (and Friends)
Marwick, B., Boettiger, C., & Mullen, L. (2018) · The American Statistician · doi:10.1080/00031305.2017.1375986 · Cited by 108
“…need in the proliferation of commentary and how-to publications across different areas of science (Wilson et al. 2014; Stodden and Miguez 2014; Stanisic and Legrand 2014; LeVeque, Mitchell, and Stodden 2012; Donoho 2010; Sandve et al. 2013; Rokem, Marwick, and Staneva 2017; Munafo et al. 2017).”
BackgroundRe-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions
Benureau, F. C. Y., & Rougier, N. P. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00069 · Cited by 88
“But precisely because it is easy to believe that if a program runs once and gives the expected results it will do so forever, crucial steps to transform working code into meaningful scientific contributions are rarely undertaken (Schwab et al., 2000; Sandve et al., 2013; Collberg and Proebsting, 2016).”
Reproducible research and GIScience: an evaluation using AGILE conference papers
Nüst, D., Granell, C., Hofer, B., Konkol, M., Ostermann, F., Šileryté, R., & Cerutti, V. (2018) · PeerJ · doi:10.7717/peerj.5072 · Cited by 137
“A key requirement (Sandve et al., 2013) concerning methods is sharing used or developed software, where software should be published using persistent links (Stodden et al.”
BackgroundMethodGenomics Virtual Laboratory: A Practical Bioinformatics Workbench for the Cloud
Afgan, E., Sloggett, C., Goonasekera, N., Makunin, I. V., Benson, D., Crowe, M. L., Gladman, S. L., Kowsar, Y., Pheasant, M., Horst, R., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0140829 · Cited by 129
“At the most basic level reproducibility can be achieved with shell scripting and documentation, but issues in ease of use, maintenance and genuine reproducibility are well-known [19], [20].”
BackgroundMiłkowski, M., Hensel, W. M., & Hohol, M. (2018) · Journal of Computational Neuroscience · doi:10.1007/s10827-018-0702-z · Cited by 112
“A number of authors in the computational science community are drawing attention to problems with model replicability and reproducibility (Hutson 2018; Peng 2011; Rougier et al. 2017; Sandve et al. 2013).”
BackgroundMethodInfluentialEnhancing Reuse of Data and Biological Material in Medical Research: From FAIR to FAIR-Health
Holub, P., Kohlmayer, F., Prasser, F., Mayrhofer, M. T., Schlünder, I., Martin, G. M., Casati, S., Koumakis, L., Wutte, A., Kozera, Ł., et al. (2018) · Biopreservation and Biobanking · doi:10.1089/bio.2017.0110 · Cited by 102
“13 Recently, this has stimulated the proposal of countermeasures that are technically infeasible, such as an idea to store all the intermediate data used for publications 14, Rule 5 .”
Towards Personal Exposures: How Technology Is Changing Air Pollution and Health Research
Larkin, A., & Hystad, P. (2017) · Current Environmental Health Reports · doi:10.1007/s40572-017-0163-y · Cited by 107
Open and reproducible neuroimaging: From study inception to publication
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119623 · Cited by 126
The five pillars of computational reproducibility: bioinformatics and beyond
Ziemann, M., Poulain, P., & Bora, A. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad375 · Cited by 121
“To make such workflows deterministic, the pseudo-random number generator can be initialized with a fixed value (sometimes called ‘setting the seed’) [15].”
BackgroundReproducibility and replicability: opportunities and challenges for geospatial research
Kedron, P., Li, W., Fotheringham, S., & Goodchild, M. (2021) · International Journal of Geographical Information Science · doi:10.1080/13658816.2020.1802032 · Cited by 108
Metaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F., Chee, Y. E., Wintle, B. C., Burgman, M. A., McCarthy, M. A., & Gordon, A. (2017) · BioScience · doi:10.1093/biosci/biw159 · Cited by 94
Current RNA-seq methodology reporting limits reproducibility
Simoneau, J., Dumontier, S., Gosselin, R., & Scott, M. S. (2019) · Briefings in Bioinformatics · doi:10.1093/bib/bbz124 · Cited by 116
“Users can therefore publish more readily readable and reproducible code, all while following previously described rules for reproducible computational research [24].”
BackgroundInfluentialAlbanese, K. I., Barbe, S., Tagami, S., Woolfson, D. N., & Schiex, T. (2025) · Nature Reviews Methods Primers · doi:10.1038/s43586-025-00383-1 · Cited by 113
Reproducibility in Neuroimaging Analysis: Challenges and Solutions
Botvinik‐Nezer, R., & Wager, T. D. (2022) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2022.12.006 · Cited by 116
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. B., John, B. E., O'Flaherty, P., Horvath, A., & Myers, B. A. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300322 · Cited by 95
Measuring the microbiome: Best practices for developing and benchmarking microbiomics methods
Bokulich, N. A., Ziemski, M., Robeson, M. S., & Kaehler, B. D. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.11.049 · Cited by 114
Nine quick tips for pathway enrichment analysis
Chicco, D., & Agapito, G. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010348 · Cited by 107
“This step is important for you and your future analysis comparison, but also for the reproducibility of your research study [66,67].”
BackgroundGenomics pipelines and data integration: challenges and opportunities in the research setting
Davis‐Turak, J., Courtney, S. M., Hazard, E. S., Glen, W. B., da Silveira, W. A., Wesselman, T., Harbin, L. P., Wolf, B. J., Chung, D., & Hardiman, G. (2017) · Expert Review of Molecular Diagnostics · doi:10.1080/14737159.2017.1282822 · Cited by 112
“Although the scientific community has long recognized the requirements for improving the reproducibility of computational research [10], these fundamentals of computer science are often eschewed for the more enticing challenge of developing novel algorithms.”
BackgroundTen simple rules for conducting a mendelian randomization study
Taliun, S. A. G., & Evans, D. M. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009238 · Cited by 111
“PLOS Computational Biology | https://doi.org/10.1371/journal.pcbi.1009238 August 12, 2021 5 / 8”
BackgroundInfluentialKhan, F. Z., Soiland‐Reyes, S., Sinnott, R., Lonie, A. J., Goble, C., & Crusoe, M. R. (2019) · GigaScience · doi:10.1093/gigascience/giz095 · Cited by 107
“Record the exact software versions used [57, 59].”
InfluentialProgress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95
Giannoulatou, E., Park, S. H., Humphreys, D. T., & Ho, J. W. K. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-s16-s15 · Cited by 108
“Most importantly, there have been numerous published papers that attempt to train scientists to adopt best practices for scientific computing [13-17].”
BackgroundProceedings of the 15th Python in Science Conference
(2016) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-629e541a-014 · Cited by 115
Containers for computational reproducibility
Moreau, D., Wiebels, K., & Boettiger, C. (2023) · Nature Reviews Methods Primers · doi:10.1038/s43586-023-00236-9 · Cited by 121
“Providing reproducible content is a fundamental aspect of scientific research and has been discussed at length elsewhere, either generally 23 or in the context of containerization 102,103 .”
BackgroundTen Simple Rules for Developing Usable Software in Computational Biology
List, M., Ebert, P., & Albrecht, F. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005265 · Cited by 83
Methodological Principles for Reproducible Performance Evaluation in Cloud Computing
Papadopoulos, A. V., Versluis, L., Bauer, A., Herbst, N., Kistowski, J. V., Ali-Eldin, A., Abad, C. L., Amaral, J. N., Tuma, P., & Iosup, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2927908 · Cited by 93
“Poor experimental design and/or execution are often the cause for many pitfalls encountered by well-meaning researchers and practitioners [1], [2].”
BackgroundKonkol, M., Kray, C., & Pfeiffer, M. (2019) · International Journal of Geographical Information Science · doi:10.1080/13658816.2018.1508687 · Cited by 74
“Moreover, fi gures should be created by using scripts instead of ready-to-use toolboxes, which hide the computational steps (Sandve et al .”
BackgroundMethodA Beginner's Guide to Conducting Reproducible Research
Alston, J. M., & Rick, J. A. (2021) · Bulletin of the Ecological Society of America · doi:10.1002/bes2.1801 · Cited by 91
Big data, integrative omics and network biology
Tolani, P., Gupta, S., Yadav, K., Aggarwal, S., & Yadav, A. K. (2021) · Advances in protein chemistry and structural biology · doi:10.1016/bs.apcsb.2021.03.006 · Cited by 108
Methods That Support the Validation of Agent-Based Models: An Overview and Discussion
Collins, A. J., Koehler, M., & Lynch, C. J. (2024) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.5258 · Cited by 106
“Forreproducibility,andassessingstatisticalvaliditywhenapplicable,therawdataand anyintermediarydatamanipulation,suchashistogrambinning,behindeachvisualizationshouldbestored (Sandveetal.2013).”
Eleven quick tips for data cleaning and feature engineering
Chicco, D., Oneto, L., & Tavazzi, E. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010718 · Cited by 94
“As Sandve and colleagues [222] wrote: “For every result, keep track of how it was produced.””
BackgroundPractical Reproducibility in Geography and Geosciences
Nüst, D., & Pebesma, E. J. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806028 · Cited by 85
“When studies are impossible to replicate for conceptual or practical reasons, reproducibility is the only way we can ensure that a scientific claim can be evaluated, and it becomes a minimal standard (Peng 2011; Sandve et al. 2013).”
BackgroundExperimenting with reproducibility: a case study of robustness in bioinformatics
Kim, Y.-M., Poline, J.-B., & Dumas, G. (2018) · GigaScience · doi:10.1093/gigascience/giy077 · Cited by 76
“…by suggesting tools and best practices following the programming best practices of Wilson et al., such as modularizing and re-using code, unit testing, document design, data management, and project organization [2, 27], as well as keeping data provenance and recording all intermediate results [28].”
Reproducible Research: A Retrospective
Peng, R. D., & Hicks, S. C. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-012420-105110 · Cited by 64
Data Sharing: Convert Challenges into Opportunities
Figueiredo, A. S. (2017) · Frontiers in Public Health · doi:10.3389/fpubh.2017.00327 · Cited by 85
“Make certain that the computer code is efficient and reproducible by following the ten simple rules for reproducible computational research laid out by Sandve et al. (31).”
MethodBenchmarking the translational potential of spatial gene expression prediction from histology
Wang, C., Chan, A. S., Fu, X., Ghazanfar, S., Kim, J., Patrick, E., & Yang, J. Y. H. (2025) · Nature Communications · doi:10.1038/s41467-025-56618-y · Cited by 95
Lemmon, V., Ferguson, A. R., Popovich, P. G., Xu, X., Snow, D. M., Igarashi, M., Beattie, C. E., & Bixby, J. L. (2014) · Journal of Neurotrauma · doi:10.1089/neu.2014.3400 · Cited by 81
Formalizing atom-typing and the dissemination of force fields with foyer
Klein, C., Summers, A. Z., Thompson, M. W., Gilmer, J., McCabe, C., Cummings, P. T., Sallai, J., & Iacovella, C. R. (2019) · Computational Materials Science · doi:10.1016/j.commatsci.2019.05.026 · Cited by 78
Creating and sharing reproducible research code the workflowr way
Blischak, J. D., Carbonetto, P., & Stephens, M. (2019) · F1000Research · doi:10.12688/f1000research.20843.1 · Cited by 77
Metadata
- Authors
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E.
- Affiliations
- University of Oslo (NO)
- Pennsylvania State University (US)
- Emory University (US)
- Oslo University Hospital (NO)
- Corresponding author
- Sandve, Geir Kjetil
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-24
- Volume
- 9
- Issue
- 10
- Pages
- e1003285
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3812051
- OpenAlex
- W2036318837
- Cited by
- 880
- References
- 27
- Citations by year
- 2026: 176 · 2025: 60 · 2024: 52 · 2023: 50 · 2022: 58 · 2021: 67 · 2020: 63 · 2019: 88 · 2018: 76 · 2017: 64 · 2016: 56 · 2015: 42 · 2014: 24 · 2013: 3
- FWCI
- 91.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (0850103)
- Pennsylvania Department of Health (HG004909)
- University of Pennsylvania
- Pennsylvania State University
- National Institutes of Health (HG006620)
- Huck Institutes of the Life Sciences
- Institute for Computational and Data Sciences, Pennsylvania State University
- National Human Genome Research Institute (HG005133, U41HG006620, R21HG005133, R01HG004909, HG006620, HG004909)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducible research, computational reproducibility, research practices, bioinformatics analysis, research transparency, version control, software dependency management, workflow management, research data management, research culture, research evaluation, research dissemination, scholarly communication
- MeSH terms
- Reproducibility of Results, Computational Biology, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.