Ten Simple Rules for Reproducible Computational Research
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Replication is the cornerstone of a cumulative science [1]. However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2]. As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3]. This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4]. The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8], [9]. This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3]. We want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run. We further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities. In a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys. As a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible. With this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6], [14]. We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your researc
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival1,11,12, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
From The FAIR Guiding Principles for scientific data management and stewardship (2016) · cites this paper as [12]
In the citing paper
“In a recent publication reviewing the automated citation screening methods reported for systematic reviews, the number of studies that met the current standards for reproducible software engineering experiments was low [29, 30, 36]; of 33 studies reporting approaches to automated screening, no study provided all the criteria for reproducible reporting [36].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundMinimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundGoecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome Biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,453 · cited 1×
“For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19], the GenePattern Word add-in [4], and Galaxy Pages [20].”
MethodSelf-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundReich, M., Liefeld, T., Gould, J., Lerner, J., Tamayo, P., & Mesirov, J. P. (2006) · Nature Genetics · doi:10.1038/ng0506-500 · Cited by 2,032
Galaxy: A platform for interactive large-scale genome analysis
Giardine, B., Riemer, C., Hardison, R. C., Burhans, R., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,026 · cited 2×
“If you are instead performing your analyses through an integrated framework (such as Gene-Pattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules.”
BackgroundSelf-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“…through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8,9].”
ResultTaverna: a tool for the composition and enactment of bioinformatics workflows
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M. R., Wipat, A., et al. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth361 · Cited by 1,602 · cited 2×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 3×
“In order to foster such aspects, it has been suggested that the scientific community needs to develop a ‘‘culture of reproducibility’’ for computational science, and to require it for published claims [3].”
BackgroundPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222 · cited 1×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
Background
Cited by
Items 301–400 of 1,001
Advancing Cardiovascular Research With Single-Cell and Spatial Transcriptomics
Li, K., Samiei, S., Pikulska, D., Foecking, S., & Kuppe, C. (2026) · Circulation Research · doi:10.1161/circresaha.125.325795 · Cited by 19
Enhanced reproducibility of SADI web service workflows with Galaxy and Docker
Aranguren, M. E., & Wilkinson, M. D. (2015) · GigaScience · doi:10.1186/s13742-015-0092-3 · Cited by 16
“Computational reproducibility is becoming an important consideration in the life sciences [26, 27].”
BackgroundMolecular Simulation Meets Machine Learning
Sadus, R. J. (2023) · Journal of Chemical & Engineering Data · doi:10.1021/acs.jced.3c00553 · Cited by 16
Liu, L., Wang, F., Fu, X., Kötter, T., Sturm, K., Guan, W. W., & Bao, S. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103948 · Cited by 16
A Framework for Global Collaborative Data Management for Malaria Research
Gutiérrez, J. B., Harb, O. S., Zheng, J., Tisch, D. J., Charlebois, E. D., Stoeckert, C. J., & Sullivan, S. A. (2015) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.15-0003 · Cited by 17
Costa, R. L., Gadelha, L. M. R., Jr., Alves, M. R., & Porto, F. A. M. (2017) · PeerJ · doi:10.7717/peerj.3509 · Cited by 18
“, 2014), used in these activities to support validation by peers (Sandve et al., 2013).”
BackgroundMethodDocumentation strategy for facilitating the reproducibility of geo-simulation experiments
Zhu, Z., Chen, M., Qian, Z., Li, H., Wu, K., Ma, Z., Wen, Y., Yue, S., & Lü, G. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105687 · Cited by 16
Towards integration of data-driven agronomic experiments with data provenance
da Cruz, S. M. S., & do Nascimento, J. A. P. (2019) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2019.01.044 · Cited by 16
The Data Preparation Process in Real Estate: Guidance and Review
Krause, A., & Lipscomb, C. A. (2016) · Journal of Real Estate Practice and Education · doi:10.1080/10835547.2016.12091756 · Cited by 16
Hatton, L., & Warr, G. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.06897 · Cited by 16
“In this sense we will go much further than detailing guidelines for good practice [41] by providing an actual example, although this naturally embodies a number of these guidelines.”
BackgroundHorton, N. J., Alexander, R., Parker, M. S., Piekut, A., & Rundel, C. W. (2022) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2022.2141001 · Cited by 16
“Many instructors incorporate their research in teaching and research-led teaching (Schapper and Mayson 2010).”
BackgroundSilva, L. L., Silvola, R. M., Haas, D. M., & Quinney, S. K. (2022) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15018 · Cited by 15
“In fact, identical results should be obtained if the same initial seed values are used.(32) Kirouac et al.”
A how-to guide for code sharing in biology
Abdill, R. J., Talarico, E., & Grieneisen, L. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002815 · Cited by 17
“In a general sense, code that is “open” is freely available for inspection, but open-source advocates frequently incorporate additional licensing considerations when deciding whether software is “open,” such as free redistribution of the software [30].”
BackgroundGene signatures for cancer research: A 25-year retrospective and future avenues
Liu, W., He, H., & Chicco, D. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012512 · Cited by 15
An architecture for genomics analysis in a clinical setting using Galaxy and Docker
Digan, W. E., Countouris, H., Barritault, M., Baudoin, D., Laurent‐Puig, P., Blons, H., Burgun, A., & Rance, B. (2017) · GigaScience · doi:10.1093/gigascience/gix099 · Cited by 15
“[9] have identified good practices that would contribute to ensuring research quality.”
BackgroundTen simple rules for researchers who want to develop web apps
Saia, S. M., Nelson, N. G., Young, S. N., Parham, S., & Vandegrift, M. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009663 · Cited by 15
Cwlprov - Interoperable Retrospective Provenance Capture And Its Challenges
Khan, F. Z., Soiland‐Reyes, S., Sinnott, R., Lonie, A. J., Goble, C., & Crusoe, M. R. (2018) · Research Explorer (The University of Manchester) · doi:10.5281/zenodo.1208477 · Cited by 15
Rosenberg, D., & Horn, C. C. (2016) · Journal of Neurophysiology · doi:10.1152/jn.00137.2016 · Cited by 15
Ze-HB Hierarchical Bayesian Extension of the ZePreprint
Tqemaladze, J. (2025) · Preprints.org · doi:10.20944/preprints202512.0103.v1 · Cited by 15
Rosignoli, S., Lustrino, E., Shevchuk, O., Rinaldo, S., Rubini, E., Paiardini, A., & Carev, I. (2026) · Biomolecules · doi:10.3390/biom16020197 · Cited by 16
“General guidelines for reproducible computational research emphasize that reliability critically depends on data quality, transparent reporting of work-flows, and explicit documentation of algorithmic parameters [203,204].”
Picault, S., Huang, Y.-L., Sicard, V., & Ezanno, P. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/53 · Cited by 14
Cyber Security and Privacy Experiments: A Design and Reporting Toolkit
Coopamootoo, K., & Groß, T. (2018) · IFIP advances in information and communication technology · doi:10.1007/978-3-319-92925-5_17 · Cited by 15
“…planned analyses, which involves the documentation of the plan as well as the recording of all the analyses done, we suggest inspiration from reproducibility principles from general computing science research [41] or more specific sources with focus on computation-supported sci-entific practice [45].”
pyKNEEr: An image analysis workflow for open and reproducible research on femoral knee cartilage
Bonaretti, S., Gold, G. E., & Beaupré, G. S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0226501 · Cited by 16
“Openness and reproducibility are essential to researchers to assess the accuracy of scientific claims [5], build on the work of other scientists with confidence and efficiency (i.”
BackgroundMethodA PLACE FOR A DATA SCIENCE PROJECT IN SCHOOL: BETWEEN STATISTICS AND EPISTEMIC PROGRAMMING
Podworny, S., Hüsing, S., & Schulte, C. (2022) · Statistics Education Research Journal · doi:10.52041/serj.v21i2.46 · Cited by 14
Fostering Better Coding Practices for Data Scientists
Pruim, R. J., Gîrjău, M.-C., & Horton, N. J. (2023) · Harvard Data Science Review · doi:10.1162/99608f92.97c9f60f · Cited by 19
“Sandve et al. (2013) offers a set of rules for reproducible computations research.”
BackgroundMethodHasham, K., Munir, K., & McClatchey, R. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2017.07.015 · Cited by 14
Jansen, C., Annuscheit, J., Schilling, B., Strohmenger, K., Witt, M., Bartusch, F., Herta, C., Hufnagl, P., & Krefting, D. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.05.007 · Cited by 16
Text mining in business libraries
Anderson, C. B., & Craiglow, H. A. (2017) · Journal of Business & Finance Librarianship · doi:10.1080/08963568.2017.1285749 · Cited by 14
“See Sandve, Nekrutenko, Taylor, and Hovig (2013) for guidelines on making computational research reproducible.”
BackgroundStanding Together for Reproducibility in Large-Scale Computing: Report on reproducibility@XSEDEPreprint
James, D., Wilkins‐Diehr, N., Stodden, V., Colbry, D., Rosales, C., Fahey, M. R., Shi, J. Y., da Silva, R. F., Lee, K., Roskies, R. Z., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.5557 · Cited by 14
“Finally, we recognize that there is little controversy regarding best practices for individuals and small research teams: e.g. change management and automation [28].”
BackgroundSelf-citationPeters, K., Gorzolka, K., Bruelheide, H., & Neumann, S. (2018) · Scientific Data · doi:10.1038/sdata.2018.179 · Cited by 14
Computational reproducibility of Jupyter notebooks from biomedical publicationsPreprint
Samuel, S., & Mietchen, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.04308 · Cited by 16
FAIR Data and Software: Improving Efficiency and Quality of Biocatalytic Science
Pleiss, J. (2024) · ACS Catalysis · doi:10.1021/acscatal.3c06337 · Cited by 13
Ten simple rules for scientific code review
Rokem, A. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012375 · Cited by 13
Resilience Metrics for Socio-Ecological and Socio-Technical Systems: A Scoping Review
Steinmann, P., Tobi, H., & van Voorn, G. A. K. (2024) · Systems · doi:10.3390/systems12090357 · Cited by 13
The governance of open science: A comparative analysis of two open science consortia
Abrams, E., Leone, P. V., Cambrosio, A., & Faraj, S. A. (2025) · Research Policy · doi:10.1016/j.respol.2025.105195 · Cited by 13
Marwick, B. (2025) · Journal of Archaeological Science · doi:10.1016/j.jas.2025.106281 · Cited by 14
“…should provide their materials organised such that a reader can successfully run all code as-is, without making any manual modifications (e.g. use relative rather than absolute file paths so that readers don ’ t have to rename or move files around to make the code work) (Sandve et al., 2013).”
Continuous Reproducibility in GNSS Signal Processing
Fernández–Prades, C., Vilà‐Valls, J., Arribas, J., & Ramos, A. (2018) · IEEE Access · doi:10.1109/access.2018.2822835 · Cited by 15
“…requires the observance of a set of best practices, which should be considered from the inception of the research activity, in order to permit the effective communications not only of the research results, but the tools and data required to verify and extend the generated knowledge [6]–[9].”
BackgroundAn Improved Convolutional Neural Network Based on a Parameter Modification of the Convolution Layer
Rodríguez, R., González, C. I., Martínez, G. E., & Melín, P. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-68776-2_8 · Cited by 13
Meeting in the Middle: Towards Successful Multidisciplinary Bioimage Analysis Collaboration
Schlaeppi, A., Adams, W. R., Haase, R., Huisken, J., MacDonald, R. B., Eliceiri, K. W., & Kugler, E. (2022) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2022.889755 · Cited by 14
Recent Advances in Procedural Generation of Buildings: From Diversity to Integration
Kutzias, D., & von Mammen, S. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2023.3262507 · Cited by 14
“Regarding the importance of source code, it has been claimed that all scripts and parameters have to be published to ensure scientific reproducibility [74].”
BackgroundMonte Carlo Null Models for Genomic Data
Ferkingstad, E., Holden, L., & Sandve, G. K. (2015) · Statistical Science · doi:10.1214/14-sts484 · Cited by 14
“All calculations were performed using the R programming language (R Development Core Team, 2011) and the Genomic HyperBrowser.”
BackgroundMethodSelf-citationVisualising three-dimensional genome organisation in two dimensions
Ing‐Simmons, E., & Vaquerizas, J. M. (2019) · Development · doi:10.1242/dev.177162 · Cited by 13
“Interactive data exploration is also difficult to reproduce, a consideration that is becoming more important as reproducibility is seen as a tool for increasing confidence in scientific results (Peng, 2011; Sandve et al., 2013).”
BackgroundTRACTUS: Understanding and Supporting Source Code Experimentation in Hypothesis-Driven Data Science
Subramanian, K., Maas, J., & Borchers, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376764 · Cited by 13
“Additionally, data scientists document their code as well as insights obtained during their work and rationale that justifies their analysis methods [1, 29].”
BackgroundDeardorff, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230697 · Cited by 13
Bodner, K., Brimacombe, C., Chenery, E. S., Greiner, A., McLeod, A. M., Penk, S. R., & Soto, J. S. V. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008539 · Cited by 13
“For more details on learning to program, creating reproducible research, programming with Jupyter notebooks, and using Git and Github, see the 10 simple rules by Carey and Papin [13], Sandve and colleagues [14], Rule and colleagues [15], and Perez-Riverol and colleagues [16], respectively.”
Vo, T. H., & Le, N. Q. K. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag227 · Cited by 15
A very simple and fast way to access and validate algorithms in reproducible research
Stegmayer, G. S., Pividori, M., & Milone, D. H. (2015) · Briefings in Bioinformatics · doi:10.1093/bib/bbv054 · Cited by 12
Lerma‐Usabiaga, G., Liu, M., Paz‐Alonso, P. M., & Wandell, B. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-32924-7 · Cited by 12
Recommendations for the FAIRification of genomic track metadata
Gundersen, S., Boddu, S., Capella-Gutiérrez, S., Drabløs, F., Fernández, J. M., Kompova, R., Taylor, K., Titov, D., Zerbino, D. R., & Hovig, E. J. (2021) · F1000Research · doi:10.12688/f1000research.28449.1 · Cited by 13
Self-citationOpinionated Practices for Teaching Reproducibility: Motivation, Guided Instruction and Practice
Östblom, J., & Timbers, T. (2022) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2022.2074922 · Cited by 16
“While there are many articles outlining recommendations on which reproducible practices to adhere to (Sandve et al. 2013; Wilson et al. 2017; Lowndes et al. 2017), case studies of failures are not as frequent in the literature.”
BackgroundDocumenting research software in engineering science
Hermann, S., & Fehr, J. (2022) · Scientific Reports · doi:10.1038/s41598-022-10376-9 · Cited by 18
“x x Sandve GK, Nekrutenko A, et al.(5) x x Karimzadeh M, Hoffman MM.”
BackgroundInfluentialToward Enabling Reproducibility for Data-Intensive Research Using the Whole Tale Platform
Kyle, C., Niall, G., Hategan, M., Kacper, K., Bertram, L., McPhillips, T., Jarek, N., Stodden, V., Taylor, I., Thomas, T., et al. (2020) · Advances in parallel computing · doi:10.3233/apc200107 · Cited by 14
“producible research objects that capture artifacts such as data, software, and sufficient details from a computational experiment to enable reproducibility, transparency, and re-execution is a challenge the research community is addressing in a variety of ways [2,3,4,5,6,7,8,9,10].”
BackgroundHOW TO MAKE A PIE: REPRODUCIBLE RESEARCH FOR EMPIRICAL ECONOMICS AND ECONOMETRICS
Orozco, V., Bontemps, C., Maigné, É., Piguet, V., Hofstetter, A., Lacroix, A., Levert, F., & Rousselle, J.-M. (2020) · Journal of Economic Surveys · doi:10.1111/joes.12389 · Cited by 14
Comparing time series transcriptome data between plants using a network module finding algorithm
Lee, J., Heath, L. S., Grene, R., & Li, S. (2019) · Plant Methods · doi:10.1186/s13007-019-0440-x · Cited by 13
GenomicDataCommons: a Bioconductor Interface to the NCI Genomic Data CommonsPreprint
Morgan, M. T., & Davis, S. R. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/117200 · Cited by 13
Ten simple rules for building and maintaining a responsible data science workflow
Stoudt, S., Jernite, Y., Marshall, B., Marwick, B., Sharan, M., Whitaker, K., & Danchev, V. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1012232 · Cited by 11
“You can also integrate your computational notebooks with version-control software such as Git, GitHub, and GitLab [72,83] so that others can access your code and its history, and possibly collaborate and/or reproduce your research.”
BackgroundTwelve quick tips for AI-assisted coding in science
Bridgeford, E., Campbell, I. D., Chen, Z., Lin, Z., Ritz, H., Vandekerckhove, J., & Poldrack, R. A. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014428 · Cited by 11
“These worked examples with detailed analysis will reinforce readers’ understanding of what good and flawed AI interactions look like, and put the strategies discussed herein into practice.”
regionReport: Interactive reports for region-based analyses
Collado‐Torres, L., Jaffe, A. E., & Leek, J. T. (2015) · F1000Research · doi:10.12688/f1000research.6379.1 · Cited by 11
Inferring ASF transmission in domestic pigs and wild boars using a paired model iterative approach
Beaunée, G., Deslandes, F., & Vergu, E. (2023) · Epidemics · doi:10.1016/j.epidem.2023.100665 · Cited by 11
Scharm, M., & Waltemath, D. (2016) · F1000Research · doi:10.12688/f1000research.9379.1 · Cited by 12
“Repeated mentions of problems with replication and reproducibility 2–4 led to new standards, tools, and methods for the transfer of reproducible simulation studies 5–9 .”
MethodFifteen quick tips for success with HPC, i.e., responsibly BASHing that Linux cluster
Alnasir, J. J. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009207 · Cited by 11
“See the Ten Simple Rules for Reproducible Computational Research article for some general best practice principles that can be applied when developing and running analysis workflows and computational pipelines [17].”
BackgroundHPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction
Mozaffari, A., Langguth, M., Gong, B., Ahring, J., Campos, A. R., Nieters, P., Escobar, O. J. C., Wittenbrink, M., Baumann, P., & Schultz, M. G. (2022) · Data Intelligence · doi:10.1162/dint_a_00131 · Cited by 12
“It can rise to a level where reusing previous work becomes a routine practice and leads to an increase in productivity, work habit, and continuity [5, 6, 7, 8].”
BackgroundSufi, S., Hong, N. C., Hettrick, S., Antonioletti, M., Crouch, S., Hay, A., Inupakutika, D., Jackson, M. R., Pawlik, A., Peru, G., et al. (2014) · doi:10.1145/2618137.2618140 · Cited by 11
cwepr – A Python package for analysing cw-EPR data focussing on reproducibility and simple usage
Schröder, M., & Biskup, T. (2021) · Journal of Magnetic Resonance · doi:10.1016/j.jmr.2021.107140 · Cited by 11
Geiger, R. S., Sholler, D., Culich, A., Martinez, C. C., de la Guardia, F. H., Lanusse, F., Ottoboni, K., Stuart, M. A., Vareth, M., Varoquaux, N., et al. (2018) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/a7b3m · Cited by 11
A materials data framework and dataset for elastomeric foam impact mitigating materials
Landauer, A. K., Kafka, O. L., Moser, N. H., Foster, I. T., Blaiszik, B. J., & Forster, A. M. (2023) · Scientific Data · doi:10.1038/s41597-023-02092-4 · Cited by 13
Liu, D. M., & Salganik, M. (2019) · Sage Journals Data · doi:10.25384/sage.c.4663025 · Cited by 11
Ten simple rules for managing laboratory information
Berezin, C.-T., Aguilera, L. U., Billerbeck, S., Bourne, P. E., Densmore, D. M., Freemont, P. S., Gorochowski, T. E., Hernandez, S. I., Hillson, N. J., King, C. R., et al. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011652 · Cited by 10
“This journal has published several sets of rules regarding best practices in overall research design [9,10], as well as the computational parts of research workflows, including data management [11–13] and software development practices [14–16].”
MethodCronin, L., Taylor, C. M., Briciu-Burghina, C., Lucy, F., & Regan, F. B. (2025) · MethodsX · doi:10.1016/j.mex.2025.103538 · Cited by 10
“To minimize issues with non-reproducibility, the workflow in this research followed recommendations for research reproducibility using code [17,13,14] and a detailed user guide consisting of code alongside the output of analysis was developed.”
Chicco, D., Sabino, G., Oneto, L., & Jurman, G. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3095 · Cited by 12
“In this study, we decided to employ only density-based clustering coef fi cients that are available as open source packages in Python or R, following the principles of open science for reproducibility ( Sandve et al., 2013 ).”
Lyimo, B. M. (2026) · Bioinformatics and Biology Insights · doi:10.1177/11779322261427123 · Cited by 11
“50 To ensure sustainable impact, AI-assisted coding and AMR analytics should be positioned as capacity-building tools rather than substitutes for domain expertise.”
GeenaR: A Web Tool for Reproducible MALDI-TOF Analysis
Prete, E. D., Facchiano, A. M., Profumo, A., Angelini, C., & Romano, P. D. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.635814 · Cited by 10
“The work of Sandve et al. (2013) provides golden rules to obtain a (computationally) reproducible research.”
BackgroundReproducible Research in Document Analysis and Recognition
Cacho, J. R. F., & Taghva, K. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-77028-4_51 · Cited by 10
Visualizing alpha-helical peptides in R with helixvis
Wadhwa, R., Subramanian, V., & Stevens‐Truss, R. (2018) · The Journal of Open Source Software · doi:10.21105/joss.01008 · Cited by 10
Sandhu, V., Labori, K. J., Borgida, A., Lungu, I. M., Bartlett, J. M. S., Hafezi‐Bakhtiari, S., Denroche, R. E., Jang, G. H., Pasternack, D., Mbabaali, F., et al. (2019) · JCO Clinical Cancer Informatics · doi:10.1200/cci.18.00102 · Cited by 11
Safikhani, Z., El-Hachem, N., Smirnov, P., Freeman, M., Goldenberg, A., Birkbak, N. J., Beck, A. H., Aerts, H. J. W. L., Quackenbush, J. F., & Haibe‐Kains, B. (2016) · Nature · doi:10.1038/nature20172 · Cited by 10
Guru, S. M., Hanigan, I., Nguyen, H. A., Burns, E., Stein, J., Blanchard, W., Lindenmayer, D. B., & Clancy, T. (2016) · Ecological Informatics · doi:10.1016/j.ecoinf.2016.08.003 · Cited by 10
“To address these challenges, researchers have recommended best practices for performing computational science (Sandve et al., 2013; Stodden and Miguez, 2014) (Wilson et al.”
BackgroundMethodBISTRO: Berkeley Integrated System for Transportation Optimization
Feygin, S., Lazarus, J. R., Forscher, E. H., Golfier-Vetterli, V., Lee, J., Gupta, A., Waraich, R. A., Sheppard, C., & Bayen, A. M. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3384344 · Cited by 10
Safikhani, Z., El-Hachem, N., Smirnov, P., Freeman, M., Goldenberg, A., Birkbak, N. J., Beck, A. H., Aerts, H. J. W. L., Quackenbush, J. F., & Haibe‐Kains, B. (2016) · Nature · doi:10.1038/nature20581 · Cited by 10
Popp, J., & Biskup, T. (2022) · Chemistry - Methods · doi:10.1002/cmtd.202100097 · Cited by 12
Encouraging Reproducibility in Scientific Research of the Internet (Dagstuhl Seminar 18412)
Bajpai, V. K., Bonaventure, O., claffy, K., & Karrenberg, D. (2019) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dagrep.8.10.41 · Cited by 10
Cascabel : a flexible, scalable and easy-to-use amplicon sequence data analysis pipelinePreprint
Asbun, A. A., Besseling, M., Balzano, S., van Bleijswijk, J. D. L., Witte, H. J., Villanueva, L., & Engelmann, J. C. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/809384 · Cited by 10
Data-science ready, multisite, human diffusion MRI white-matter-tract statistics
Lerma‐Usabiaga, G., Mukherjee, P., Perry, L. M., & Wandell, B. A. (2020) · Scientific Data · doi:10.1038/s41597-020-00760-3 · Cited by 10
A seven‐step guide to spatial, agent‐based modelling of tumour evolution
Colyer, B., Bąk, M., Basanta, D., & Noble, R. (2024) · Evolutionary Applications · doi:10.1111/eva.13687 · Cited by 12
“Robertson-Tessi et al. (2015) pioneered the use of these plots in cancer research in their 2015 study of the effects of metabolic heterogeneity on tumour growth.”
BackgroundSimpleMetaPipeline : Breaking the bioinformatics bottleneck in metabarcoding
Williams, J., Pettorelli, N., Dowell, R., Macdonald, K. S., Meyer, C., Steyaert, M., Tweedt, S. M., & Ransome, E. J. (2024) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14434 · Cited by 9
“…2019 ), that is, the extent to which the integration of different tools with a variety of native formats is facilitated; and (ii) reproducibility (Sandve et al., 2013 ; Powers & Hampton, 2019; Wratten et al., 2021 ), in general, the ease and reliability with which one can re-generate identical…”
Formal definition and implementation of reproducibility tenets for computational workflows
Pritchard, N., & Wicenec, A. J. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.107684 · Cited by 11
Wood, Z. T., Wiegardt, A. K., Barton, K. L., Clark, J. D., Homola, J. J., Olsen, B. J., King, B. L., Kovach, A. I., & Kinnison, M. T. (2021) · Evolutionary Applications · doi:10.1111/eva.13264 · Cited by 16
Docker4Circ: A Framework for the Reproducible Characterization of circRNAs from RNA-Seq Data
Ferrero, G., Licheri, N., Tarrero, L. C., Intinis, C. D., Miano, V., Calogero, R., Cordero, F., Bortoli, M. D., & Beccuti, M. (2019) · International Journal of Molecular Sciences · doi:10.3390/ijms21010293 · Cited by 9
“Docker4Circ consists of an R library (integrated into Docker4Seq package [23]) and a set of docker images.”
MethodStreamingHub: interactive stream analysis workflows
Jayawardana, Y., Ashok, V. G., & Jayarathna, S. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530936 · Cited by 9
“Naturally, this has led to discussions on how researchers, institutions, funding bodies, and journals can establish guidelines to increase the reusability and verifiability of research assets [54].”
BackgroundExecutable Simulation Model of the Liver
König, M. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-801238-3.11682-4 · Cited by 9
macroeco: reproducible ecological pattern analysis in Python
Kitzes, J. A., & Wilber, M. (2015) · Ecography · doi:10.1111/ecog.01905 · Cited by 9
Understanding computational web archives research methods using research objects
Maemura, E., Becker, C., & Milligan, I. (2016) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2016.7840982 · Cited by 9
EnosStack: A LAMP-like stack for the experimenter
Cherrueau, R.-A., Simonin, M., & van Kempen, A. (2018) · IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infcomw.2018.8407024 · Cited by 10
“Even if experimenters are committed to automate their experiment code, some tasks remain tedious [17].”
BackgroundPre-statistical harmonization of behavioral instruments across eight surveys and trials
Chen, D., Jutkowitz, E., Iosepovici, S. L., Lin, J. C. H., & Gross, A. L. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01431-6 · Cited by 9
HydroBench: Jupyter supported reproducible hydrological model benchmarking and diagnostic tool
Moges, E., Ruddell, B. L., Zhang, L., Driscoll, J. M., Norton, P. A., Pérez, F., & Larsen, L. G. (2022) · Frontiers in Earth Science · doi:10.3389/feart.2022.884766 · Cited by 9
Ten ways to fool the masses with machine learningPreprint
Minhas, F., Asif, A., & Ben‐Hur, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01686 · Cited by 9
“Sandve et al. have recently given a list of points on how to ensure reproducibility in computational research [24].”
BackgroundGood Enough Practices in Scientific Computing
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005510 · Cited by 8
Doncheva, N. T., Schwämmle, V., & Locard‐Paulet, M. (2025) · Journal of Proteome Research · doi:10.1021/acs.jproteome.5c00287 · Cited by 8
Heather, A., Monks, T., Harper, A. L., Mustafee, N., & Mayne, A. (2025) · Journal of Simulation · doi:10.1080/17477778.2025.2552177 · Cited by 8
“[6,7] For instance, code may need to be re-run following peer review.”
Oliveira, M., Monteiro, E., & Rouboa, A. I. (2026) · Energy Conversion and Management · doi:10.1016/j.enconman.2026.121309 · Cited by 9
Metadata
- Authors
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E.
- Affiliations
- University of Oslo (NO)
- Pennsylvania State University (US)
- Emory University (US)
- Oslo University Hospital (NO)
- Corresponding author
- Sandve, Geir Kjetil
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-24
- Volume
- 9
- Issue
- 10
- Pages
- e1003285
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3812051
- OpenAlex
- W2036318837
- Cited by
- 880
- References
- 27
- Citations by year
- 2026: 176 · 2025: 60 · 2024: 52 · 2023: 50 · 2022: 58 · 2021: 67 · 2020: 63 · 2019: 88 · 2018: 76 · 2017: 64 · 2016: 56 · 2015: 42 · 2014: 24 · 2013: 3
- FWCI
- 91.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (0850103)
- Pennsylvania Department of Health (HG004909)
- University of Pennsylvania
- Pennsylvania State University
- National Institutes of Health (HG006620)
- Huck Institutes of the Life Sciences
- Institute for Computational and Data Sciences, Pennsylvania State University
- National Human Genome Research Institute (HG005133, U41HG006620, R21HG005133, R01HG004909, HG006620, HG004909)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducible research, computational reproducibility, research practices, bioinformatics analysis, research transparency, version control, software dependency management, workflow management, research data management, research culture, research evaluation, research dissemination, scholarly communication
- MeSH terms
- Reproducibility of Results, Computational Biology, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.