Ten Simple Rules for Reproducible Computational Research
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Replication is the cornerstone of a cumulative science [1]. However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2]. As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3]. This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4]. The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8], [9]. This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3]. We want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run. We further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities. In a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys. As a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible. With this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6], [14]. We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your researc
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival1,11,12, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
From The FAIR Guiding Principles for scientific data management and stewardship (2016) · cites this paper as [12]
In the citing paper
“In a recent publication reviewing the automated citation screening methods reported for systematic reviews, the number of studies that met the current standards for reproducible software engineering experiments was low [29, 30, 36]; of 33 studies reporting approaches to automated screening, no study provided all the criteria for reproducible reporting [36].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundMinimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundGoecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome Biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,453 · cited 1×
“For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19], the GenePattern Word add-in [4], and Galaxy Pages [20].”
MethodSelf-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundReich, M., Liefeld, T., Gould, J., Lerner, J., Tamayo, P., & Mesirov, J. P. (2006) · Nature Genetics · doi:10.1038/ng0506-500 · Cited by 2,032
Galaxy: A platform for interactive large-scale genome analysis
Giardine, B., Riemer, C., Hardison, R. C., Burhans, R., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,026 · cited 2×
“If you are instead performing your analyses through an integrated framework (such as Gene-Pattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules.”
BackgroundSelf-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“…through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8,9].”
ResultTaverna: a tool for the composition and enactment of bioinformatics workflows
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M. R., Wipat, A., et al. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth361 · Cited by 1,602 · cited 2×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 3×
“In order to foster such aspects, it has been suggested that the scientific community needs to develop a ‘‘culture of reproducibility’’ for computational science, and to require it for published claims [3].”
BackgroundPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222 · cited 1×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
Background
Cited by
Items 401–500 of 1,001
Guidelines for reproducible analysis of adaptive immune receptor repertoire sequencing data
Peres, A., Klein, V., Frankel, B., Lees, W., Polak, P., Meehan, M., Rocha, A., Lopes, J. C., & Yaari, G. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae221 · Cited by 8
“In this context, other than FAIRness, data science places special emphasis on the reproducibility of pipelines [2, 3].”
BackgroundSimulation Design: Trans-Paradigm Best-Practice from Software Engineering
Rossiter, S. (2015) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.2842 · Cited by 8
Vobi One: a data processing software package for functional optical imaging
Takerkart, S., Katz, P., Garcia, F., Roux, S., Reynaud, A., & Chavane, F. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00002 · Cited by 8
“Furthermore, a software suite that would fill all these specifica-tions will also facilitate providing reproducible research, following most of the principles described in Sandve et al. (2013), and ease interactions between different research groups working on functional optical imaging data.”
MethodMORPHEUS: An automated tool for unbiased and reproducible cell morphometry
Ruffinatti, F. A., Genova, T., Mussano, F. D., & Munaron, L. (2020) · Journal of Cellular Physiology · doi:10.1002/jcp.29768 · Cited by 8
“…debated on both generalist and more specialized journals in the last years (An, 2018; Baker, 2016; Begley & Ellis, 2012; Coiera, Ammenwerth, Georgiou, & Magrabi, 2018; Fanelli, 2018; França & Monserrat, 2018; Samsa & Samsa, 2019; Sandve et al., 2013; Turkiewicz, Luta, Hughes, & Ranstam, 2018).”
Tomi-Andrino, C., Norman, R. O. J., Millat, T., Soucaille, P., Winzer, K., Barrett, D. A., King, J. R., & Kim, D. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007694 · Cited by 8
“For the sake of reproducibility [36], the complete list of files used in this study was collected in S2 Table, and are publicly available in Nottingham SBRC’s GitHub profile (https://github.”
MethodSchölzel, C., Blesius, V., Ernst, G., Goesmann, A., & Dominik, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254749 · Cited by 9
“This includes structured documentation [1, 3, 11, 12], version control [10–13, 16], Borrowing software engineering concepts for improving the methods and results reproducibility of mathematical models seems natural, since these models are, after all, software.”
BackgroundMethodBayesian estimation reveals that reproducible models in Systems Biology get more citations
Höpfl, S., Pleiss, J., & Radde, N. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-29340-2 · Cited by 10
Patterson, A., Elbasir, A., Tian, B., & Auslander, N. (2023) · Cancers · doi:10.3390/cancers15071958 · Cited by 9
“It is therefore important to follow guidelines and tools for reproducible computational work [170,171].”
BackgroundBurcham, M., Al-Zyoud, M. M., Carver, J. C., Alsaleh, M. N., Du, H., Gillani, F., Jiang, J., Rahman, A., Kafalı, Ö., Al-Shaer, E. S., et al. (2017) · doi:10.1145/3055305.3055307 · Cited by 8
“• The Computational Science community has long recognized the need for reproducibility [24, 25, 2], but has yet to develop a comprehensive solution.”
BackgroundCreating Interactive Scientific Publications using Bindings
Konkol, M., Kray, C., & Suleiman, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3331158 · Cited by 8
“Manual data processing should be avoided for the sake of reproducibility [41].”
Scientific Notebook Software: Applications for Academic Radiology
Richardson, M. L., & Amini, B. (2017) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2017.09.005 · Cited by 8
System for Quality‐Assured Data Analysis: Flexible, reproducible scientific workflows
Fowler, J., Lucas, F. A. S., & Scheet, P. A. (2018) · Genetic Epidemiology · doi:10.1002/gepi.22178 · Cited by 8
Ten simple rules for implementing deep amplicon sequencing in parasitology
Šlapeta, J. R., Krücken, J., Rojas, A., Chambers, A., Melville, L. A., Valladares, M. M., Cantón, C., Francis, E. K., Zahid, O., de Albuquerque, A. C. A., et al. (2025) · International Journal for Parasitology · doi:10.1016/j.ijpara.2025.11.003 · Cited by 7
Identification of Gene Regulatory Networks from Single-Cell Expression Data
Li, S., Yan, H., & Lee, J. (2021) · Methods in molecular biology · doi:10.1007/978-1-0716-1534-8_9 · Cited by 7
Reproducible Research in Real Estate: A Review and an Example
Krause, A. (2016) · Journal of Real Estate Practice and Education · doi:10.1080/10835547.2016.12091758 · Cited by 7
Fitzpatrick, R., & Stefan, M. I. (2022) · Neuroinformatics · doi:10.1007/s12021-022-09584-5 · Cited by 8
“Standards for reproducible computational research have been formulated both as stand-alone guidelines (Sandve et al., 2013; Elofsson et al., 2019), and within the FAIR framework (Wilkinson et al., 2016).”
MethodThe ATTAC guiding principles to openly and collaboratively share wildlife ecotoxicology data
Muñoz, C. C., Charles, S., McVey, E. A., & Vermeiren, P. (2022) · MethodsX · doi:10.1016/j.mex.2022.101987 · Cited by 9
“Nevertheless, a transparent data description increases trust which eventually contributes to greater citations of one’s work, and also makes it possible to later understand and reuse one’s own data [78] .”
BackgroundA FAIR and modular image‐based workflow for knowledge discovery in the emerging field of imageomics
Balk, M. A., Bradley, J., Maruf, M., Altıntaş, B., Bakış, Y., BART, H. L. J., Breen, D. E., Florian, C., Greenberg, J., Karpatne, A., et al. (2024) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14327 · Cited by 8
“We also create YAML files to load environ - ments, such as an R environment and for image data downloads, following best practices for version control and defining paths to components (Roach et al., 2022 ; Sandve et al., 2013 ).”
BackgroundMethodInfluentialReproducibility in the Era of Big Data
McCoach, D. B., Dineen, J. N., Chafouleas, S. M., & Briesch, A. M. (2020) · Wiley eBooks · doi:10.1002/9781118976357.ch21 · Cited by 7
A Three-Phase Transport Model for High-Temperature Concrete Simulations Validated with X-ray CT Data
Pohl, C., Šmilauer, V., & Unger, J. F. (2021) · Materials · doi:10.3390/ma14175047 · Cited by 7
“For a general introduction to reproducible computational research, see [27].”
BackgroundVlasova, A. V., Hermoso, A., Ferreira, F. C., Ponomarenko, J. V., & Guigó, R. (2021) · Genes · doi:10.3390/genes12101645 · Cited by 7
“This also applies to bioinformatics and computational analyses [54].”
Preparation of mouse pancreatic tumor for single-cell RNA sequencing and analysis of the data
Surumbayeva, A., Kotliar, M., Gabitova-Cornell, L., Kartashov, A. V., Peri, S., Salomonis, N., Barski, A., & Astsaturov, I. A. (2021) · STAR Protocols · doi:10.1016/j.xpro.2021.100989 · Cited by 7
Torres, A. M., Balbi, S., & Villa, F. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281348 · Cited by 8
Modernizing CDC’s Practices and Culture for Better Data Sharing, Impact, and Transparency
Wiltz, J. L., Lee, B., Kaufmann, R. B., Carney, T. J., Davis, K., & Briss, P. A. (2024) · Preventing Chronic Disease · doi:10.5888/pcd21.230200 · Cited by 7
Stepping Stones to Reproducible Research: A Study of Current Practices in Parallel Computing
Carpen-Amarie, A., Rougier, A., & Lübbe, F. D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-14325-5_43 · Cited by 7
“A set of guidelines for conducting reproducible research are stated in the work of Sandve et al. [8], who argue that reproducibility is tightly connected to the availability of all experimental details.”
BackgroundMAGPIE: Simplifying access and execution of computational models in the life sciences
Baldow, C., Salentin, S., Schroeder, M., Roeder, I., & Glauche, I. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005898 · Cited by 7
Gsmodutils: a python based framework for test-driven genome scale metabolic model development
Gilbert, J., Pearcy, N., Norman, R. O. J., Millat, T., Winzer, K., King, J. R., Hodgman, C. T., Minton, N. P., & Twycross, J. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz088 · Cited by 8
Reproducible Tract Profiles (RTP): from diffusion MRI acquisition to publicationPreprint
Lerma‐Usabiaga, G., Perry, L. M., & Wandell, B. A. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/680173 · Cited by 7
“In many neuroimaging publications, there is no realistic chance that a reader can replicate the experimental data acquisition or reproduce the computational analyses (Wilson et al. 2017; Sandve et al. 2013; Buckheit and Donoho 1995).”
BackgroundLynnerup, N. A., Nolling, L., Andersen, R. H., & Hallam, J. C. T. (2019) · University of Southern Denmark Research Portal (University of Southern Denmark) · doi:10.48550/arxiv.1909.03772 · Cited by 7
“Research in robotics and machine learning (ML) is not excluded from this strict scientific requirement, even though it is notoriously hard to ensure reproducibility in computational studies of this nature [1].”
BackgroundPreventing Proteomics Data Tombs Through Collective Responsibility and Community Engagement
Vadadokhau, U., Soliman, M., Castillon, L., Muñoz, P. P., Id, L., Gayathri, S. N., Srivastava, A., Runeberg, T., González-Armijos, T., Šapovalovaitė, K., et al. (2026) · Scientific Data · doi:10.1038/s41597-026-06614-8 · Cited by 7
“…(i) raw data plus open formats; (ii) identifications in PSI standards (e.g., mzIdentML/mzTab); (iii) SDRF for proteomics; (iv) basic QC summaries (e.g., mzQC when available); (v) spectral libraries for DIA where used; (vi) full parameter files and analysis code with pinned versions/containers 34 .”
Ten Recommendations for Engineering Research Software in Energy Research
Ferenz, S., Frost, E., Schrage, R., Wolgast, T., Beyers, I., Karras, O., Werth, O., & Nieße, A. (2025) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3679240.3734606 · Cited by 6
“Additionally, version control enables reproducibility of the research conducted with the code by providing specific links to use exactly the same version [54].”
Nine quick tips for trustworthy machine learning in the biomedical sciences
Oneto, L., & Chicco, D. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013624 · Cited by 6
“Open-source tools such as Python or R enable free and reproducible experimentation, while proprietary software restricts access to those with licenses [73].”
MATES: A tool for appraising the completeness with which a meta-analysis has been reported
Morrison, K., Pottier, P., Pollo, P., Ricolfi, L., Williams, C., Yang, Y., Beillouin, D., Cardoso, S. J., Ferreira, V., Gallagher, B., et al. (2025) · Environment International · doi:10.1016/j.envint.2025.109935 · Cited by 6
Schlögl, M., Waltersdorfer, L., Regner, P., Siposova, A., & Brenning, A. (2026) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2026.106962 · Cited by 6
Iacobescu, P., & Șușnea, I. (2025) · Algorithms · doi:10.3390/a18080470 · Cited by 6
Context is Key for Reproducibility of Empirical Studies in Human-Computer Interaction
Vanderdonckt, J. M., & Vatavu, R.-D. (2025) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3736731.3746142 · Cited by 6
“Reproducibility is the foundation for cumulative scientific knowledge and progress, ensuring that research findings are consistent, reliable, verifiable, and constitutes a fundamental prerequisite to establish scientific validity [27,43].”
Ten simple rules for effective research data management
Hassenstein, M. J., & Jung, K. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013779 · Cited by 7
“Several publications of the “Ten Simple Rules” series offer guidance on RDM subdomains, such as Ten Simple Rules for Creating a Good Data Management Plan [14], Ten Simple Rules for Reproducible Computational Research [15], or Ten simple rules for large-scale data processing [16].”
A Practical Guide to Support Change-proneness Prediction
Melo, C., da Cruz, M. L., Martins, A., Matos, T., Filho, J., & Machado, J. C. (2019) · International Conference on Enterprise Information Systems · doi:10.5220/0007727702690276 · Cited by 6
“There are some authors who have proposed basic rules for reproducible computational research, as (Sandve et al., 2013) which proposed list ten rules.”
BackgroundComputational Reproducibility: A Practical Framework for Data Curators
Sawchuk, S., & Khair, S. (2021) · Journal of eScience Librarianship · doi:10.7191/jeslib.2021.1206 · Cited by 8
“While the existing literature on reproducibility is extensive, it has focused particularly on proposing solutions for researchers, rather than curators (Akmon 2017; Fear 2015; Goodman et al 2014; Gray et al. 2005; Hatton and van Genuchten 2019; Ioannidis 2014; King 2011; Kitzes et al. 2017; Macneil 2018; McCoach et al…”
BackgroundInfluentialImproving ecological data science with workflow management software
Brousil, M. R., Filazzola, A., Meyer, M. F., Sharma, S., & Hampton, S. E. (2023) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14113 · Cited by 8
“Workflow management software (WMS) can make data pipe - lines more reproducible, intelligible and less computationally demanding (Leipzig et al., 2021 ; Sandve et al., 2013 ).”
BackgroundRoad user behavior: Describing, inferring, predicting and beyond
Donmez, B., McDonald, A. D., Lee, J. D., & Boyle, L. N. (2023) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2023.100932 · Cited by 6
The architecture of the genome integrates scale independence with inverse symmetry
Warr, G. W., & Hatton, L. (2025) · Academia molecular biology and genomics. · doi:10.20935/acadmolbiogen7650 · Cited by 6
Mango: combining and analyzing heterogeneous biological networks
Chang, J., Cho, H., & Chou, H.-H. (2016) · BioData Mining · doi:10.1186/s13040-016-0105-5 · Cited by 6
“Script files enhance the speed and reproducability of analysis [11].”
BackgroundHaas, C. N. (2016) · Risk Analysis · doi:10.1111/risa.12730 · Cited by 6
“(9) Sandve et al.(10) outlined 10 broad rules for reproducible (computational) research, which are paraphrased:”
BackgroundSoftware engineering principles to improve quality and performance of R software
Russell, S., Bennett, T. D., & Ghosh, D. (2019) · PeerJ Computer Science · doi:10.7717/peerj-cs.175 · Cited by 7
“As research scientists tend to think of their software products as unique tools that will not be used regularly or for a long period, they often do not consider long term maintenance issues during the development phase (Sandve et al., 2013; Prins et al., 2015).”
Backgroundoemof.tabular – Introducing Data Packages for Reproducible Workflows in Energy System Modeling
Hilpert, S., Günther, S., & Söthe, M. (2021) · Journal of Open Research Software · doi:10.5334/jors.320 · Cited by 6
Stopsack, K. H., Mucci, L. A., Tworoger, S. S., Kang, J. H., Eliassen, A. H., Willett, W. C., & Stampfer, M. J. (2023) · Epidemiology · doi:10.1097/ede.0000000000001599 · Cited by 6
Recommendations for a standardized publication protocol for a QSP model
Weddell, J. C., Gulati, A., & Yamada, A. (2024) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-024-09943-6 · Cited by 6
Rovere, S. L., NORTH, M. J., Podestá, G., & Bert, F. (2016) · IEEE Access · doi:10.1109/access.2016.2592418 · Cited by 7
Efficiently Storing and Analyzing Genome Data in Database Systems
Dorok, S., Breß, S., Teubner, J., Läpple, H., Saake, G., & Markl, V. (2017) · Datenbank-Spektrum · doi:10.1007/s13222-017-0254-9 · Cited by 6
“This separation introduces additional and partly manual effort to ensure reproducibility of results [30].”
BackgroundTen quick tips for getting the most scientific value out of numerical data
Schwen, L. O., & Rueschenbaum, S. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006141 · Cited by 6
Guidelines for systematic reporting of sequence alignments
Vihinen, M. (2020) · Biology Methods and Protocols · doi:10.1093/biomethods/bpaa001 · Cited by 6
Wailord: Parsers and Reproducibility for Quantum Chemistry
Goswami, R. R. (2022) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-212e5952-021 · Cited by 7
“Independently, the ability to reproduce results [Pen11], [SNTH13] in all fields of computational research, and has spawned a veritable flock of methodological and programmatic advances [CAB + 19], including the sophisticated provenance tracking of AiiDA [PCS + 16], [HZU + 20].”
ResultAmending the literature through version control
Kane, A., & Amin, B. (2023) · Biology Letters · doi:10.1098/rsbl.2022.0463 · Cited by 6
“This is already in place for things like supplementary data and code in line with open science practices [12,14].”
BackgroundAztec: A Platform to Render Biomedical Software Findable, Accessible, Interoperable, and ReusablePreprint
Wang, W. W., Bleakley, B. J., Ju, C. J. ‐., Kyi, V., Tan, P. B. O., Choi, H., Huang, X., Zhou, Y., Wood, J., Wang, D., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06087 · Cited by 7
Ten simple rules for large-scale data processing
Fungtammasan, A., Lee, A., Taroni, J., Wheeler, K. G., Chin, C.-S., Davis, S. R., & Greene, C. S. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009757 · Cited by 6
“Applying version control to all code is always recommended for reproducible research [16,17].”
BackgroundMore Rigorous Software Engineering Would Improve Reproducibility in Machine Learning ResearchPreprint
Wolter, M., Veeramacheneni, L., & Hoyt, C. T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.00902 · Cited by 1
Zhang, H., Maillo, A., Khan, S. A., Martinez‐de‐Morentin, X., Lehmann, R., Gómez-Cabrero, D., & Tegner, J. N. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.10.034 · Cited by 7
Yan, L., Liu, P., Yao, Y., Yang, F., & Xu, F. (2025) · Buildings · doi:10.3390/buildings15234243 · Cited by 6
Evaluating the performance of ChatGPT-5, Claude, and DeepSeek for statistical analysis in medicine
Sebo, P., & Wang, T. (2026) · Internal and Emergency Medicine · doi:10.1007/s11739-026-04291-4 · Cited by 6
Lopes, P. H., de Oliveira, B. C., Souza, A., & Blanco, W. (2022) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2022.874234 · Cited by 5
“In this sense, several recommendations have already been suggested to achieve computational reproducibility and/or replicability of numerical results obtained by computational models (Sandve et al., 2013; Benureau and Rougier, 2018; Elofsson et al., 2019).”
BackgroundMethodTen simple rules for training scientists to make better software
Gallagher, K., Creswell, R., Lambert, B. C., Robinson, M. J., Lei, C. L., Mirams, G. R., & Gavaghan, D. J. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012410 · Cited by 8
“Given the importance of high-quality software to effective research in computational biology, there has been significant literature on ensuring reproducibility [20,21] and good”
Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI
Moulton, R. H., McCully, G. A., & Hastings, J. D. (2024) · Computer Assisted Radiology and Surgery - International Congress and Exhibition · doi:10.1109/cars61786.2024.10778911 · Cited by 5
“Without the ability to reliably reproduce and scrutinize research findings, it becomes challenging to build upon existing work, identify potential flaws or limitations, and foster trust in the proposed solutions [12].”
Bacterial Genomic Data Analysis in the Next-Generation Sequencing Era
Orsini, M., Cuccuru, G., Uva, P., & Fotia, G. (2016) · Methods in molecular biology · doi:10.1007/978-1-4939-3572-7_21 · Cited by 5
Mining Biomedical Publications With The LAPPS Grid
Ide, N., Suderman, K., & Kim, J.-D. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/328h4vy5y3tj · Cited by 5
biogitflow: development workflow protocols for bioinformatics pipelines with git and GitLab
Kamoun, C., Roméjon, J., de Soyres, H., Gallois, A., Girard, E., & Hupé, P. (2020) · F1000Research · doi:10.12688/f1000research.24714.3 · Cited by 5
Intelligence artificielle et santé animale
Ezanno, P., Picault, S., Winter, N., Beaunée, G., Monod, H., & Guégan, J. (2020) · INRAE Productions Animales · doi:10.20870/productions-animales.2020.33.2.3572 · Cited by 6
Janiform intra-document analytics for reproducible research
Dittrich, J., & Bender, P. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824114 · Cited by 5
“1Just read the “ten simple rules for reproducible results” [14] and then ask yourself how little of this we implement in our papers.”
BackgroundregionReport: Interactive reports for region-level and feature-level genomic analyses
Collado‐Torres, L., Jaffe, A. E., & Leek, J. T. (2016) · F1000Research · doi:10.12688/f1000research.6379.2 · Cited by 6
Language-Agnostic Reproducible Data Analysis Using Literate Programming
Vassilev, B., Louhimo, R., Ikonen, E., & Hautaniemi, S. (2016) · PLoS ONE · doi:10.1371/journal.pone.0164023 · Cited by 5
On the reproducibility of experiments of indexing repetitive document collections
Fariña, A., Martínez-Prieto, M. A., Claude, F., Navarro, G., Lastra-Díaz, J. J., Prezza, N., & Seco, D. (2019) · Information Systems · doi:10.1016/j.is.2019.03.007 · Cited by 5
“Besides, it may be necessary to use programs in the exact versions used originally [26].”
MethodDIscBIO: A User-Friendly Pipeline for Biomarker Discovery in Single-Cell Transcriptomics
Ghannoum, S., Netto, W. L., Fantini, D., Ragan-Kelley, B., Parizadeh, A., Jonasson, E., Ståhlberg, A., Farhan, H., & Köhn‐Luque, A. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms22031399 · Cited by 5
“Due to the complex concatenation of heterogeneous programs and custom scripts via file-based inputs and outputs, as well as the program dependencies and version requirements, many computational pipelines are difficult to reproduce [Peng RD 2011; Sandve et al., 2013].”
MethodBioTEA: Containerized Methods of Analysis for Microarray-Based Transcriptomics Data
Visentin, L., Scarpellino, G., Chinigò, G., Munaron, L., & Ruffinatti, F. A. (2022) · Biology · doi:10.3390/biology11091346 · Cited by 5
“There is a growing concern about reproducibility of research results in biomedical sciences and bioinformatics [19–22]; 4.”
BackgroundTop Tips to Make Your Research IrreproduciblePreprint
Hong, N. C., Crick, T., Gent, I. P., Kotthoff, L., & Takeda, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00062 · Cited by 5
Schwarz, F., Thormann, J., & Zimmermann, H. (1975) · PubMed · Cited by 5
LabInform: A Modular Laboratory Information SystemBuilt From Open Source ComponentsPreprint
Biskup, T. (2022) · ChemRxiv · doi:10.26434/chemrxiv-2022-vz360 · Cited by 5
Deploy-Master: Automating the Deployment of 50,000+ Agent-Ready Scientific Tools in One DayPreprint
Wang, Y., Huang, Z., Ding, Z., Liao, R., Huang, Y., Liu, X., Xie, J., Chen, S., & Zhang, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.03513 · Cited by 5
“Most tools remain difficult to compile, configure, and reproduce, relying heavily on manual intervention, undocumented assumptions, and environment-specific expertise [1, 2].”
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.08.07.239855 · Cited by 131
Zegrari, R., Ouchaoui, A. A., Gaouzi, Z., Abbou, H., Festali, R., Eljaoudi, R., Boutayeb, S., Belyamani, L., & Bourais, I. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf519 · Cited by 4
“Bioinformatics is a relatively recent discipline that still faces challenges such as reproducibility and false positives [9, 10].”
The computational reproducibility of articles published under the Open Data + FAIR policy of IJGIS
Kedron, P., Li, Z., & Liu, L. (2026) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2603586 · Cited by 4
“Synthesizing insights from multiple domain-specific efforts to improve the management and archiving of data and code (see Sandve et al .”
Nine quick tips for software containerization
Moreau, D., & Wiebels, K. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014197 · Cited by 4
“Several prior publications have outlined best practices for reproducible computational research and container usage [2,4,11,12].”
Odoom, S., Osei, E. O., Effah, E. Q., Boafo, V., Dusu, S., & Appiah, A. K. (2026) · Discover Global Society · doi:10.1007/s44282-026-00378-7 · Cited by 4
Cheimarios, N. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1711356 · Cited by 7
ACCESS climate data management
Godøy, Ø., & Saadatnejad, B. (2017) · AMBIO · doi:10.1007/s13280-017-0963-1 · Cited by 4
“By leaving the first and second steps (documentation and quality control) to the scientists, data managers can focus on developing tools, templates, best practise documentation and how to achieve reproducible research (Sandve et al. 2013).”
BackgroundInfluentialSánchez‐Reyes, L. L., & McTavish, E. J. (2022) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2022.2099487 · Cited by 6
“…rates long term in the natural sciences would benefit from addressing specific barriers for learners in the field, to support them in acquiring the skills needed to reproduce research workflows that rely heavily on computer code (Peng 2011; Sandve et al. 2013; Powers and Hampton 2019).”
BackgroundMethodTen simple rules for working with other people’s code
Pilgrim, C., Kent, P., Hosseini, K., & Chalstrey, E. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011031 · Cited by 5
“There are a wealth of resources addressing how to write better code in academia, including PLOS ONE “Ten Simple Rules” articles on reproducibility [2], documentation [3], and writing open-source software [4,5].”
BackgroundA Commentary on Multi-omics Data Integration in Systems Vaccinology
Shannon, C. P., Lee, A. H., Tebbutt, S. J., & Singh, A. (2024) · Journal of Molecular Biology · doi:10.1016/j.jmb.2024.168522 · Cited by 4
Eight quick tips for including chromosome X in genome-wide association studies
Bellavance, J., Wang, L. R., & Taliun, S. A. G. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012160 · Cited by 5
Ten simple rules for computational biologists collaborating with wet lab researchers
Robinson, M. D., Cai, P., Emons, M., Gerber, R., Germain, P., Gunz, S., Luo, S., Moro, G., Sonder, E., Sonrel, A., et al. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012174 · Cited by 4
“…(e.g., use FAIR file formats, version control, free and open source software, good coding practices and reproducible reporting [Wilkinson et al., 2016, Sandve et al., 2013, Stawarczyk and Roos, 2023] to ensure that data analysis is transparent and everyone can recreate your results and findings.”
Improving bioinformatics software quality through teamwork
Ferenc, K., Rauluševičiūtė, I., Hovan, L., Kumar, V., Kuijjer, M. L., & Mathelier, A. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae632 · Cited by 4
“…the usage of version control systems such as GitHub (http://github.com/), Bitbucket (https://bitbucket.org/), or GitLab (https://about. gitlab.com/), which has been recommended and discussed in previous papers (Nguyen-Hoan et al. 2010, Sandve et al. 2013, Blischak et al. 2016, Russell et al. 2018).”
BackgroundBhagwat, N., Urchs, S. G. W., Poline, J., & Yang, Y. (2025) · Imaging Neuroscience · doi:10.1162/imag.a.21 · Cited by 4
“…data principles for instilling trust and ensuring the reliability of our research endeavours ( Forscher, 1963; Guide for Reproducible Research — the Turing Way, n.d.; Mari ć et al., 2022; Niso et al., 2022; Prli ć & Procter, 2012; Sandve et al., 2013; Wagner et al., 2022; Wilkinson et al., 2016).”
CGAT: a model for immersive personalized training in computational genomics
Sims, D., Ponting, C. P., & Heger, A. (2015) · Briefings in Functional Genomics · doi:10.1093/bfgp/elv021 · Cited by 4
Data mining of effect sizes from PubMed abstracts: a cross-study conceptual replication
Monsarrat, P., & Vergnes, J. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty153 · Cited by 4
Patias, N. (2020) · REGION · doi:10.18335/region.v6i3.277 · Cited by 4
“The idea of using reproducible analyses in computational research has a growing number of advocates (Peng 2011, Sandve et al. 2013, Rule et al. 2019).”
BackgroundInfluenza Classification Suite: An automated Galaxy workflow for rapid influenza sequence analysis
Eisler, D. L., Fornika, D., Tindale, L. C., Chan, T., Sabaiduc, S., Hickman, R., Chambers, C., Krajden, M., Skowronski, D. M., Jassem, A. N., et al. (2020) · Influenza and Other Respiratory Viruses · doi:10.1111/irv.12722 · Cited by 4
NPARS—A Novel Approach to Address Accuracy and Reproducibility in Genomic Data Science
Ma, L., Peterson, E. A., Shin, I. J., Muesse, J. L., Marino, K., Steliga, M. A., & Johann, D. J. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.725095 · Cited by 5
A quantile fusion methodology for deep forecasting
Wang, B., Lü, J., Li, T., Yan, Z., & Zhang, G. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.029 · Cited by 4
Parker, C., Nelson, E., & Zhang, T. (2022) · Entropy · doi:10.3390/e24121747 · Cited by 4
“[46], Waltemath & Wolkenhauer [41], and Rule et al.”
BackgroundJohnson, A., Bouvette, M. J., Rangu, N., Morley, T., Schultz, A., Torgerson, T., & Vassar, M. B. (2023) · Annals of Otology Rhinology & Laryngology · doi:10.1177/00034894231185642 · Cited by 4
“33 Open-source software can remove barriers such as proprietary code which inhibits researchers’ ability to properly replicate and examine studies.”
BackgroundWhy Watershed Analysts Should Use R for Data Processing and Analysis
Moore, R. D., & Hutchinson, D. (2017) · Confluence Journal of Watershed Science and Management · doi:10.22230/jwsm.2017v1n1a2 · Cited by 4
Metadata
- Authors
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E.
- Affiliations
- University of Oslo (NO)
- Pennsylvania State University (US)
- Emory University (US)
- Oslo University Hospital (NO)
- Corresponding author
- Sandve, Geir Kjetil
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-24
- Volume
- 9
- Issue
- 10
- Pages
- e1003285
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3812051
- OpenAlex
- W2036318837
- Cited by
- 880
- References
- 27
- Citations by year
- 2026: 176 · 2025: 60 · 2024: 52 · 2023: 50 · 2022: 58 · 2021: 67 · 2020: 63 · 2019: 88 · 2018: 76 · 2017: 64 · 2016: 56 · 2015: 42 · 2014: 24 · 2013: 3
- FWCI
- 91.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (0850103)
- Pennsylvania Department of Health (HG004909)
- University of Pennsylvania
- Pennsylvania State University
- National Institutes of Health (HG006620)
- Huck Institutes of the Life Sciences
- Institute for Computational and Data Sciences, Pennsylvania State University
- National Human Genome Research Institute (HG005133, U41HG006620, R21HG005133, R01HG004909, HG006620, HG004909)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducible research, computational reproducibility, research practices, bioinformatics analysis, research transparency, version control, software dependency management, workflow management, research data management, research culture, research evaluation, research dissemination, scholarly communication
- MeSH terms
- Reproducibility of Results, Computational Biology, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.