Ten Simple Rules for Reproducible Computational Research
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Replication is the cornerstone of a cumulative science [1]. However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2]. As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3]. This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4]. The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8], [9]. This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3]. We want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run. We further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities. In a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys. As a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible. With this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6], [14]. We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your researc
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival1,11,12, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
From The FAIR Guiding Principles for scientific data management and stewardship (2016) · cites this paper as [12]
In the citing paper
“In a recent publication reviewing the automated citation screening methods reported for systematic reviews, the number of studies that met the current standards for reproducible software engineering experiments was low [29, 30, 36]; of 33 studies reporting approaches to automated screening, no study provided all the criteria for reproducible reporting [36].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundMinimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundGoecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome Biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,453 · cited 1×
“For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19], the GenePattern Word add-in [4], and Galaxy Pages [20].”
MethodSelf-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundReich, M., Liefeld, T., Gould, J., Lerner, J., Tamayo, P., & Mesirov, J. P. (2006) · Nature Genetics · doi:10.1038/ng0506-500 · Cited by 2,032
Galaxy: A platform for interactive large-scale genome analysis
Giardine, B., Riemer, C., Hardison, R. C., Burhans, R., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,026 · cited 2×
“If you are instead performing your analyses through an integrated framework (such as Gene-Pattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules.”
BackgroundSelf-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“…through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8,9].”
ResultTaverna: a tool for the composition and enactment of bioinformatics workflows
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M. R., Wipat, A., et al. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth361 · Cited by 1,602 · cited 2×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 3×
“In order to foster such aspects, it has been suggested that the scientific community needs to develop a ‘‘culture of reproducibility’’ for computational science, and to require it for published claims [3].”
BackgroundPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222 · cited 1×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
Background
Cited by
Items 501–600 of 1,001
Reproducibility as a Technical SpecificationPreprint
Crick, T., Hall, B. A., & Ishtiaq, S. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.01310 · Cited by 6
Research data management for open science
Lefebvre, A. (2021) · Utrecht University Repository (Utrecht University) · doi:10.33540/447 · Cited by 6
Empirical study on software and process quality in bioinformatics toolsPreprint
Ferenc, K., Otto, K., Neto, F. G. D. O., López, M. D., Horkoff, J., & Schliep, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.03.10.483804 · Cited by 6
Open Science in phonetics and phonologyPreprint
Coretta, S. (2020) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/4dz5t · Cited by 4
Minimalist Data Wrangling with PythonPreprint
Ga̧golewski, M. (2022) · arXiv (Cornell University) · doi:10.5281/zenodo.6451068 · Cited by 4
Koch, T., Gläser, D., Seeland, A., Roy, S., Schulze, K., Weishaupt, K., Boehringer, D., Hermann, S., & Flemisch, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12830 · Cited by 4
“Therefore, it is essential for the reproducibility of computer-based research and the reuse of research software that the source code is archived together with comprehensive installation instructions and a description of the computation environment [36].”
BackgroundHow computers broke science – and what we can do to fix it
Marwick, B. (2015) · The Conversation · doi:10.64628/aai.3ne3hpkvw · Cited by 4
eRNA-Min: Minimum Information Standard for Environmental RNA ReportingPreprint
Ahi, E. P. (2025) · Preprints.org · doi:10.20944/preprints202512.1765.v1 · Cited by 6
Enhancing Declarative Business Process Management Availability through Generative AI
Santos, W. D. S., Coutinho, J. R., Baião, F., Miranda Spyrides, G., & Vieira Lopes, H. C. (2025) · Process Science · doi:10.1007/s44311-025-00029-1 · Cited by 3
“Following the guidelines of (Sandve et al. 2013) in combination with the proposal of (Ivie and Thain 2018), we created an environment that promotes reproducible and replicable results 1 .”
Research into evidence-based psychological interventions needs a stronger focus on replicability
Niemeyer, H., Knaevelsrud, C., van Aert, R. C. M., & Ehring, T. (2023) · Clinical Psychology in Europe · doi:10.32872/cpe.9997 · Cited by 3
“To date, reproducibility attempts are highly uncommon in research on psychological interventions (see also, Sandve, Nekrutenko, Taylor, & Hovig, 2013).”
Reviving, reproducing, and revisiting Axelrod’s second tournament
Knight, V., Campbell, O., Harper, M., Gaffney, T. J., & Glynatsi, N. E. (2026) · Communications AI & Computing · doi:10.1038/s44488-026-00016-w · Cited by 3
Revealing the Best Strategies for Rare Cell Type Detection in Multi-Sample Single-Cell Datasets
Ye, Z., Yan, Y., Yu, Y., & Wu, H. S. (2025) · Genes · doi:10.3390/genes17010031 · Cited by 4
“In addition, we have written open and reproducible Jupyter notebooks with tutorials for all the available tools, which can be downloaded from our website [41].”
Bianchi, N., Lefebvre, A., Wolstencroft, K., & Spruit, M. R. (2026) · BMC Bioinformatics · doi:10.1186/s12859-026-06376-5 · Cited by 3
“For software, this involves maintaining comprehensive documentation of datasets and external sources, annotating code with clear explanations, reporting all random seeds used, and making source code publicly available [27].”
Bonaretti, S., Barzegari, M., Bevers, M. S. A. M., Boyd, S. K., Burghardt, A. J., Cameron, D., Chiumento, F., Crimi, G., Degenhart, G., Durongbhan, P., et al. (2026) · JBMR Plus · doi:10.1093/jbmrpl/ziag025 · Cited by 4
FastMDAnalysis: Software for Automated Analysis of Molecular Dynamics Trajectories
Aina, A., & Kwan, D. (2026) · Journal of Computational Chemistry · doi:10.1002/jcc.70350 · Cited by 3
“Inconsistent atom selections, alignment procedures, or algorithm parameters between bespoke scripts can yield different results, undermining direct comparison between studies and the scientific method itself [11].”
Verrelst, J., Clerck, E. D., Verma, B., Mishra, K., & Caballero, G. R. (2026) · Remote Sensing · doi:10.3390/rs18081154 · Cited by 4
Yue, L., Ko, C.-Y., Chen, P.-Y., Di, S., & Pan, S. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1820375 · Cited by 8
BOLE: A script-knowledge driven multi-agent framework for reproducible llm-assisted genomic breeding
Wang, Z., Liang, C., Zhang, Y., Zhou, R., Zhang, X., Wei, W., & Li, K. (2026) · Journal of Integrative Agriculture · doi:10.1016/j.jia.2026.07.011 · Cited by 3
Using Cloud-Aware Provenance to Reproduce Scientific Workflow Execution on Cloud
Hasham, K., Munir, K., & McClatchey, R. (2015) · International Conference on Cloud Computing and Services Science · doi:10.5220/0005452800490059 · Cited by 3
“This paper (Sandve et al. 2013) suggests archiving the exact versions of all programs and enabling version control on all scripts used in an experiment.”
BackgroundStanding Together for Reproducibility in Large-Scale Computing
James, D. L., Wilkins‐Diehr, N., Stodden, V., Colbry, D., & Rosales, C. (2014) · Cited by 3
Ranchers Adapting to Climate Variability in the Upper Colorado River Basin, Utah
Akbar, H., Allen, L., Rosenberg, D. E., & Chikamoto, Y. (2020) · Climate · doi:10.3390/cli8090096 · Cited by 3
biogitflow: development workflow protocols for bioinformatics pipelines with git and GitLab
Kamoun, C., Roméjon, J., de Soyres, H., Gallois, A., Girard, E., & Hupé, P. (2020) · F1000Research · doi:10.12688/f1000research.24714.2 · Cited by 3
A Practical Guide to Reproducible Modeling for Biochemical Networks
Porubsky, V. L., & Sauro, H. M. (2023) · Methods in molecular biology · doi:10.1007/978-1-0716-3008-2_5 · Cited by 3
Blockchain from a Modern Perspective: An Evolution to Health Science
Chaudhary, A., Kaushik, K., & Kumar, S. (2023) · IntechOpen eBooks · doi:10.5772/intechopen.109178 · Cited by 3
Private, Public, Personal: Shifting Patterns in Geospatial Data Sources in Geographic Research
Appiah, G., Kaufman, M., Cooney, B., & Andris, C. (2024) · Annals of the American Association of Geographers · doi:10.1080/24694452.2024.2394078 · Cited by 3
“We might lack a set of standards or guidelines regarding the format or file types of geo-spatial data shared through repositories (Sandve et al. 2013; Stodden, et al. 2014), but researcher activity has reflected aspects of FAIR standards that suggest that research artifacts are findable,…”
Building a Cloud Service for Reproducible Simulation Management
Congo, F. Y. (2015) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-7b98e3ed-01d · Cited by 3
“From more general approaches like defining rules that have to be fulfilled [Sandve2013], to more specific approaches [Heroux2011], we can define a set of metadata that are useful to determine the reproducibility of a simulation.”
BackgroundEnabling reproducible obia with open-source software in docker containers
Knoth, C., & Nüst, D. (2016) · International Conference on Geographic Object-Based Image Analysis (GEOBIA) · doi:10.3990/2.456 · Cited by 3
“…argue for reproducibility in “Five selfish reasons to work reproducibly” (Markowetz, 2015) or publish “Ten Simple Rules for Reproducible Computational Research” (Sandve et al., 2013), which essentially argue in favour of a proper scientific workflow simply to be able to reproduce your own results.”
BackgroundCreation of Reusable Bioinformatics Workflows for Reproducible Analysis of LC-MS Proteomics Data
Uszkoreit, J., Ahrens, M., Barkovits, K., Marcus, K., & Eisenacher, M. (2017) · Neuromethods · doi:10.1007/978-1-4939-7119-0_19 · Cited by 3
Short-format Workshops Build Skills and Confidence for Researchers to Work with Data
Jordan, K. L., Corvellec, M., Wickes, E., Zimmerman, N. B., Duckles, J. M., & Teal, T. K. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--30960 · Cited by 3
“Combined, these workshops provide a strong opportunity to learn and the needed conceptual framework for participants to excel in data-driven research [8] - [9].”
BackgroundCongressbr: An R Package for Analyzing Data from Brazil’s Chamber of Deputies and Federal Senate
McDonnell, R. M., Duarte, G. J., & Freire, D. (2019) · Latin American Research Review · doi:10.25222/larr.447 · Cited by 5
Evaluation of a Random Forest Model to Identify Invasive Carp Eggs Based on Morphometric Features
Goode, K., Weber, M. J., Matthews, A., & Pierce, C. L. (2021) · North American Journal of Fisheries Management · doi:10.1002/nafm.10616 · Cited by 4
“The seed ensures the use of the same pseudorandom numbers that were used to generate the samples and subsets (Sandve et al. 2013).”
MethodStawarczyk, B., & Roos, M. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011052 · Cited by 3
“D10: Avoid questionable research practices [3] S10: Connect textual statements to underlying results [1]”
BackgroundMethodInfluentialReproducibility in 2023 – An End-to-End Template for Analysis and Manuscript Writing
Mang, J. M., Prokosch, H., & Kapsner, L. A. (2023) · Studies in health technology and informatics · doi:10.3233/shti230064 · Cited by 3
“Many tutorials, tools, and reports are available to facilitate approaching the topic of reproducibility in science [5,11,12].”
BackgroundTerpsichora: A Tool to Generate Synthetic MP-Declare Process Models
Santos, W. D. S., Coutinho, J. R., Baião, F., Spyrides, G. M., & Lopes, H. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-82225-4_46 · Cited by 3
Kuczyńska, M., Kot, M., Stocki, M., Zapora, E., Jagielski, T., Perlińska-Teresiak, M., & Kalińska, A. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26125451 · Cited by 3
“Data preprocessing, statistical testing, and graphical visualization were conducted in accordance with current standards of reproducible data analysis workflows [67–69].”
Equipping Physiologists with an Informatics Tool Chest: Toward an Integerated Mitochondrial Phenome
Garlid, A. O., Polson, J., Garlid, K., Hermjakob, H., & Ping, P. (2016) · Handbook of experimental pharmacology · doi:10.1007/164_2016_93 · Cited by 3
Agile Development of an Attitude-Behaviour Driven Simulation of Alcohol Consumption Dynamics
Moyo, D., Ally, A. K., Brennan, A., Norman, P., Purshouse, R. C., & Strong, M. (2015) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.2841 · Cited by 5
Melchert, O., Meinhardt‐Wollweber, M., & Roth, B. (2018) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/aaa51a · Cited by 3
Simulation Standardization: Current State and Cross-Platform System for Network Simulators
Latif, Z., Sharif, K., Alvi, M. K., & Li, F. (2018) · Communications in computer and information science · doi:10.1007/978-981-10-8890-2_38 · Cited by 6
Tools for Understanding miRNA–mRNA Interactions for Reproducible RNA Analysis
Bagnacani, A., Wolfien, M., & Wolkenhauer, O. (2019) · Methods in molecular biology · doi:10.1007/978-1-4939-8982-9_8 · Cited by 3
Developing a Data Management Platform for the Ocean Science Community
Turner, C., & Gill, I. (2018) · Marine Technology Society Journal · doi:10.4031/mtsj.52.3.8 · Cited by 3
DCA++ project: Sustainable and scalable development of a high-performance research code
Hähner, U. R., Balduzzi, G., Doak, P., Maier, T., Solcà, R., & Schulthess, T. C. (2019) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1290/1/012017 · Cited by 3
Transparent, Efficient, and Robust Word Embedding Access with WOMBATPreprint
Müller, M.-C., & Strube, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00717 · Cited by 3
“WOMBAT implements some Best Practices for research reproducibility (Sandve et al., 2013; Stodden and Miguez, 2014), and complements existing approaches towards WEC standardization and sharing.”
BackgroundMethodBISTRO: Berkeley Integrated System for Transportation OptimizationPreprint
Feygin, S., Lazarus, J. R., Forscher, E. H., Golfier-Vetterli, V., Lee, J., Gupta, A., Waraich, R. A., Sheppard, C., & Bayen, A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03821 · Cited by 3
“Research conducted using BISTRO strives to meet the highest standard of reproducibility in computational experiments [58–60] as well as fact-based policymaking [61] by making all data, models, and algorithms freely available and open source30.”
MethodPatterns of PatternsPreprint
Corneli, J., Murphy, A., Puzio, R., Vivier, L., Alhasan, N., Danoff, C. J., Bruno, V., & Pierce, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10497 · Cited by 3
Prabhu, A., & Fox, P. (2021) · Encyclopedia of earth sciences series/Encyclopedia of earth sciences · doi:10.1007/978-3-030-26050-7_277-1 · Cited by 3
Tool-driven Revolutions in Archaeological SciencePreprint
Schmidt, S. C., & Marwick, B. (2019) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/4nkxv · Cited by 3
Cerna-Turoff, I., Chillrud, L., Rudolph, K. E., & Casey, J. A. (2023) · International Journal of Epidemiology · doi:10.1093/ije/dyad066 · Cited by 3
Editorial: Guidelines and Best Practices to Share Deidentified Data and Code
Horton, N. J., & Stoudt, S. (2024) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2024.2364737 · Cited by 3
“Sandve et al. (2013) provide ten simple rules for reproducible computational research.”
Hillemann, F., Burant, J. B., Čulina, A., & Vriend, S. J. G. (2025) · doi:10.32942/x26s6p · Cited by 3
Qualitative evaluation of word embeddings : investigating the instability in neural-based models
Pierrejean, B. (2020) · theses.fr (ABES) · doi:10.70675/a1747243zc752z4da7z86dezf358c5a1d487 · Cited by 9
Geiger, R. S., Varoquaux, N., Mazel-Cabasse, C., & Holdgraf, C. (2018) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/2fdhr · Cited by 2
A Practical Guide to Evaluating and Using Big Data in Digestive Disease Research
Alizadeh, M. A., Moura, N. S., Schledwitz, A., Patil, S. A., El‐Serag, H. B., Ravel, J., & Raufman, J. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.11.292 · Cited by 3
Addressing Reproducibility Challenges in HPC with Continuous Integration
Hayot‐Sasson, V., Hudson, N., Bauer, A., Gonthier, M., Foster, I. T., & Chard, K. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759874 · Cited by 4
“Modern computational and experimental methodologies have created new challenges in reproducing research outputs [35, 51, 56].”
Physics computational literacy: Programming, modeling, and collaboration at the journeyman level
Fredly, K. H., Odden, T. O. B., & Zwickl, B. M. (2025) · Physical Review Physics Education Research · doi:10.1103/6fny-yzyb · Cited by 2
“We see this specialization and independence as development toward expertise in the cognitive pillar.”
Database of recurrent mutations, an unbiased web resource to browse recurrent mutations in cancers
Chakroborty, D., Vaparanta, K. M., Ghimire, B. R., Paatero, I., Kurppa, K. J., & Elenius, K. (2025) · iScience · doi:10.1016/j.isci.2025.114561 · Cited by 2
Tutorial for variant interrogation in tumor samples
Arseneau, R. J., MacLean, L. K., Boudreau, J. E., & Gaston, D. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013924 · Cited by 2
“Genomic analyses rely on complex workflows that must be documented for reproducibility, validation, and reuse [105] (Table 7).”
Motoyama, Y., Yoshimi, K., Aoyama, T., Terayama, K., Tsuda, K., & Tamura, R. (2026) · Computer Physics Communications · doi:10.1016/j.cpc.2026.110368 · Cited by 2
ACVI-Med, an open source variant interpretation tool for medical genomics
Stadler, M., Gansberger, S., Borik‐Heil, L., Griss, J., Okamoto, I., Weninger, W. P., & Wielscher, M. (2026) · BMC Bioinformatics · doi:10.1186/s12859-026-06414-2 · Cited by 2
“Beyond accessibility, a major obstacle in medical genomics is the lack of transparent and reproducible pipelines for variant classification [19].”
Ramos, M. C. S., da Camara, A. O., & Freitas, H. R. (2026) · Standards · doi:10.3390/standards6020013 · Cited by 2
“Reporting guidelines such as ARRIVE 2.0 improve the description of experimental conditions [11].”
InfluentialPipelines for Proof: Devops Automation for Sustainable Research Software
Kavitha, J. J. S. (2026) · International Conference on Innovative Practices in Technology and Management (ICIPTM) · doi:10.1109/iciptm69057.2026.11466246 · Cited by 2
“Research reproducibility success rates improved from 58% to 92% as measured by independent verification of published results using only the automated artifacts and documentation [26].”
InfluentialEconstellar: An open-source AI-augmented research engine for computational financial econometrics
Bhandari, A. (2026) · Finance Research Open · doi:10.1016/j.finr.2026.100187 · Cited by 2
Romero-Nuñez, C. S., Rivera-Montalvo, T., Calderón-Arenas, J. A., & Sadek, A. M. (2026) · Radiation Measurements · doi:10.1016/j.radmeas.2026.107749 · Cited by 2
Ang, M. Y., Chen, L., Song, L., Lipovich, L., & Choo, S. W. (2026) · Life · doi:10.3390/life16061032 · Cited by 2
“In microbial genomics, an acceptable output must also be biologically accurate, evidence-supported, reproducible, transparent, and safe for its intended use [4,17].”
InfluentialPumas: A Unified Platform for Pharmaceutical Modeling and Simulation
Mogensen, P. K., Noack, A., Tarek, M., Elrod, C., Müller‐Widmann, D., Krumbiegel, J., Hatherly, M., Korsbo, N., Rackauckas, C., & Ivaturi, V. D. (2026) · Quantitative Medicine · doi:10.70534/qm-2026-0011 · Cited by 3
COMBINE archive: One File To Share Them All.
Bergmann, F., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., Hucka, M., Laibe, C., Miller, A. K., Nickerson, D. P., et al. (2014) · Cited by 2
Udod, K., Kushnarenko, V., Wesner, S., & Svjatnyj, V. (2019) · IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS) · doi:10.1109/idaacs.2019.8924459 · Cited by 2
“programs) should be stored including their exact versions [4] and required dependencies.”
BackgroundNine Tips to Improve Your Everyday Forest Data Analysis
Russell, M. B. (2020) · Journal of Forestry · doi:10.1093/jofore/fvaa034 · Cited by 2
“Integrating the concept of reproducibility into your work sends the signal that your analysis is high quality, trustworthy, and transparent (Sandve et al. 2013).”
Epistemic Programming - An insight-driven programming concept for Data Science
Hüsing, S. (2021) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3488042.3490510 · Cited by 2
“In addition, the documentation of the code and the interpretation of the output should be done close to the programming code and the outputs to enable the most direct and traceable documentation of the programming and cognition process in terms of reproducible research [11, 15, 16].”
BackgroundRibeiro, C., Oliveira, L. P., Batista, R., & de Sousa, M. P. A. (2021) · Biodiversity Data Journal · doi:10.3897/bdj.9.e78132 · Cited by 2
Analysis Pipelines and a Platform Solution for Next‐Generation Sequencing Data
Duarte, V., Gómez, A., & Corchado, J. M. (2024) · Wiley eBooks · doi:10.1002/9781119846567.ch11 · Cited by 3
He, Y., Hao, X., Hu, B., Xia, N., Wang, C., Chen, X., Zhang, H.-Y., Duan, Y., Ying, Q., & Dong, Q. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1524714 · Cited by 2
“Machine learning identi fi es data patterns by simulating human learning and employs techniques like support vector machines, random forests, and logistic regression to enhance performance (46, 47).”
Ferreira-Santos, J., & Pombo, L. (2026) · Computers · doi:10.3390/computers15040243 · Cited by 2
Electronic laboratory notebooks in a public–private partnership
Vaas, L. A. I., Witt, G., Windshügel, B., Bosin, A., Serra, G., Brüngger, A., Winterhalter, M., Gribbon, P. M., Levy-Petelinkar, C. J., & Köhler, M. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.83 · Cited by 2
Data-first manifesto: Shifting priorities in scholarly communications
Anderson, C. B. (2017) · Information Services & Use · doi:10.3233/isu-170852 · Cited by 2
Computation of the impedance of a vacuum electron diode with emission current delay
Egorov, I. S., Poloskov, A. V., & Remnev, G. E. (2018) · Nuclear Instruments and Methods in Physics Research Section A Accelerators Spectrometers Detectors and Associated Equipment · doi:10.1016/j.nima.2018.12.008 · Cited by 2
Integrating Scientific Programming in Communities of Practice for Students in the Life Sciences
Salsbury, A. M., Brown, A. M., & Lemkul, J. A. (2019) · Practice and Experience in Advanced Research Computing · doi:10.1145/3332186.3333040 · Cited by 2
“Learning experiences should demonstrate the importance of commenting scripts, keeping track of how results are produced, tracking intermediate results, and avoiding any steps or techniques that involve manual data manipulation [14].”
BackgroundPWD: A Petrological Workspace and Database tool
Angel, D. S., Albert, H., Tuan, L. N., Phi, N. X., Utami, S. B., Li, W., Khanh, P. M., Widiwijayanti, C., & Costa, F. (2019) · Geochemistry Geophysics Geosystems · doi:10.1029/2019gc008710 · Cited by 2
“…numerous general‐purpose repositories have been created to make data available (Roche et al., 2015; Wilkinson et al., 2016), improvement is still needed for practices relating the repositories management to ensure data reproducibility (Roche et al., 2015; Sandve et al., 2013; White et al., 2013).”
BackgroundEmmert‐Streib, F., Dehmer, M., & Yli-Harja, O. P. (2019) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2019.00349 · Cited by 2
“For this reason, minimal standards have been established that should be followed (Sandve et al., 2013) and important elements of such standards include: • documentation of all steps (data generation and data analysis).”
BackgroundCUF-Links: Continuous and Ubiquitous FAIRness Linkages for Reproducible Research
Foster, I. T., & Kesselman, C. (2022) · Computer · doi:10.1109/mc.2022.3160876 · Cited by 2
“The need to follow proven practices such as automating repeated actions, using version control to track the evolution of software systems, documenting assumptions, and generating unit tests are, we hope, taken as a given [16].”
BackgroundGayle, V., & Connelly, R. (2022) · Methodological Innovations · doi:10.1177/20597991221111681 · Cited by 2
“Baiocchi (2007) proposed guidelines for computational economics, Hofner et al. (2016) for biometrics, Begley and Ioannidis (2015) for medical research, Sandve et al. (2013) for computational research, and both Nosek et al. (2012) and Obels et al. (2020) for psychology.”
BackgroundTowards more reproducibility in vegetation research
Sperandii, M. G., Bazzichetto, M., Mendieta‐Leiva, G., Schmidtlein, S., Bott, M., de Lima, R. A. F., Pillar, V. D., Price, J. N., Wagner, V., & Chytrý, M. (2024) · Journal of Vegetation Science · doi:10.1111/jvs.13224 · Cited by 2
Establishing a Reproducible and Sustainable Analysis Workflow
Ramduny, J., Garcia, M., & Kelly, C. A. M. (2024) · Neuromethods · doi:10.1007/978-1-0716-4260-3_4 · Cited by 2
Frery, A. C. (2015) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2015.2404211 · Cited by 2
“This greatly enhances the value of your contribution, particularly if organized according to the Reproducible Research guidelines (see, for instance, [3]–[5]).”
MethodMerging OpenLifeData with SADI services using Galaxy and DockerPreprint
Aranguren, M. E. (2015) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/013615 · Cited by 2
“Computational reproducibility is becoming an important area in life sciences [Garijo et al. (2013); Sandve et al. (2013)].”
BackgroundThe Solution to Science's Replication CrisisPreprint
Knuteson, B. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2835131 · Cited by 2
“[78] Sandve GK, Nekrutenko A, Taylor J, Hovig E.”
Storing Reproducible Results from Computational Experiments using Scientific Python Packages
Oxvig, C. S., Arildsen, T., & Larsen, T. (2016) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-629e541a-006 · Cited by 2
“Several studies have given best practices for how to detail such metadata to make computational results reproducible, see e.g. [VKV09], [SNTH13], [SM14], [Dav12].”
BackgroundMethodInfluentialA generic multi-level stochastic modelling framework in computational epidemiologyPreprint
Picault, S., Huang, Y.-L., Sicard, V., Hoch, T., Vergu, E., Beaudeau, F., & Ezanno, P. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/491605 · Cited by 2
“To avoid misinterpretations coming from programming biases [1], several good 20 practices in software development were proposed [2] (e.”
MethodEMULSION: transparent and flexible multiscale stochastic models in epidemiologyPreprint
Picault, S., Huang, Y.-L., Sicard, V., Arnoux, S., Beaunée, G., & Ezanno, P. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/563791 · Cited by 2
A multi-omics digital research object for the genetics of sleep regulationPreprint
Jan, M., Gobet, N., Diessler, S., Franken, P., & Xénarios, I. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/586206 · Cited by 2
Ten simple rules for getting started with statistics in graduate school
Zitomer, R. A., Karr, J., Kerstens, M. E., Perry, L. R., Ruth, K. A., Adrean, L. J., Austin, S. H., Cornelius, J. M., Dachenhaus, J., Dinkins, J. B., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010033 · Cited by 2
“Using a statistical programming language can also increase reproducibility as scripts not only enable data manipulation and statistical testing but also record the nature and sequence of these steps [15].”
BackgroundResistance to Adoption of Best PracticesPreprint
Sholler, D., Stoudt, S., Kennedy, C. J., de la Guardia, F. H., Lanusse, F., Ram, K., Ottoboni, K., Stuart, M. A., Vareth, M., Varoquaux, N., et al. (2019) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/qr8cz · Cited by 5
Peer Review #1 of "Software citation principles (v0.2)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.86v0.2/reviews/1 · Cited by 2
Peer Review #1 of "Software citation principles (v0.1)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.86v0.1/reviews/1 · Cited by 2
Peer Review #2 of "Software citation principles (v0.1)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.86v0.1/reviews/2 · Cited by 2
Analysis reproducibility in mental health research: challenges and solutionsPreprint
Botvinik‐Nezer, R., & Wager, T. D. (2022) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ujnmp · Cited by 2
The Role Of Software Engineering In Bioinformatics
Lawlor, B. (2021) · doi:10.34719/rjbz6161 · Cited by 2
10 quick tips for making your software outlive your jobPreprint
Littauer, R., Wilson, G., Ainali, J., AlOmar, E. A., Arabas, S., Saibene, Y. B., Bubendorfer, K., Champion, K., Dillon, C., Helske, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.06484 · Cited by 3
“Rather than repeating what others have said [33–44], this tip is a prioritized checklist of things to do if you have time .”
EZInput: A Cross-Environment Python Library for Easy UI Generation in Scientific ComputingPreprint
Saraiva, B. M., Hidalgo-Cenalmor, I., Brito, A. D., Martínez, D., Shakespeare, T., Jacquemet, G., & Henriques, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.08859 · Cited by 2
“This addresses reproducibility concerns raised by Sandve et al. (Sandve et al., 2013) regarding inadequate documentation of computational methods as it provides a practical infrastructure for systematic validation workflows.”
Ali, S. A., & Lynn, A. (2026) · Aperture Neuro · doi:10.52294/001c.160138 · Cited by 3
Metadata
- Authors
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E.
- Affiliations
- University of Oslo (NO)
- Pennsylvania State University (US)
- Emory University (US)
- Oslo University Hospital (NO)
- Corresponding author
- Sandve, Geir Kjetil
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-24
- Volume
- 9
- Issue
- 10
- Pages
- e1003285
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3812051
- OpenAlex
- W2036318837
- Cited by
- 880
- References
- 27
- Citations by year
- 2026: 176 · 2025: 60 · 2024: 52 · 2023: 50 · 2022: 58 · 2021: 67 · 2020: 63 · 2019: 88 · 2018: 76 · 2017: 64 · 2016: 56 · 2015: 42 · 2014: 24 · 2013: 3
- FWCI
- 91.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (0850103)
- Pennsylvania Department of Health (HG004909)
- University of Pennsylvania
- Pennsylvania State University
- National Institutes of Health (HG006620)
- Huck Institutes of the Life Sciences
- Institute for Computational and Data Sciences, Pennsylvania State University
- National Human Genome Research Institute (HG005133, U41HG006620, R21HG005133, R01HG004909, HG006620, HG004909)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducible research, computational reproducibility, research practices, bioinformatics analysis, research transparency, version control, software dependency management, workflow management, research data management, research culture, research evaluation, research dissemination, scholarly communication
- MeSH terms
- Reproducibility of Results, Computational Biology, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.