Ten Simple Rules for Reproducible Computational Research
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Replication is the cornerstone of a cumulative science [1]. However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2]. As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3]. This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4]. The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8], [9]. This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3]. We want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run. We further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities. In a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys. As a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible. With this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6], [14]. We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your researc
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival1,11,12, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
From The FAIR Guiding Principles for scientific data management and stewardship (2016) · cites this paper as [12]
In the citing paper
“In a recent publication reviewing the automated citation screening methods reported for systematic reviews, the number of studies that met the current standards for reproducible software engineering experiments was low [29, 30, 36]; of 33 studies reporting approaches to automated screening, no study provided all the criteria for reproducible reporting [36].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundMinimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundGoecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome Biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,453 · cited 1×
“For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19], the GenePattern Word add-in [4], and Galaxy Pages [20].”
MethodSelf-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundReich, M., Liefeld, T., Gould, J., Lerner, J., Tamayo, P., & Mesirov, J. P. (2006) · Nature Genetics · doi:10.1038/ng0506-500 · Cited by 2,032
Galaxy: A platform for interactive large-scale genome analysis
Giardine, B., Riemer, C., Hardison, R. C., Burhans, R., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,026 · cited 2×
“If you are instead performing your analyses through an integrated framework (such as Gene-Pattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules.”
BackgroundSelf-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“…through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8,9].”
ResultTaverna: a tool for the composition and enactment of bioinformatics workflows
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M. R., Wipat, A., et al. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth361 · Cited by 1,602 · cited 2×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 3×
“In order to foster such aspects, it has been suggested that the scientific community needs to develop a ‘‘culture of reproducibility’’ for computational science, and to require it for published claims [3].”
BackgroundPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222 · cited 1×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
Background
Cited by
Items 201–300 of 1,001
BioModel Engineering with Petri Nets
Blätke, M. A., Heiner, M., & Marwan, W. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-801213-0.00007-1 · Cited by 40
Tjerkaski, J., Červenka, S., Farde, L., & Matheson, G. J. (2020) · EJNMMI Research · doi:10.1186/s13550-020-00664-8 · Cited by 35
Reproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 43
A call for clean code to effectively communicate science
Filazzola, A., & Lortie, C. J. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13961 · Cited by 40
“…‘ 10 Simple Rules ’ series also provides multiple great recommendations for improving computational reproducibility such as using version con-trol, providing ancillary documentation (e.g. README files, worked examples, vignettes) and automation (Lee, 2018; Sandve et al., 2013; Wilson et al., 2017).”
BackgroundTowards Automated Melanoma Screening: Proper Computer Vision & Reliable ResultsPreprint
Fornaciali, M. S., Carvalho, M. C., Bittencourt, F. V., Avila, S., & Junior, E. A. D. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.04024 · Cited by 37
“As we mentioned, reproducibility is a thorny issue, plaguing all Science [41], but in particular Computer Science [39, 40].”
BackgroundInfluentialGeiger, R. S., Varoquaux, N., Mazel-Cabasse, C., & Holdgraf, C. (2018) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-018-9333-1 · Cited by 32
“…analyzes are also gaining popularity due to the open science and reproducibility movements, as the software is free to use, open to review and modification, and lets researchers share every step taken in processing and analyzing the data (Sandve et al. 2013; Wilson et al. 2014; Kitzes et al. 2018).”
BackgroundInfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid
Pradal, C., Artzet, S., Chopard, J., Dupuis, D., Fournier, C., Mielewczik, M., Nègre, V., Neveu, P., Parigot, D., Valduriez, P., et al. (2016) · Future Generation Computer Systems · doi:10.1016/j.future.2016.06.002 · Cited by 32
“One of the major aims of our infrastructure is to be reproducibility-friendly [18].”
Modeling Cellular Response in Large-Scale Radiogenomic Databases to Advance Precision Radiotherapy
Manem, V. S., Lambie, M., Smith, I. C. P., Smirnov, P., Kofia, V., Freeman, M., Koritzinsky, M., Abazeed, M. E., Haibe‐Kains, B., & Bratman, S. V. (2019) · Cancer Research · doi:10.1158/0008-5472.can-19-0179 · Cited by 33
The journey to transparency, reproducibility, and replicability
Henry, S. B. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz007 · Cited by 32
“• Record all intermediate results.(3) • Deposit data in a repository.”
BackgroundMethodInfluentialTool-Driven Revolutions in Archaeological Science
Schmidt, S. C., & Marwick, B. (2020) · Journal of Computer Applications in Archaeology · doi:10.5334/jcaa.29 · Cited by 41
“…publications that describe and recommend code sharing in statistics (Baumer et al. 2014), genome biology (Markowetz 2015), computational biology (Sandve et al. 2013), hydrology (Slater et al. 2019), biostatistics (Peng 2009), computer science (Mitchell et al. 2012; Peng 2011), applied…”
BackgroundDetecting adaptive convergent amino acid evolution
Rey, C., Lanore, V., Veber, P., Guéguen, L., Lartillot, N., Sémon, M., & Boussau, B. (2019) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2018.0234 · Cited by 28
Camargo, A. V., Papadopoulou, D., Spyropoulou, Z., Vlachonasios, K. E., Doonan, J. H., & Gay, A. P. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096889 · Cited by 31
“Here we have provided equivalent analysis in open source software and have provided the equations the calculation image descriptors which are closer to the ideal for reproducible computational research [39].”
MethodBystro: rapid online variant annotation and natural-language filtering at whole-genome scale
Kotlar, A. V., Trevino, C. E., Zwick, M. E., Cutler, D. J., & Wingo, T. S. (2018) · Genome Biology · doi:10.1186/s13059-018-1387-3 · Cited by 32
“Existing variant filtering solutions are even more limited, with most analyses requiring researchers to program custom scripts, which can result in errors that impact reproducibility [5].”
BackgroundCategorizing metadata to help mobilize computable biomedical knowledge
Alper, B. S., Flynn, A. J., Bray, B. E., Conte, M. L., Eldredge, C. E., Gold, S., Greenes, R. A., Haug, P. J., Jacoby, K., Koru, G., et al. (2021) · Learning Health Systems · doi:10.1002/lrh2.10271 · Cited by 32
“37 Some suggest that private software code repositories, such as GitHub, Source Forge, and Bitbucket, are suitable for hosting CBKs. 38 However, others point out the policies governing these repositories may not fully support the CBK long-term sharing needs of biomedical scientists.”
BackgroundKelleher, C. A., & Braswell, A. (2021) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2021.105113 · Cited by 31
maplet : an extensible R toolbox for modular and reproducible metabolomics pipelines
Chetnik, K., Benedetti, E., Gomari, D. P., Schweickart, A., Batra, R., Büyüközkan, M., Wang, Z., Arnold, M., Zierer, J., Suhre, K., et al. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab741 · Cited by 30
MzJava: An open source library for mass spectrometry data processing
Horlacher, O., Nikitin, F., Alocci, D., Mariethoz, J., Müller, M., & Lisacek, F. (2015) · Journal of Proteomics · doi:10.1016/j.jprot.2015.06.013 · Cited by 30
“Development methodology The methodology that was employed to develop MzJava follows best practice for scientific computing [32][33][34][35] and is influenced by agile software development, especially test driven development (TDD) [36] and continuous integration (CI) [37].”
MethodReproducibility in Research: Systems, Infrastructure, Culture
Crick, T., Hall, B. A., & Ishtiaq, S. S. (2017) · Journal of Open Research Software · doi:10.5334/jors.73 · Cited by 30
Software engineering for scientific big data analysis
Grüning, B. A., Lampa, S., Vaudel, M., & Blankenberg, D. J. (2019) · GigaScience · doi:10.1093/gigascience/giz054 · Cited by 29
“Increasing attention is being paid to various facets surrounding computational research, including reproducibility [1, 2], reusability [3, 4], and open source efforts and community building [5, 6].”
BackgroundHuman Health and Ecosystem Impacts of Deep Decarbonization of the Energy System
Astudillo, M. F., Vaillancourt, K., Pineau, P., & Amor, B. (2019) · Environmental Science & Technology · doi:10.1021/acs.est.9b04923 · Cited by 31
Hensel, W. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-020-00317-6
Dempsey, W. P., Foster, I. T., Fraser, S. E., & Kesselman, C. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.44d21b86 · Cited by 33
“For example, Sandve et al. (Sandve et al., 2013) make ten suggestions on improving reproducibility via the structured use of existing tools, including GitHub for code repositories and programs rather than manual operations for data manipulation.”
Achievements and challenges in structural bioinformatics and computational biophysics
Samish, I., Bourne, P. E., & Najmanovich, R. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu769 · Cited by 36
A Tool to Encourage Minimum Reporting Guideline Uptake for Data Analysis in Metabolomics
Considine, E. C., & Salek, R. M. (2019) · Metabolites · doi:10.3390/metabo9030043 · Cited by 30
Toward open science in marketing research
Deer, L., Adler, S. J., Datta, H., Mizik, N., & Sarstedt, M. (2024) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2024.12.005 · Cited by 28
The Challenge and Promise of Software Citation for Credit, Identification, Discovery, and Reuse
Niemeyer, K. E., Smith, A. M., & Katz, D. S. (2016) · Journal of Data and Information Quality · doi:10.1145/2968452 · Cited by 28
“Provenance of research results and data requires, among other things, a record of the software used to generate or process that data [Sandve et al. 2013; Wilson et al. 2014].”
MethodNew Methods for Macro-Financial Model Comparison and Policy Analysis
Wieland, V. W., Afanasyeva, E., Kuete, M., & Yoo, J. (2016) · Handbook of macroeconomics · doi:10.1016/bs.hesmac.2016.04.004 · Cited by 28
“For example, Smets and Wouters (2007), De Graeve (2008), Smets and Wouters (2003) and Gelain (2010) use seven macroeconomic time series: real GDP, inflation, consumption, investment, real wages, employment and the short-term nominal interest rate.”
BackgroundMethodInfluentialKarl, J. A., & Stanley, S. K. (2024) · Mindfulness · doi:10.1007/s12671-024-02427-1 · Cited by 28
“Furthermore, as bootstrapping outcomes rely partially on the initial seed, we re-ran this analysis 100 times using different seeds (Sandve et al., 2013).”
Toward Reproducibility in Online Social Network Research
Hutton, L., & Henderson, T. (2015) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2015.2458574 · Cited by 32
“[32], which argues that “all input data, scripts, versions, parameters, and intermediate results should be made publicly and easily accessible”, and by the Yale Law School Roundtable on reproducible research [41]: “When publishing computational results, including statistical analyses and simulation, provide links to t…”
BackgroundThe information science of microbial ecology
Hahn, A. S., Konwar, K. M., Louca, S., Hanson, N. W., & Hallam, S. (2016) · Current Opinion in Microbiology · doi:10.1016/j.mib.2016.04.014 · Cited by 27
“Among the ‘ten commandments of reproducible science’, cloud-computing allows researchers to document how every result was produced, avoid manual data manipulation, archive programs, store raw data and provide public access to developed code and data products [58].”
BackgroundDo journal data sharing mandates work? Life sciences evidence from Dryad
Thelwall, M. A., & Kousha, K. (2017) · Aslib Journal of Information Management · doi:10.1108/ajim-09-2016-0159 · Cited by 27
Streamlining the Inclusion of Computer Experiments In a Research Paper
Hallé, S., Khoury, R., & Awesso, M. (2018) · Computer · doi:10.1109/mc.2018.2876075 · Cited by 27
“Cell (2,2) in Table #2 Cell (9,1) in Table #1 Cell (9,2) in Table #1”
A Guide to Reproducible Code in Ecology and Evolution
Bes, & Cooper, N. (2017) · Natural History Museum, London · Cited by 34
Ten simple rules for quick and dirty scientific programming
Balaban, G., Grytten, I., Rand, K. D., Scheffer, L., & Sandve, G. K. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008549 · Cited by 30
“Although you can be “quick and dirty” in the early phases of your projects, you should ensure quality and reproducibility when you publish [7].”
BackgroundInfluentialSelf-citationGruenstaeudl, M., Gerschler, N., & Borsch, T. (2018) · Life · doi:10.3390/life8030025 · Cited by 27
“Specifically, we aimed to provide a simple, stable and component-based set of software scripts that allows subsequent researchers to replicate our analyses [79] and customize the workflow [78].”
MethodArbas, S. M., Busi, S. B., Queirós, P., de Nies, L., Herold, M., May, P., Wilmes, P., Muller, E. E. L., & Narayanasamy, S. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.666244 · Cited by 28
Prabhu, A., Morrison, S. M., Fox, P., Ma, X., Wong, M. L., Williams, J., McGuinness, K. N., Krivovichev, S. V., Lehnert, K. A., Ralph, J., et al. (2023) · American Mineralogist · doi:10.2138/am-2022-8613 · Cited by 28
“Ongoing research and recommendations on designing FAIR and Open scientific workflows will help improve the methodology of data-driven exploration (Sandve et al. 2013; Kluyver et al. 2016; Prabhu and Fox 2021).”
MethodChallenges and directions in analytical paleobiology
Dillon, E., Dunne, E. M., Womack, T. M., Kouvari, M., Larina, E., Claytor, J. R., Ivkić, A., Juhn, M., Carmona, P. S. M., Robson, S. V., et al. (2023) · Paleobiology · doi:10.1017/pab.2023.3 · Cited by 29
Biase, F. H. (2017) · OMICS A Journal of Integrative Biology · doi:10.1089/omi.2016.0177 · Cited by 42
“For this study, standard practices for reproducible computational research (Sandve et al., 2013) were adopted.”
EnosLib: A Library for Experiment-Driven Research in Distributed Computing
Cherrueau, R.-A., Delavergne, M., van Kempen, A., Lèbre, A., Pertin, D., Balderrama, J. R., Simonet, A., & Simonin, M. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3111159 · Cited by 35
“While several studies have been dealing with the reproducibility challenge in our community [11], [22], [35], [36], [39], progress on how to help researchers conduct experiments on complex distributed software is marginal.”
BackgroundEPIGEN-Brazil Initiative resources: a Latin American imputation panel and the Scientific Workflow
Magalhães, W. C. S., Araujo, N. M., Leal, T. P., de Araújo, G. S., Viriato, P. J. S., Kehdy, F. S. G., Costa, G. N. D. O., Barreto, M. L., Horta, B. L., Lima‐Costa, M. F., et al. (2018) · Genome Research · doi:10.1101/gr.225458.117 · Cited by 28
“Althoughthereareseveralclaims for more transparency and reproducibility of all the scientific process in biomedical literature (Sandve et al. 2013; Kolker et al. 2014; Iqbal et al. 2016), advances”
Leman, J. K., Lyskov, S., Lewis, S. M., Adolf‐Bryfogle, J., Alford, R. F., Barlow, K. A., Ben-Aharon, Z., Farrell, D. P., Fell, J. S., Hansen, W. A., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-27222-7 · Cited by 25
“Guidelines to enhance reproducibility 13,14 are certainly applicable, are outlined in Table 3, and are discussed in detail in an excellent editorial 15 describing the Ten Year Reproducibility Challenge 16 that is published in its own reproducibility journal ReScience C 17 .”
Backgroundslendr: a framework for spatio-temporal population genomic simulations on geographic landscapes
Petr, M., Haller, B. C., Ralph, P. L., & Racimo, F. (2023) · Peer Community Journal · doi:10.24072/pcjournal.354 · Cited by 25
“A key principle in the design of slendr has been reproducibility (Sandve et al., 2013): a complete slendr simulation and analysis workflow can be written as a single R script.”
BackgroundMethodReproducibility and Practical Adoption of GEOBIA with Open-Source Software in Docker Containers
Knoth, C., & Nüst, D. (2017) · Remote Sensing · doi:10.3390/rs9030290 · Cited by 29
“[2]) and builds the European Open Science Cloud [3]; on the publishing side, journals such as Science [4] and Bioinformatics [5] encourage reproducibility; on the research institution side, scientists themselves argue for reproducibility with “Five selfish reasons to work reproducibly” [6], develop guidelines like the…”
BackgroundReplicable Services for Reproducible Research: A Model for Academic Libraries
Sayre, F., & Riegelman, A. (2019) · College & Research Libraries · doi:10.5860/crl.80.2.260 · Cited by 25
“есть аналогичные рекомендации для воспроизводимых вычислительных исследований; подобные рекомендации существуют и для когнитивной нейробиологии [29].”
Scharm, M., Wolkenhauer, O., & Waltemath, D. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btv484 · Cited by 24
“4 Discussion Reproducibility of model-based scientific results has gained increasing attention [20–27].”
BackgroundGheran, B.-F., Villarreal-Narvaez, S., Vatavu, R.-D., & Vanderdonckt, J. M. (2022) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3531073.3531112 · Cited by 25
“Reproducibility is the cornerstone of cumulative knowledge and a requirement to confirm scientific truth [19,48].”
BackgroundQuantifying differences in cell line population dynamics using CellPD
Juarez, E. F., Lau, R., Friedman, S. H., Ghaffarizadeh, A., JONCKHEERE, E. A., Agus, D. B., Mumenthaler, S. M., & Macklin, P. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0337-5 · Cited by 25
“A significant issue in biological experimentation is inter-and intra-laboratory variability [10, 15 – 17 , 32].”
BackgroundCyber Literacy for GIScience: Toward Formalizing Geospatial Computing Education
Shook, E., Bowlick, F. J., Kemp, K. K., Ahlqvist, O., Carbajeles-Dale, P., DiBiase, D., Kim, E., Lathrop, S. A., Ricker, B. A., Rickles, P., et al. (2019) · The Professional Geographer · doi:10.1080/00330124.2018.1518720 · Cited by 27
“Combining these technologies with best practices (Sandve et al. 2013) will help make geocomputational results accessible and reproducible.”
BackgroundTissue specificity of in vitro drug sensitivity
Yao, F., Tonekaboni, S. A. M., Safikhani, Z., Smirnov, P., El-Hachem, N., Freeman, M., Manem, V. S., & Haibe‐Kains, B. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx062 · Cited by 24
“Research reproducibility This study complies with the standards of research reproducibility published by Sandve et al.(52) The datasets are freely available through our PharmacoGx platform.”
Methoduap: reproducible and robust HTS data analysis
Kämpf, C., Specht, M., Scholz, A., Puppel, S.-H., Doose, G., Reiche, K., Schor, J., & Hackermüller, J. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3219-1 · Cited by 24
Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences
Sandia National Labs., A., Heroux, M. A., (NNSA), U. N. N. S. A., Barba, L. A., Parashar, M., Stodden, V., & Taufer, M. (2018) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1481626 · Cited by 24
Where next for the reproducibility agenda in computational biology?
Lewis, J., Breeze, C. E., Charlesworth, J., Maclaren, O. J., & Cooper, J. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0288-x · Cited by 23
“As computation becomes integrated with the experimental and statistical sciences, questions arise about how the classical standards of re-producibility might also apply to computational science [1].”
Backgroundvan Kampen, A. H. C., Mahamune, U., Jongejan, A., van Schaik, B. D. C., Balashova, D., Lashgari, D., Pras‐Raves, M. L., Wever, E. J. M., Dane, A. D., García-Valiente, R., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-52446-8 · Cited by 23
“There are clear advantages of working reproducibly, which has also been argued by other groups (e.g. 4,19,52 .”
Best Practices for Data Sharing in Phylogenetic Research
Cranston, K., Harmon, L. J., O’Leary, M. A., & Lisle, C. R. (2014) · PLoS Currents · doi:10.1371/currents.tol.bf01eff4a6b60ca4825c69293dc59645 · Cited by 23
Replication and generalization in applied neuroimaging
Lerma‐Usabiaga, G., Mukherjee, P., Ren, Z., Perry, L. M., & Wandell, B. A. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116048 · Cited by 25
“In many neuroimaging publications, there is no realistic chance that a reader can repeat the experimental data acquisition or even the computational analyses (Buckheit and Donoho, 1995; Sandve et al., 2013; Wilson et al., 2017).”
Cohen, N. R., Alexander, H., Krinos, A. I., Hu, S. K., & Lampe, R. H. (2022) · Frontiers in Marine Science · doi:10.3389/fmars.2022.867007 · Cited by 27
An Open Framework for the Reproducible Study of the Iterated Prisoner’s Dilemma
Knight, V., Campbell, O., Harper, M., Langner, K. M., Campbell, J., Campbell, T., Carney, A., Chorley, M., Davidson-Pilon, C., Glass, K., et al. (2016) · Journal of Open Research Software · doi:10.5334/jors.125 · Cited by 29
“o dene and contribute new and original Iterated Prisoner’s Dilemma strategies The presented library is partly motivated by an ongoing discussion in the academic community about reproducible research [9, 16, 39, 40], and is: Open: all code is released under an MIT license; Reproducible and well-tested: at the time of w…”
BackgroundInfluentialModel-R: A Framework for Scalable and Reproducible Ecological Niche Modeling
Sánchez‐Tapia, A., de Siqueira, M. F., Lima, R. O., Barros, F. S. M., Gall, G., Gadelha, L. M. R., Jr., da Silva, L. A. E., & Osthoff, C. (2017) · Communications in computer and information science · doi:10.1007/978-3-319-73353-1_15 · Cited by 23
“The inclusion of this feature is motivated by many academic journals recommending that authors of computational studies should also provide the required data sets, tools, and workflows used to generate the results [19] [20] so that reviewers and readers could better validate them.”
BackgroundComputational reproducibility of scientific workflows at extreme scales
Pouchard, L. C., Baldwin, S., Elsethagen, T., Jha, S., Raju, B., Stephan, E., Tang, L., & van Dam, K. K. (2019) · The International Journal of High Performance Computing Applications · doi:10.1177/1094342019839124 · Cited by 23
“Contemporary studies focus on guidelines for publishing papers, appropriate use of statistical methods when reporting results, a more rigorous approach to the design of experiments, better documentation of the experimental process, and making data and source code available (Chirigati et al., 2013; Davison, 2012; Gil e…”
Methodautoman: A Python-Based Automation Framework for Numerical Computing
Ramachandran, P. (2018) · Computing in Science & Engineering · doi:10.1109/mcse.2018.05329818 · Cited by 23
Harjes, J., Link, A., Weibulat, T., Triebel, D., & Rambold, G. (2020) · Database · doi:10.1093/database/baaa059 · Cited by 25
“Compared to requirements of data reusability (19), requirements of ‘repeatability’ of study setups and ‘reproducibility’ of research results go one step further, meeting study operation design, methods applied, data provenience and dataflow details (20, 21).”
BackgroundTen simple rules for making a software tool workflow-ready
Brack, P., Crowther, P., Soiland‐Reyes, S., Owen, S., Lowe, D., Williams, A. R., Groom, Q., Dillen, M., Coppens, F., Grüning, B. A., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009823 · Cited by 26
“Version control of your software, including managing its full dependency graph [49], is critical to maintain reproducibility.”
BackgroundMethodTowards Holistic Continuous Software Performance Assessment
Ferme, V., & Pautasso, C. (2017) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3053600.3053636 · Cited by 22
“2.2 Objective-Driven Performance Testing Enable users to declaratively”
MethodInfluentialEleven quick tips for finding research data
Gregory, K., Khalsa, S. S., Michener, W. K., Psomopoulos, F. E., de Waard, A., & Wu, M. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006038 · Cited by 22
Essential guidelines for computational method benchmarking
Weber, L. M., Saelens, W., Cannoodt, R., Soneson, C., Hapfelmeier, A., Gardner, P. P., Boulesteix, A.-L., Saeys, Y., & Robinson, M. D. (2019) · Genome Biology · doi:10.1186/s13059-019-1738-8 · Cited by 21
Sensor Reproducibility Analysis: Challenges and Potential Solutions
Sekhar, P. K., Billey, W., Begay, M., Thomas, B., Woody, C., & Soundappan, T. (2024) · ECS Sensors Plus · doi:10.1149/2754-2726/ad9936 · Cited by 23
“Sandve et al. 58 provided ten simple rules to conduct reproducible computational research.”
A Guide to Bayesian Optimization in Bioprocess Engineering
Siska, M., Pajak, E., Rosenthal, K., del Rio‐Chanona, E. A., von Lieres, E., & Helleckes, L. M. (2026) · Biotechnology and Bioengineering · doi:10.1002/bit.70129 · Cited by 24
Twelve quick tips for designing sound dynamical models for bioprocesses
Mairet, F., & Olivier Bernard, A. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007222 · Cited by 21
“More tips concerning computational aspects can be found in [2, 3].”
BackgroundLao‐Martil, D., Verhagen, K. J. A., Schmitz, J. P. J., Teusink, B., Wahl, A., & van Riel, N. A. W. (2022) · Metabolites · doi:10.3390/metabo12010074 · Cited by 24
Impact of analytic provenance in genome analysis
Morrison, S. S., Pyzh, R., Jeon, M. S., Amaro, C., Roig, F. J., Baker‐Austin, C., Oliver, J. D., & Gibas, C. (2014) · BMC Genomics · doi:10.1186/1471-2164-15-s8-s1 · Cited by 22
“Systems such as Galaxy [21], Taverna [22], and MOLGENIS [23], among others, have made reproducible workflows more accessible to users of bioinformatics software, and many workflow systems now include the means for tracking analytic provenance,[19]but it is clear from our survey of the literature that these innovations…”
BackgroundHuman Genome Variation and the Concept of Genotype Networks
Dall’Olio, G. M., Bertranpetit, J., Wagner, A., & Laayouni, H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099424 · Cited by 22
“Following the best practices described in [33] the whole project presented in this manuscript, including the raw data, the scripts to produce plots and analysis, and a versioned log of all the commands used, are available at https://bitbucket.”
MethodComputational Reproducibility via Containers in Social PsychologyPreprint
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/mf82t · Cited by 21
Reproducible Research Using Biomodels
Mendes, P. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0498-z · Cited by 20
Recent advances in biomedical simulations: a manifesto for model engineering
Hellerstein, J. L., Gu, S., Choi, K., & Sauro, H. M. (2019) · F1000Research · doi:10.12688/f1000research.15997.1 · Cited by 20
Why do biomedical researchers learn to program? An exploratory investigation
Deardorff, A. (2020) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2020.819 · Cited by 20
“It is no longer possible to simply document a manual data-cleaning-and-analysis workflow; instead, new guidelines recommend that researchers automate their steps with programming scripts and version control [2–4].”
BackgroundA reporting format for field measurements of soil respiration
Bond‐Lamberty, B., Christianson, D., Crystal‐Ornelas, R., Mathes, K. C., & Pennington, S. (2021) · Ecological Informatics · doi:10.1016/j.ecoinf.2021.101280 · Cited by 20
Sharing is caring: Ethical implications of transparent research in psychology.
Bosma, C. M., & Granger, A. M. (2022) · American Psychologist · doi:10.1037/amp0001002 · Cited by 24
Error tight: Exercises for lab groups to prevent research mistakes.
Strand, J. F. (2023) · Psychological Methods · doi:10.1037/met0000547 · Cited by 21
Why sharing data and code during peer review can enhance behavioral ecology research
Fernández‐Juricic, E. (2021) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-021-03036-x · Cited by 20
“) used to process raw data, simulate data, run statistical analyses, and develop figures so that readers can generate the same outcomes as those presented in the paper (Sandve et al. 2013; Ilhe et al. 2017; Perkel 2019; Powers and Hampton 2019).”
MethodAdvancing code sharing in the computational biology community
Cadwallader, L., Gabhann, F. M., Papin, J. A., & Pitzer, V. E. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010193 · Cited by 23
Pesce, F., & Cheungpasitporn, W. (2025) · Renal Failure · doi:10.1080/0886022x.2025.2581933 · Cited by 21
“…practice is a cornerstone of reproducible research, ensuring that the methods and logic behind a result can be independently verified and extended [19]. it stands in stark contrast to the proprietary ‘black box’ nature of many commercial medical software products and fosters a more trustworthy…”
Revisiting reproducibility in transportation simulation studies
Riehl, K., Kouvelas, A., & Makridis, M. (2025) · European Transport Research Review · doi:10.1186/s12544-025-00718-9 · Cited by 22
“In other fields, such as computational science it already has become a common practice [9, 10, 15].”
The Quality of Methods Reporting in Parasitology Experiments
Flórez-Vargas, O., Bramhall, M., Noyes, H. A., Cruickshank, S. M., Stevens, R., & Brass, A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101131 · Cited by 21
“scientific community there is a growing concern over the related problem of lack of reproducibility [3,4].”
BackgroundAssessing therapy response in patient-derived xenografts
Ortmann, J., Rampášek, L., Tai, E., Mer, A. S., Shi, R., Stewart, E., Mascaux, C., Fares, A. F., Pham, N., Beri, G., et al. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.abf4969 · Cited by 19
Choi, Y.-D., Roy, B., Nguyen, J., Ahmad, R., Maghami, I., Nassar, A., Li, Z., Castronova, A. M., Malik, T., Wang, S., et al. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105760 · Cited by 20
Nguyen, S. L., Greenberg, J. W., Wang, H., Collaer, B. W., Wang, J., & Petroff, M. G. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218270 · Cited by 20
Implementing Computational Reproducibility in the Whole Tale Environment
Chard, K., Gaffney, N. I., Jones, M. B., Kowalik, K., Ludäscher, B., Nabrzyski, J., Stodden, V., Taylor, I., Turk, M., & Willis, C. (2019) · International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS) · doi:10.1145/3322790.3330594 · Cited by 19
“Scripting the computational steps is important to expose the implementation of the reasoning behind a proposed claim [34-35].”
BackgroundThe R Language as a Tool for Biometeorological Research
Charalampopoulos, I. (2020) · Atmosphere · doi:10.3390/atmos11070682 · Cited by 19
“Data analysis can yield results if you complete the process correctly, but the value of the results is directly a ff ected by the quality of the inputs [48–50].”
BackgroundA meta-epidemiological assessment of transparency indicators of infectious disease models
Zavalis, E. A., & Ioannidis, J. P. A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0275380 · Cited by 22
“Yet, there is a growing literature of recommendations and tutorials for researchers and other stakeholders (41–44), plus the EPIFORGE guidelines (45) which are guidelines for the reporting of epidemic forecasting and prediction research.”
Heise, V., Holman, C., Lo, H., Lyras, E. M., Adkins, M. C., Aquino, M. R. J., Bougioukas, K. I., Bray, K. O., Gajos, M., Guō, X., et al. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010750 · Cited by 20
Ten Simple Rules for Experiments’ Provenance
Kazic, T. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004384 · Cited by 19
“So, in the spirit of the rules for the provenance of computational experiments and instrument data [7,8], I offer ten simple rules for interdisciplinary collaborations on provenance capture.”
Simulation Foundry: Automated and F.A.I.R. Molecular Modeling
Gygli, G., & Pleiss, J. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00018 · Cited by 18
ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks
Samuel, S., & König‐Ries, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80960-7_12 · Cited by 18
Ten (not so) simple rules for clinical trial data-sharing
Pellen, C., Louarn, A. L., Spurrier‐Bernard, G., Decullier, É., Chrétien, J., Rosenthal, É., Goff, G. L., Moher, D., Ioannidis, J. P. A., & Naudet, F. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010879 · Cited by 18
“Statistical codes can also easily be shared with fewer restrictions because they do not contain individual data, but they are not so easy to understand and reuse; the name and version of the software and libraries used should be specified along with all other details to make the research reproducible [23].”
BackgroundSafikhani, Z., El-Hachem, N., Smirnov, P., Freeman, M., Goldenberg, A., Birkbak, N. J., Beck, A. H., Aerts, H. J. W. L., Quackenbush, J. F., & Haibe‐Kains, B. (2016) · Nature · doi:10.1038/nature19839 · Cited by 18
Eleven quick tips for architecting biomedical informatics workflows with cloud computing
Cole, B. S., & Moore, J. H. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005994 · Cited by 17
Marwick, B., Barton, C. M., Bates, L., Bollwerk, E., Bocinsky, R. K., Carter, A. K., Conrad, C., Costa, S., Crema, E. R., Davies, B., et al. (2017) · doi:10.31235/osf.io/72n8g · Cited by 17
EviewsR: An R Package for Dynamic and Reproducible Research Using EViews, R, R Markdown and Quarto
Mati, S., Civcir, İ., & Abba, S. I. (2023) · The R Journal · doi:10.32614/rj-2023-045 · Cited by 21
Knight, V., Harper, M., Glynatsi, N. E., & Campbell, O. (2018) · PLoS ONE · doi:10.1371/journal.pone.0204981 · Cited by 22
“All source code for this work has been written in a sustainable manner: it is open source, under version control and tested which ensures that all results can be reproduced [45, 49, 57].”
Background
Metadata
- Authors
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E.
- Affiliations
- University of Oslo (NO)
- Pennsylvania State University (US)
- Emory University (US)
- Oslo University Hospital (NO)
- Corresponding author
- Sandve, Geir Kjetil
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-24
- Volume
- 9
- Issue
- 10
- Pages
- e1003285
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3812051
- OpenAlex
- W2036318837
- Cited by
- 880
- References
- 27
- Citations by year
- 2026: 176 · 2025: 60 · 2024: 52 · 2023: 50 · 2022: 58 · 2021: 67 · 2020: 63 · 2019: 88 · 2018: 76 · 2017: 64 · 2016: 56 · 2015: 42 · 2014: 24 · 2013: 3
- FWCI
- 91.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (0850103)
- Pennsylvania Department of Health (HG004909)
- University of Pennsylvania
- Pennsylvania State University
- National Institutes of Health (HG006620)
- Huck Institutes of the Life Sciences
- Institute for Computational and Data Sciences, Pennsylvania State University
- National Human Genome Research Institute (HG005133, U41HG006620, R21HG005133, R01HG004909, HG006620, HG004909)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducible research, computational reproducibility, research practices, bioinformatics analysis, research transparency, version control, software dependency management, workflow management, research data management, research culture, research evaluation, research dissemination, scholarly communication
- MeSH terms
- Reproducibility of Results, Computational Biology, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.