Ten Simple Rules for Reproducible Computational Research
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Replication is the cornerstone of a cumulative science [1]. However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2]. As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3]. This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4]. The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8], [9]. This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3]. We want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run. We further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities. In a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys. As a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible. With this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6], [14]. We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your researc
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival1,11,12, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
From The FAIR Guiding Principles for scientific data management and stewardship (2016) · cites this paper as [12]
In the citing paper
“In a recent publication reviewing the automated citation screening methods reported for systematic reviews, the number of studies that met the current standards for reproducible software engineering experiments was low [29, 30, 36]; of 33 studies reporting approaches to automated screening, no study provided all the criteria for reproducible reporting [36].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundMinimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundGoecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome Biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,453 · cited 1×
“For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19], the GenePattern Word add-in [4], and Galaxy Pages [20].”
MethodSelf-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundReich, M., Liefeld, T., Gould, J., Lerner, J., Tamayo, P., & Mesirov, J. P. (2006) · Nature Genetics · doi:10.1038/ng0506-500 · Cited by 2,032
Galaxy: A platform for interactive large-scale genome analysis
Giardine, B., Riemer, C., Hardison, R. C., Burhans, R., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,026 · cited 2×
“If you are instead performing your analyses through an integrated framework (such as Gene-Pattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules.”
BackgroundSelf-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“…through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8,9].”
ResultTaverna: a tool for the composition and enactment of bioinformatics workflows
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M. R., Wipat, A., et al. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth361 · Cited by 1,602 · cited 2×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 3×
“In order to foster such aspects, it has been suggested that the scientific community needs to develop a ‘‘culture of reproducibility’’ for computational science, and to require it for published claims [3].”
BackgroundPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222 · cited 1×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
Background
Cited by
Items 701–800 of 1,001
Ankenbrand, M. J. (2018) · Online Publication Service of Würzburg University (Würzburg University)
Nguyen, S. L., Greenberg, J. W., Wang, H., Collaer, B. W., Wang, J., & Petroff, M. G. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/503292
Cirillo (2019) · Research Publications (Maastricht University) · doi:10.26481/dis.20190118ec
pyscses: a PYthon Space-Charge Site-Explicit Solver
Wellock, G., & Morgan, B. J. (2019) · The Journal of Open Source Software · doi:10.21105/joss.01209
Integration of Omics Data to Identify Cancer-Related MicroRNA
Cascione, L. (2019) · Methods in molecular biology · doi:10.1007/978-1-0716-5392-0_12
Reproducibilidad científica: ¿Qué es y por qué debemos interesarnos en geo-ciencias?
Ballari, D. (2019) · Revista Cartográfica · doi:10.35424/rcarto.i97.179
Redaktionen, R. (2019) · Revy · doi:10.22439/revy.v42i4.5880
Autonomous Management of Internet of Things Technologies
Hernandez, A. B. (2019) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.56557
Computeren som værktøj i forskning: er vi gode håndværkere?
Jensen, T. L., & Knudsen, E. V. (2019) · Revy · doi:10.22439/revy.v42i4.5868
Replicable services for reproducible research: a model for academic libraries
Sayre, F., & Riegelman, A. (2020) · Bibliosphere · doi:10.20913/1815-3186-2019-4-33-45
Reproducibilidad en AGILE: Experiencias, logros y recomendaciones
Granell, C., Hofer, B., Nüst, D., Ostermann, F., & Šileryté, R. (2020) · Revista Cartográfica · doi:10.35424/rcarto.i100.668
Hesselbarth, M. H. K. (2020) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-7959
Curtis, C. (2020) · Chemical Engineering Education · doi:10.18260/2-1-370.660-119316
The Solution to Science's Replication CrisisPreprint
Knuteson, B. (2016) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1609.03223 · Cited by 1
“[78] Sandve GK, Nekrutenko A, Taylor J, Hovig E.”
Preregistration, Data Sharing, and Other Salutary Behaviors
Bausell, R. B. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197536537.003.0011
Hagevall, A. E., & Wikström, C. (2021)
Fatty acid composition in four serum lipid fractions and the pathogenesis of diabetes
Johnston, L. (2017) · TSpace (University of Toronto)
Computational Workflow Optimization for Magnetic Fluctuation Measurements of 3D Nano-Tetrapods
Pieper, J. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5760260
CCG Platform - Body of Knowledge: Review of Good Practice
Usher, W., Beltramo, A., Gardumi, F., Martin, V., & Petrarulo, L. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5911545
Ortega-Laurel, C., Pérez-Martínez, F., Franco, V. A. S., & López-Maldonado, G. (2022) · Journal of Computational Systems and ICTs · doi:10.35429/jcsi.2022.21.8.1.10
“…benefit from investments in the generation and collection of data, especially in health, whatever the purpose from which they were derived (Kjetil-Sandve et al., 2013), (Casino, 2019), (Alterovitz et al., 2020), since depending on the aspect in which the data of a given application is framed or…”
Open Access and Research Reproducibility in Biomedical Sciences
Assefa, S. G. (2022) · Advances in knowledge acquisition, transfer, and management book series/Advances in knowledge acquisition, transfer and management book series · doi:10.4018/978-1-7998-9805-4.ch006
Peer Review #2 of "Electronic laboratory notebooks in a public–private partnership (v0.2)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.83v0.2/reviews/2
Low Template, Low Level or Low Copy Number DNA
(2016) · CRC Press eBooks · doi:10.1201/b19654-7
Frery, A. C. (2018) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2018.2881327
Peer Review #2 of "Electronic laboratory notebooks in a public–private partnership (v0.1)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.83v0.1/reviews/2
Peer Review #1 of "Electronic laboratory notebooks in a public–private partnership (v0.1)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.83v0.1/reviews/1
Peer Review #3 of "Electronic laboratory notebooks in a public–private partnership (v0.2)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.83v0.2/reviews/3
Peer Review #1 of "Sustainable computational science: the ReScience initiative (v0.1)"
(2017) · PeerJ Computer Science · doi:10.7287/peerj-cs.142v0.1/reviews/1
Peer Review #3 of "Electronic laboratory notebooks in a public–private partnership (v0.1)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.83v0.1/reviews/3
Peer Review #2 of "Sustainable computational science: the ReScience initiative (v0.1)"
(2017) · PeerJ Computer Science · doi:10.7287/peerj-cs.142v0.1/reviews/2
Frery, A. C. (2014) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2014.2299891
Peer Review #1 of "Electronic laboratory notebooks in a public–private partnership (v0.2)"
(2016) · PeerJ Computer Science · doi:10.7287/peerj-cs.83v0.2/reviews/1
Creating interactive scientific publications using bindingsPreprint
Konkol, M., Kray, C., & Suleiman, J. (2018) · doi:10.31223/osf.io/jtd8c
(2019) · Elsevier eBooks · doi:10.1016/b978-0-12-817443-2.00025-2
Toward a FAIR Reproducible Research
Bontemps, C., & Orozco, V. (2020) · Toulouse Capitole Publications (University Toulouse 1 Capitole)
Brooks‐Kieffer, J., & Albin, T. (2022) · The Journal of Play in Adulthood · doi:10.5920/jpa.1030 · Cited by 1
Computation of output parameters of a submicrosecond electron accelerator
Egorov, I. S., Serebrennikov, M., & Poloskov, A. V. (2022) · International Congress on Energy Fluxes and Radiation Effects (EFRE) · doi:10.56761/efre2022.s6-o-029605
Open-Source Workflows for Reproducible Molecular Simulation
Fothergill, J. W. (2022) · ScholarWorks (Boise State University) · doi:10.18122/td.1968.boisestate
Can Blockchain Technology Change Contemporary Medicine as It is Currently Understood?
Román-Belmonte, J. M., la Corte‐Rodríguez, H. D., & Rodriguez-Merchan, E. C. (2023) · Integrated science · doi:10.1007/978-3-031-14591-9_2
Prabhu, A., & Fox, P. (2023) · Encyclopedia of earth sciences series/Encyclopedia of earth sciences · doi:10.1007/978-3-030-85040-1_277
Reusing Biomedical Data as Agreed – Towards Structured Metadata for Data Use Agreements
Bönisch, C., Hanß, S., Spicher, N., Sax, U., & Krefting, D. (2023) · Studies in health technology and informatics · doi:10.3233/shti230690
Cortie, C. H., Garne, D., Parker-Newlyn, L. A., Ivers, R. G., Mullan, J., Mansfield, K. J., & Bonney, A. D. (2023) · Australian Journal of Rural Health · doi:10.1111/ajr.13053
A recipe for extremely reproducible enrichment analysis (and why we need it)Preprint
Bora, A., & Ziemann, M. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/r6kxg
Genetic stratification based on biological networks in autism spectrum disorders
Kim, Y.-M. (2020) · theses.fr (ABES) · doi:10.70675/1403a354za2d4z4124zbbefz968fa3d238f4
In Silico Evaluation and Prediction of Pesticide Supported by Reproducible Evolutionary Workflows
de Oliveira, A. R. M., de Faria, F. F., Cruz, P. V., Sampaio, J. D. O., & da Cruz, S. M. S. (2024) · Springer eBooks · doi:10.1007/978-3-031-49740-7_6
Chen, D., Jutkowitz, E., Iosepovici, S. L., Lin, J. C. H., & Gross, A. L. (2021) · Figshare · doi:10.6084/m9.figshare.16864735
Reproducible WorkflowPreprint
Prabhu, A., & Fox, P. (2020) · Encyclopedia of Mathematical Geosciences · doi:10.48550/arxiv.2012.13427 · Cited by 3
“These levels are defined based on either the specific domain of research or the scientific approach (Stodden 2011, 2013; Goodman et. al 2016) 2.”
(2024) · Elsevier eBooks · doi:10.1016/b978-0-443-13829-4.00015-5
(2024) · Environmental science and engineering · doi:10.46632/ese/2/3/4
Smith, A. K., Davis, E., Derti, A., & Chasalow, S. D. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/4rszv
Computational Pipelines and Workflows in Bioinformatics
Milton, M. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95502-7.00023-3
Reprodutibilidade de Experimentos em Redes de Computadores através do Catálogo de Dados RNP
Zanotelli, V. F., Damasceno, N. L., Vianna, A. A., Araújo, G. H. D., Hernandez, M. P., Comarela, G., Martinello, M., & Rocha, A. A. D. A. (2024) · doi:10.5753/wgrs.2024.3264
AnVILWorkflow: A runnable workflow package for Cloud-implemented bioinformatics analysis pipelines
Oh, S., Gravel-Pucillo, K., Ramos, M., Schatz, M. C., Davis, S. R., Carey, V. J., Morgan, M. T., & Waldron, L. (2024) · F1000Research · doi:10.12688/f1000research.155449.1
Programming in the Life Sciences #8: coding standards
Willighagen, E. (2013) · doi:10.59350/ckryb-b4v19
Konkol, M., Labuce, A., Domisch, S., Buurman, M., & Bremerich, V. (2025) · Open Research Europe · doi:10.12688/openreseurope.20221.1
“Numerous tools, guidelines, and best practices have been developed to support this trend, including the FAIR (Findability, Accessibility, Interoperability, and Reusability) principles for data 17 and software 18 and the “Ten Simple Rules for Computational Reproducible Research” 19 .”
Architectural Proposal for Reproducible, Standardized Deep Learning Research
Lübbering, M., Shah, V., Chatterjee, M., Priya, P., Soliman, O. I. I., & Sifa, R. (2025) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c65153.2025.00021 · Cited by 1
“We ) [19], the checklists of premier venues, model cards [17], and the HPC reproducibility rules [22]. following the inversion of control (IoC) paradigm [16].”
Marwick, B. (2025) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/fdhvb_v4
A Virtual Laboratory for Managing Computational Experiments
Adamidi, E. S., Deligiannis, P., Foutris, N., & Vergoulis, T. (2025) · International Conference on Scalable Scientific Data Management (SSDBM) · doi:10.1145/3733723.3733743
Kotowicz, M., Fengler, S., Kurkowsky, B., Meyer-Berhorn, A., Moretti, E., Blersch, J., Shumanska, M., Schmidt, G., Kreye, J., van Hoof, S., et al. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326678
“Similarly, manual integration of data from multiple sources lacks standardization, has limited scalability, and can hinder early error detection [18].”
Konkol, M., Labuce, A., Domisch, S., Buurman, M., & Bremerich, V. (2025) · Open Research Europe · doi:10.12688/openreseurope.20221.2
GrainStat: A Python Framework for Automated Grain Microstructure Analysis in Materials SciencePreprint
Kokati, P. (2025) · Engineering Archive (engrXiv) · doi:10.31224/4960
De nouveaux outils informatiques pour une activité scientifique ouverte (2)
Marwick, B. (2015) · The Conversation · doi:10.64628/aak.p7yp347ar
Six Elements of Data-Driven Precision Horticulture
Alavi, M., Robson-Williams, R., Lin, H. T., Bentall, D., & Jesson, L. K. (2025) · Progress in precision agriculture · doi:10.1007/978-3-031-96534-0_10
Chaotic and Resonant Behaviors in a Coupled Spring–Pendulum System: A Numerical StudyPreprint
Saefan, J., Khoiri, N., Marwoto, P., & Nugroho, S. E. (2025) · Research Square · doi:10.21203/rs.3.rs-7586848/v1
Konkol, M., Labuce, A., Domisch, S., Buurman, M., & Bremerich, V. (2025) · Open Research Europe · doi:10.12688/openreseurope.20221.3
Davis‐Dusenbery, B. N., Fisher, C. R., Beck, R., & Worman, Z. (2025) · Oxford University Press eBooks · doi:10.1093/med/9780192867636.003.0018
Enhancing Feasibility and Integrity in Health Research: The S-FIPR PlatformPreprint
Dârzan, G. (2025) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/gkmj2_v1
Ten simple rules for PIs to integrate Research Software Engineering into their research groupPreprint
Allen, S. M., Hong, N. C., Druskat, S., Hodges, T. M., Katz, D. S., Linxweiler, J., Löffler, F., Grunske, L., Seibold, H., Thiele, J. P., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.20217 · Cited by 1
In Pursuit of Total ReproducibilityPreprint
Beber, M. E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.11635
“A situation that remains unchanged despite the clear benefits at an institutional level (Sandve et al. 2013).”
ADEPT: A Pedagogical Framework for Integrating Agentic AI with Deterministic Scientific Workflows
George, A., USDOE, Pena, A. B., Agarwal, K., Mejía-Rodríguez, D., Samantray, S., Kim, H. S., Rice, P. S., Baer, M. D., Jacob, B., et al. (2025) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/3006464 · Cited by 3
DESALEGN, A., Katakweba, A. A. S., Kessy, S. T., Ngoda, U. R., Christopher, S., & Rija, A. A. (2025) · African Journal of Ecology · doi:10.1111/aje.70132
Popereshnyak, S., Zhebka, V., Лащевська, Н. О., & Герцюк, М. (2025) · Наука і техніка сьогодні · doi:10.52058/2786-6025-2025-11(52)-2597-2611
demografr : A toolkit for simulation-based inference in population geneticsPreprint
Petr, M., Pötzsch, I. M., & Racimo, F. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.18.694482
“Although seemingly a technicality, less manual work, more automation, and a tighter link between code and data dramatically decrease the number of software bugs [28].”
Schloss Dagstuhl - Jahresbericht / Annual Report 2018
Informatik, S. D. L.-Z. F. (2019) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dagannrep.2018
A Virtual Laboratory for Managing Computational ExperimentsPreprint
Adamidi, E. S., Deligiannis, P., Foutris, N., & Vergoulis, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.01197
“As highlighted by Sandve et al. [10] and Peng [9], computational reproducibility demands meticulous management of workflows, configurations, and performance data.”
Additional file 1 of Research perspectives on animal health in the era of artificial intelligence
Ezanno, P., Picault, S., Beaunée, G., Bailly, X., Muñoz, F., Duboz, R., Monod, H., & Guégan, J. (2021) · Figshare · doi:10.6084/m9.figshare.14175914
Marini, F., Ludt, A., Linke, J., & Strauch, K. (2021) · Figshare · doi:10.6084/m9.figshare.17468606
Aiding researchers making their computational experiments reproducible
Costa, L. (2025) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 2
Belghait, F. (2020) · Espace ÉTS (ETS)
Erratic Patterns : Unravelling the Cultural Transfers of Library Classifiers
Lindkvist, K. (2023) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Addressing researcher degrees of freedom in applications, methodological research, and teaching
Mandl, M. M. (2025) · Electronic Theses of LMU Munich (Ludwig-Maximilians-Universität München) · doi:10.5282/edoc.35690
Programming in the Life Sciences #8: coding standards
Willighagen, E. (2013) · doi:10.59350/kk6b3-d3k79
Ramirez, C. E. (2025) · Iowa Research Online (The University of Iowa) · doi:10.25820/etd.008071
Adetunji, S. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.14523
Programming in the Life Sciences #23: research output for the future
Willighagen, E. (2018) · doi:10.59350/2jr82-qp897
Programming in the Life Sciences #23: research output for the future
Willighagen, E. (2018) · doi:10.59350/3aha2-9ct41
A Power Query alkalmazása mezőgazdasági adatok kezelésére
György, H., & Zoltán, F. (2025) · SZTE Publicatio Repozitórium (University of Szeged)
György, H., Krisztián, K., Edit, M., & Tamás, M. (2025) · SZTE Publicatio Repozitórium (University of Szeged)
Metadata
- Authors
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E.
- Affiliations
- University of Oslo (NO)
- Pennsylvania State University (US)
- Emory University (US)
- Oslo University Hospital (NO)
- Corresponding author
- Sandve, Geir Kjetil
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-24
- Volume
- 9
- Issue
- 10
- Pages
- e1003285
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3812051
- OpenAlex
- W2036318837
- Cited by
- 880
- References
- 27
- Citations by year
- 2026: 176 · 2025: 60 · 2024: 52 · 2023: 50 · 2022: 58 · 2021: 67 · 2020: 63 · 2019: 88 · 2018: 76 · 2017: 64 · 2016: 56 · 2015: 42 · 2014: 24 · 2013: 3
- FWCI
- 91.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (0850103)
- Pennsylvania Department of Health (HG004909)
- University of Pennsylvania
- Pennsylvania State University
- National Institutes of Health (HG006620)
- Huck Institutes of the Life Sciences
- Institute for Computational and Data Sciences, Pennsylvania State University
- National Human Genome Research Institute (HG005133, U41HG006620, R21HG005133, R01HG004909, HG006620, HG004909)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducible research, computational reproducibility, research practices, bioinformatics analysis, research transparency, version control, software dependency management, workflow management, research data management, research culture, research evaluation, research dissemination, scholarly communication
- MeSH terms
- Reproducibility of Results, Computational Biology, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.