Ten Simple Rules for Reproducible Computational Research
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Replication is the cornerstone of a cumulative science [1]. However, new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research [2]. As full replication of studies on independently collected data is often not feasible, there has recently been a call for reproducible research as an attainable minimum standard for assessing the value of scientific claims [3]. This requires that papers in experimental science describe the results and provide a sufficiently clear protocol to allow successful repetition and extension of analyses based on original data [4]. The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8], [9]. This has led to discussions on how individual researchers, institutions, funding bodies, and journals can establish routines that increase transparency and reproducibility. In order to foster such aspects, it has been suggested that the scientific community needs to develop a “culture of reproducibility” for computational science, and to require it for published claims [3]. We want to emphasize that reproducibility is not only a moral responsibility with respect to the scientific field, but that a lack of reproducibility can also be a burden for you as an individual researcher. As an example, a good practice of reproducibility is necessary in order to allow previously developed methodology to be effectively applied on new data, or to allow reuse of code and results for new projects. In other words, good habits of reproducibility may actually turn out to be a time-saver in the longer run. We further note that reproducibility is just as much about the habits that ensure reproducible research as the technologies that can make these processes efficient and realistic. Each of the following ten rules captures a specific aspect of reproducibility, and discusses what is needed in terms of information handling and tracking of procedures. If you are taking a bare-bones approach to bioinformatics analysis, i.e., running various custom scripts from the command line, you will probably need to handle each rule explicitly. If you are instead performing your analyses through an integrated framework (such as GenePattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules. What is needed on your part is then merely the knowledge of how to exploit these existing possibilities. In a pragmatic setting, with publication pressure and deadlines, one may face the need to make a trade-off between the ideals of reproducibility and the need to get the research out while it is still relevant. This trade-off becomes more important when considering that a large part of the analyses being tried out never end up yielding any results. However, frequently one will, with the wisdom of hindsight, contemplate the missed opportunity to ensure reproducibility, as it may already be too late to take the necessary notes from memory (or at least much more difficult than to do it while underway). We believe that the rewards of reproducibility will compensate for the risk of having spent valuable time developing an annotated catalog of analyses that turned out as blind alleys. As a minimal requirement, you should at least be able to reproduce the results yourself. This would satisfy the most basic requirements of sound research, allowing any substantial future questioning of the research to be met with a precise explanation. Although it may sound like a very weak requirement, even this level of reproducibility will often require a certain level of care in order to be met. There will for a given analysis be an exponential number of possible combinations of software versions, parameter values, pre-processing steps, and so on, meaning that a failure to take notes may make exact reproduction essentially impossible. With this basic level of reproducibility in place, there is much more that can be wished for. An obvious extension is to go from a level where you can reproduce results in case of a critical situation to a level where you can practically and routinely reuse your previous work and increase your productivity. A second extension is to ensure that peers have a practical possibility of reproducing your results, which can lead to increased trust in, interest for, and citations of your work [6], [14]. We here present ten simple rules for reproducibility of computational research. These rules can be at your disposal for whenever you want to make your researc
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there have been a number of recent, often domain-focused publications advocating for specific improvements in practices relating to data management and archival1,11,12, FAIR differs in that it describes concise, domain-independent, high-level principles that can be applied to a wide range of scholarly outputs.”
From The FAIR Guiding Principles for scientific data management and stewardship (2016) · cites this paper as [12]
In the citing paper
“In a recent publication reviewing the automated citation screening methods reported for systematic reviews, the number of studies that met the current standards for reproducible software engineering experiments was low [29, 30, 36]; of 33 studies reporting approaches to automated screening, no study provided all the criteria for reproducible reporting [36].”
From A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? (2019) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gene Expression Omnibus: NCBI gene expression and hybridization array data repository
Edgar, R. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.207 · Cited by 13,288 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundMinimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · cited 1×
“Various solutions have now become available to make data sharing more convenient, standardized, and accessible in particular domains, such as for gene expression data [21– 23].”
BackgroundGoecks, J., Nekrutenko, A., Taylor, J., & The Galaxy Team (2010) · Genome Biology · doi:10.1186/gb-2010-11-8-r86 · Cited by 3,453 · cited 1×
“For an even tighter integration, there are tools available to help integrate reproducible analyses directly into textual documents, such as Sweave [19], the GenePattern Word add-in [4], and Galaxy Pages [20].”
MethodSelf-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundReich, M., Liefeld, T., Gould, J., Lerner, J., Tamayo, P., & Mesirov, J. P. (2006) · Nature Genetics · doi:10.1038/ng0506-500 · Cited by 2,032
Galaxy: A platform for interactive large-scale genome analysis
Giardine, B., Riemer, C., Hardison, R. C., Burhans, R., Elnitski, L., Shah, P., Zhang, Y., Blankenberg, D., Albert, I., Taylor, J., et al. (2005) · Genome Research · doi:10.1101/gr.4086505 · Cited by 2,026 · cited 2×
“If you are instead performing your analyses through an integrated framework (such as Gene-Pattern [10], Galaxy [11], LONI pipeline [12], or Taverna [13]), the system may already provide full or partial support for most of the rules.”
BackgroundSelf-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“…through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], and through a high number of failing clinical trials [8,9].”
ResultTaverna: a tool for the composition and enactment of bioinformatics workflows
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M. R., Wipat, A., et al. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth361 · Cited by 1,602 · cited 2×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
BackgroundReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 3×
“In order to foster such aspects, it has been suggested that the scientific community needs to develop a ‘‘culture of reproducibility’’ for computational science, and to require it for published claims [3].”
BackgroundPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G. B., Good, J., et al. (2005) · Scientific Programming · doi:10.1155/2005/128026 · Cited by 1,222 · cited 1×
“Such executable descriptions [10] might come in the form of simple shell scripts or makefiles [15,16] at the command line, or in the form of stored workflows in a workflow management system [10,11,13,17,18].”
Background
Cited by
Items 101–200 of 1,001
Computational reproducibility of Jupyter notebooks from biomedical publications
Samuel, S., & Mietchen, D. (2024) · GigaScience · doi:10.1093/gigascience/giad113 · Cited by 93
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77
“1 A nn ua l R ev ie w o g A cc es s pr ov id”
BackgroundResultThe nlrx r package: A next‐generation framework for reproducible NetLogo model analyses
Salecker, J., Sciaini, M., Meyer, K., & Wiegand, K. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13286 · Cited by 91
“(b) it enables reproducible research (Sandve et al., 2013) by storing a simulation experiment in one single r object (including parameter input matrix, applied simulation method and simulation results) and allows easy sharing of code and results among colleagues, with students and for pub‐ lication.”
BackgroundChallenges and opportunities in sharing microbiome data and analyses
Huttenhower, C., Finn, R. D., & McHardy, A. C. (2023) · Nature Microbiology · doi:10.1038/s41564-023-01484-x · Cited by 78
“…Perspective https://doi.org/10.1038/s41564-023-01484-x with programs used, versions and parameter settings, avoiding manual data manipulation 10 , and providing associated code under open-source licences in commonly used repositories such as GitHub, and archiving them in permanent, public…”
BackgroundMethodInfluentialAyllón, D., Railsback, S. F., Gallagher, C. A., Augusiak, J., Baveco, J. M., Berger, U., Charles, S., Martin, R., Focks, A., Galić, N., et al. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104932 · Cited by 73
Reproducible bioinformatics project: a community for reproducible bioinformatics analysis pipelines
Kulkarni, N., Alessandrì, L., Panero, R., Arigoni, M., Olivero, M., Ferrero, G., Cordero, F., Beccuti, M., & Calogero, R. A. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2296-x · Cited by 60
“The transition between development and stable branch is done when a module (R function(s)/docker container(s)) fulfills the 10 rules suggested by Sandve [5] for the good bioinformatics practice (Table 1).”
BackgroundInfluentialTen Simple Rules for Reducing Overoptimistic Reporting in Methodological Computational Research
Boulesteix, A.-L. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004191 · Cited by 74
“Some rules presented in other ten-simple-rules articles are also directly or indirectly related to overoptimistic reporting, for instance, those on writing papers [16], better figures [17], getting published [12], efficient computational research [18], and reproducible research [14].”
BackgroundMethodUnderstanding experiments and research practices for reproducibility: an exploratory study
Samuel, S., & König‐Ries, B. (2021) · PeerJ · doi:10.7717/peerj.11140 · Cited by 76
“It is also favored to keep data in raw format whenever possible, which can facilitate future re-analysis and analytical reproducibility ( Sandve et al., 2013 ; Hart et al., 2016 ).”
BackgroundMethodInfluentialComputational Reproducibility via Containers in Psychology
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.892 · Cited by 69
“While there are many useful guides to best practices for scientific research (Wilson et al., 2017; Sandve, Nekrutenko, Taylor, and Hovig, 2013), adopting them is an investment of scarce time and attention.”
MethodAgbor-Enoh, S. T., Tunc, I., Vlaminck, I. D., Fideli, U., Davis, A. A., Cuttin, K., Bhatti, K., Marishta, A., Solomon, M. A., Jackson, A. M., et al. (2017) · The Journal of Heart and Lung Transplantation · doi:10.1016/j.healun.2017.05.026 · Cited by 72
Maximizing computational tools for successful drug discovery
Nantasenamat, C., & Prachayasittikul, V. (2015) · Expert Opinion on Drug Discovery · doi:10.1517/17460441.2015.1016497 · Cited by 69
Knoben, W. J. M., Clark, M., Bales, J. D., Bennett, A. R., Gharari, S., Marsh, C. B., Nijssen, B., Pietroniro, A., Spiteri, R. J., Tarboton, D. G., et al. (2022) · Water Resources Research · doi:10.1029/2021wr031753 · Cited by 72
“Reproducibility of computational science can be improved by following certain recommended best practices for open, accessible, and reproducible science (e.g., Gil et al., 2016; Hutton et al., 2016; Sandve et al., 2013; Stodden & Miguez, 2013).”
BackgroundMethodInfluentialOpenAlea: scientific workflows combining data analysis and simulation
Pradal, C., Fournier, C., Valduriez, P., & Cohen‐Boulakia, S. (2015) · International Conference on Scientific and Statistical Database Management · doi:10.1145/2791347.2791365 · Cited by 76
“Among others, they may provide a user interface to design workflows by composing tools [15], a scheduler to optimize the processing of huge amounts of data [6], a provenance module [4] to keep track of the data used and generated during an execution and ensure the reproducibility of the experiments [18].”
Ten Simple Rules for a Computational Biologist’s Laboratory Notebook
Schnell, S. D. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004385 · Cited by 65
“If you do not systematically archive changes, it will be very difficult or impossible to track the codes that you used to generate certain results [8,9].”
BackgroundThe Crucial Role of Interdisciplinary Conferences in Advancing Explainable AI in Healthcare
Patel, A., Gu, Q., Esper, R. N., Maeser, D., & Maeser, N. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020075 · Cited by 78
Tracking materials science data lineage to manage millions of materials experiments and analyses
Soedarmadji, E., Stein, H. S., Suram, S. K., Guevarra, D., & Gregoire, J. M. (2019) · npj Computational Materials · doi:10.1038/s41524-019-0216-x · Cited by 65
A survey of biodiversity informatics: Concepts, practices, and challenges
Gadelha, L. M. R., Jr., de Siracusa, P. C., Dalcin, E., da Silva, L. A. E., Augusto, D. A., Krempser, E., Affe, H. M. D. J., Costa, R. L., Mondelli, M. L., Meirelles, P. M., et al. (2020) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1394 · Cited by 72
“Therefore, it is important to use methodologies and techniques that are reproducible (Cohen-Boulakia et al., 2017; Ivie & Thain, 2018; Peng, 2011; Sandve et al., 2013) when executing these activities.”
BackgroundMethodTen simple rules for biologists learning to program
Carey, M. A., & Papin, J. A. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005871 · Cited by 66
“For specific advice, see “Ten Simple Rules for a Computational Biologist’s Laboratory Notebook” [13], “Ten Simple Rules for Reproducible Computational Research” [14], and “Ten Simple Rules for Taking Advantage of Git and GitHub” [15].”
BackgroundOn best practices in the development of bioinformatics software
Leprevost, F. D. V., Barbosa, V. C., Francisco, E. L., Perez-Riverol, Y., & Carvalho, P. C. (2014) · Frontiers in Genetics · doi:10.3389/fgene.2014.00199 · Cited by 62
“…in biology, and yet the formal training of new professionals (Perez-Riverol et al., 2013; Via et al., 2013), the availability of good services for data deposition, and the development of new standards and software coding rules (Sandve et al., 2013; Seemann, 2013) are still major concerns.”
BackgroundScotson, L., Johnston, L., Iannarilli, F., Wearn, O. R., Mohd‐Azlan, J., Wong, W. M., Gray, T. N. E., Dinata, Y., Suzuki, A., Willard, C. E., et al. (2017) · Remote Sensing in Ecology and Conservation · doi:10.1002/rse2.54 · Cited by 61
“7 R e c o r d i ng o f i nd i v i du a l s un i qu e l y i d e n t i fi a b l e b y n a t u r a l o r a rt i fi c i a l m a r k s . derived from these raw data ideally using well-documented computer code that facilitates transparency and reproducibility (Sandve et al. 2013).”
BackgroundGene isoforms as expression-based biomarkers predictive of drug response in vitro
Safikhani, Z., Smirnov, P., Thu, K. L., Silvester, J., El-Hachem, N., Quevedo, R., Lupien, M., Mak, T. W., Cescon, D. W., & Haibe‐Kains, B. (2017) · Nature Communications · doi:10.1038/s41467-017-01153-8 · Cited by 68
Baichoo, S., Souilmi, Y., Panji, S., Botha, G., Meintjes, A., Hazelhurst, S., Bendou, H., de Beste, E., Mpangase, P. T., Souiai, O., et al. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2446-1 · Cited by 61
Ten Simple Rules to Enable Multi-site Collaborations through Data Sharing
Boland, M. R., Karczewski, K. J., & Tatonetti, N. P. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005278 · Cited by 61
“provide Ten Simple Rules for facilitating research reproducibility in general [20].”
BackgroundThompson, M. W., Gilmer, J., Matsumoto, R. A., Quach, C. D., Shamaprasad, P., Yang, A. H., Iacovella, C. R., McCabe, C., & Cummings, P. T. (2020) · Molecular Physics · doi:10.1080/00268976.2020.1742938 · Cited by 64
ideal: an R/Bioconductor package for interactive differential expression analysis
Marini, F., Linke, J., & Binder, H. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03819-5 · Cited by 64
A Comprehensive Survey of Tools and Software for Active Subnetwork Identification
Nguyen, H., Shrestha, S., Tran, D., Shafi, A., Drăghici, S., & Nguyen, T. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.00155 · Cited by 62
“More importantly, since reproducibility has become an outstanding issue recently, software availability and version control are critical for quality control and for reproducing the results reported in scientific papers (Sandve et al., 2013).”
BackgroundA large-scale analysis of bioinformatics code on GitHub
Russell, P. H., Johnson, R. L., Ananthan, S., Harnke, B., & Carlson, N. E. (2018) · PLOS ONE · doi:10.1371/journal.pone.0205898 · Cited by 51
“Reproducibility requires that authors publish original data and a clear protocol to allow repetition of the analysis in a paper [7].”
Tellurium notebooks—An environment for reproducible dynamical modeling in systems biology
Medley, J. K., Choi, K., König, M., Smith, L., Gu, S., Hellerstein, J., Sealfon, S. C., & Sauro, H. M. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006220 · Cited by 55
Recommendations for the packaging and containerizing of bioinformatics software
Grüening, B., Sallou, O., Moreno, P. A., Leprevost, F. D. V., Ménager, H., Søndergaard, D., Röst, H. L., Sachsenberg, T. T., O’Connor, B. D., Madeira, F., et al. (2018) · F1000Research · doi:10.12688/f1000research.15140.2 · Cited by 67
Ciw: An open-source discrete event simulation library
Palmer, G., Knight, V., Harper, P., & Hawa, A. L. (2018) · Journal of Simulation · doi:10.1080/17477778.2018.1473909 · Cited by 56
“Reproducibility and replicability, described as “the cornerstoneof cumulative science” (Sandve,Nekrutenko, Taylor & Hovig, 2013), is critical in order to assert correct results and build on the work of others (Hong, Crick, Gent, & Kotthoff, 2015; Sandve et al., 2013).”
BackgroundResultInfluentialData science vs. statistics: two cultures?
Carmichael, I. D., & Marron, J. S. (2018) · Japanese Journal of Statistics and Data Science · doi:10.1007/s42081-018-0009-3 · Cited by 68
“Others 42 developing tools to solve technical issues 43 in reproducible research include (Sandve et al, 2013; Kiar et al, 2017).”
BackgroundTowards computational reproducibility: researcher perspectives on the use and sharing of software
AlNoamany, Y., & Borghi, J. (2018) · PeerJ Computer Science · doi:10.7717/peerj-cs.163 · Cited by 60
“Software-related best practices have been outlined for both individuals working in specific research disciplines ( Eglen et al., 2017 ; Marwick, 2017 ) and for the research community in general (e.g., Piccolo & Frampton, 2016 ; Sandve et al., 2013 ; Jimenez et al., 2017 ).”
BackgroundA tale of two 'opens': intersections between Free and Open Source Software and Open ScholarshipPreprint
Tennant, J., Agarwal, R., Baždarić, K., Brassard, D., Crick, T., Dunleavy, D. J., Evans, T. R., Gardner, N., González-Márquez, M., Graziotin, D., et al. (2020) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/2kxq8 · Cited by 58
“What if it can be reproduced, because it solves a particular problem, leads to an innovation, or makes a big jump forward in a field (Sandve et al. 2013; Fanelli 2018)?”
BackgroundInfluentialOlorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
“This way, it will be possible to anticipate areas of ambiguities and insufficient information [2, 5, 26].”
BackgroundInfluentialCyberGIS‐Jupyter for reproducible and scalable geospatial analytics
Yin, D., Liu, Y., Hu, H., Terstriep, J., Hong, X., Padmanabhan, A., & Wang, S. (2018) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5040 · Cited by 54
Ten simple rules for good research practice
Schwab, S., Janiaud, P., Dayan, M., Amrhein, V., Panczak, R., Palagi, P. M., Hemkens, L. G., Ramon, M., Rothen, N., Senn, S. J., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010139 · Cited by 55
“, R Markdown, Jupyter); see also the 10 simple rules for reproducible computational research [76].”
BackgroundTop considerations for creating bioinformatics software documentation
Karimzadeh, M., & Hoffman, M. M. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbw134 · Cited by 51
Fitak, R. R., Antonides, J. D., Baitchman, E. J., Bonaccorso, E., Braun, J. M., Kubiski, S. V., Chiu, E. S., Fagre, A. C., Gagne, R. B., Lee, J. S., et al. (2019) · Journal of Heredity · doi:10.1093/jhered/esz001 · Cited by 57
“” This issue applies to study design, sample collection, sequencing, analysis methodology, publication, and data and software sharing (Vaught et al. 2010; Whitlock 2011; Sandve et al. 2013; Olson et al. 2015; Conesa et al. 2016; Wilson et al. 2017).”
BackgroundKrafczyk, M. S., Shi, A., Bhaskar, A., Marinov, D., & Stodden, V. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0069 · Cited by 61
“Like previous work [72,107], we provide guidelines, and we also contribute an explicit framing with the three fundamental principles we present.”
BackgroundMethodResultAI-Driven Enzyme Engineering: Emerging Models and Next-Generation Biotechnological Applications
Khan, M. F., & Khan, M. T. (2025) · Molecules · doi:10.3390/molecules31010045 · Cited by 61
“Open-access repositories and MLOps frameworks, including con-https tainerised pipelines and continuous integration, are essential to support transparent and scalable model development [192].”
Harnessing QbD, Programming Languages, and Automation for Reproducible Biology
Sadowski, M. I., Grant, C., & Fell, T. S. (2015) · Trends in biotechnology · doi:10.1016/j.tibtech.2015.11.006 · Cited by 60
“(iv) Machine writing of processes is facilitated, enables automated optimization and avoids human error (v) Practices for managing complexity in software development become available: version control, automated testing and validation, code validity checking, and others [118,119].”
BackgroundKey Attributes of a Modern Statistical Computing Tool
McNamara, A. A. (2018) · The American Statistician · doi:10.1080/00031305.2018.1482784 · Cited by 55
“Reproducibility supports the aims of science, and should therefore be integrated with the work of data science (Buckheit and Donoho, 1995; Sandve et al., 2013; Ince et al., 2012; De Leeuw, 2009).”
BackgroundMethodOut-of-the-Box Reproducibility: A Survey of Machine Learning Platforms
Isdahl, R., & Gundersen, O. E. (2019) · eScience · doi:10.1109/escience.2019.00017 · Cited by 80
“There is also a lack of openness and version control of code and data [10], and the quality of documentation is poor [11].”
BackgroundBogart, E., & Myers, C. R. (2016) · PLoS ONE · doi:10.1371/journal.pone.0151722 · Cited by 54
Packaging data analytical work reproducibly using R (and friends)Preprint
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · doi:10.7287/peerj.preprints.3192v1 · Cited by 50
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49
“and funding bodies, “good habits of reproducibility” do save time and money in the longer run [71].”
BackgroundAdministrative social science data: The challenge of reproducible research
Playford, C., Gayle, V., Connelly, R., & Gray, A. J. G. (2016) · Big Data & Society · doi:10.1177/2053951716684143 · Cited by 52
“Our strong advice is that researchers using micro-level administrative social science data must never undertake manual manipulations of data that are undocumented (this point is emphasised by Sandve et al., 2013).”
BackgroundHullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 59
“ML faces analogous issues to the lack of open data and code in social psychology [68, 98, 106, 178, 193].”
BackgroundMethodTen simple rules for initial data analysis
Baillie, M., le Cessie, S., Schmidt, C. O., Lusa, L., Huebner, M., & for the Topic Group “Initial Data Analysis” of the STRATOS Initiative (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009819 · Cited by 49
“A version control of the scripts [16] and automatization of steps will save a lot of time at second use, for example, when major revisions or additions are required.”
BackgroundLiu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47
Reasons, challenges, and some tools for doing reproducible transportation research
Zheng, Z. (2021) · Communications in Transportation Research · doi:10.1016/j.commtr.2021.100004 · Cited by 47
Toward Reproducible Environmental Modeling for Decision Support: A Worked Example
White, J. T., Foster, L. K., Fienen, M. N., Knowling, M. J., Hemmings, B., & Winterle, J. R. (2020) · Frontiers in Earth Science · doi:10.3389/feart.2020.00050 · Cited by 47
Kirouac, D. C., Cicali, B., & Schmidt, S. (2019) · CPT: Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12390 · Cited by 46
“In “Ten Simple Rules for Reproducible Computational Research,” the last, but not least, important rule states that all data and model code should be publicly available and easily accessible.(9) CPT: Pharmacometrics & Systems Pharmacology has a crucial role to play here.”
BackgroundAssessing FAIR Data Principles Against the 5-Star Open Data Principles
Hasnain, A., & Rebholz‐Schuhmann, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98192-5_60 · Cited by 46
“There have been a number of recent domain specific practices and guidelines for data publishing and management [15,13,6], FAIR guiding principles stand unique in number of ways as these are domain-independent and high-level that can be applied to both metadata as well as data.”
BackgroundTen quick tips for deep learning in biology
Lee, B. D., Gitter, A., Greene, C. S., Raschka, S., Maguire, F., Titus, A., Kessler, M. D., Lee, A. J., Chevrette, M. G., Stewart, P. A., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009803 · Cited by 50
“On the whole, all code, random seeds, parameters, and results must be carefully corralled using general coding standards and best practices (for example, version control [49] and continuous integration [50]) to be reproducible and interpretable [51,52].”
BackgroundGeneTonic: an R/Bioconductor package for streamlining the interpretation of RNA-seq data
Marini, F., Ludt, A., Linke, J., & Strauch, K. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04461-5 · Cited by 48
A practical guide for transparency in psychological sciencePreprint
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H. R., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/rtygm · Cited by 46
Best Practices for Making Reproducible Biochemical Models
Porubsky, V. L., Goldberg, A. P., Rampadarath, A. K., Nickerson, D. P., Karr, J. R., & Sauro, H. M. (2020) · Cell Systems · doi:10.1016/j.cels.2020.06.012 · Cited by 45
“Computational models of biochemical system dynamics face the same criticism (Elofsson et al., 2019; Sandve et al., 2013; Peng, 2011; Medley et al., 2016; Waltemath and Wolkenhauer, 2016).”
BackgroundImproving functional magnetic resonance imaging reproducibility
Pernet, C. R., & Poline, J. (2015) · GigaScience · doi:10.1186/s13742-015-0055-8 · Cited by 45
“Add comments to scripts, providing information not just about what is computed, but also reflecting what hypothesis is being tested, or question answered, by that specific piece of code [24].”
BackgroundGendoo, D. M. A., Zon, M., Sandhu, V., Manem, V. S., Ratanasirigulchai, N., Chen, G. M., Waldron, L., & Haibe‐Kains, B. (2019) · Scientific Reports · doi:10.1038/s41598-019-45165-4 · Cited by 45
Asbun, A. A., Besseling, M., Balzano, S., van Bleijswijk, J. D. L., Witte, H. J., Villanueva, L., & Engelmann, J. C. (2020) · Frontiers in Genetics · doi:10.3389/fgene.2020.489357 · Cited by 51
“The code of Cascabel is open source and all analyses are protocolled in the reports and config file, complying with the rules for reproducible computational research described by Sandve et al. (2013).”
MethodComputational Reproducibility in The Wildlife Society's Flagship Journals
ArchMiller, A. A., Johnson, A. D., Nolan, J., Edwards, M., Elliott, L. H., Ferguson, J. M., Iannarilli, F., Vélez, J., Vitense, K., Johnson, D. H., et al. (2020) · Journal of Wildlife Management · doi:10.1002/jwmg.21855 · Cited by 45
“…wildlife ecologists review the many additional ways to improve reproducibility of data analysis, such as documentation of analysis program versions and creation of thorough metadata, described by Whitlock (2011), Sandve et al. (2013), British Ecological Society (2017), and Carey and Papin (2018).”
BackgroundMethodPicault, S., Huang, Y.-L., Sicard, V., Arnoux, S., Beaunée, G., & Ezanno, P. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007342 · Cited by 44
Data Science in Undergraduate Life Science Education: A Need for Instructor Skills Training
Emery, N. C., Crispo, E., Supp, S. R., Farrell, K. J., Kerkhoff, A. J., Bledsoe, E. K., O’Donnell, K., McCall, A. C., & Aiello‐Lammens, M. E. (2021) · BioScience · doi:10.1093/biosci/biab107 · Cited by 47
Computing Workflows for Biologists: A Roadmap
Shade, A., & Teal, T. K. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002303 · Cited by 40
“Recent works have promoted methods for reproducibility in computational research [2,3], provided guidelines for writing software and tools [1,4–6] and advocated for training for the next generation of scientists in computing [7–9].”
BackgroundMethodInfluentialReproducible big data science: A case study in continuous FAIRness
Madduri, R., Chard, K., D’Arcy, M., Jung, S. C., Rodriguez, A., Sulakhe, D., Deutsch, E. W., Funk, C. C., Heavner, B., Richards, M. A., et al. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213013 · Cited by 46
Konkol, M., Nüst, D., & Goulier, L. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00095-y · Cited by 47
“Also, this paper does not discuss how to work reproducibly since this is covered already in literature (e.g., Rule et al. (2019), Sandve et al. (2013), Greenbaum et al. (2017), Markowetz (2015)).”
BackgroundImproving reproducibility in computational biology research
Papin, J. A., Mac Gabhann, F., Sauro, H. M., Nickerson, D., & Rampadarath, A. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007881 · Cited by 42
“Many tools have emerged that we can leverage to make computational biology research more reproducible (e.g., http://co.mbine.org/ and https://normsys.h-its.org/) and there exist articles that propose best practices, such as Ten Simple Rules for Reproducible Computational Research [5] or Ten Simple Rules for Writing an…”
BackgroundFASTAptameR 2.0: A web tool for combinatorial sequence selections
Kramer, S. T., Gruenke, P. R., Alam, K. K., Xu, D. H., & Burke, D. H. (2022) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2022.08.030 · Cited by 42
A CyberGIS-Jupyter Framework for Geospatial Analytics at Scale
Yin, D., Liu, Y., Padmanabhan, A., Terstriep, J., Rush, J., & Wang, S. (2017) · Practice and Experience in Advanced Research Computing · doi:10.1145/3093338.3093378 · Cited by 42
“Computational reproducibility requires computational environments, including the exact versions of all external libraries, to be recorded and reproducible [4, 19].”
BackgrounddtoolAI: Reproducibility for Deep Learning
Hartley, M., & Olsson, T. S. G. (2020) · Patterns · doi:10.1016/j.patter.2020.100073 · Cited by 48
“Results that support hypotheses must be replicable by others, within reasonable parameters.(7) This reproducibility has come under close scrutiny recently, with initial attention directed toward the reproducibility of studies in psychology(8) before widening to science as a whole.”
BackgroundScalable Data Analysis in Proteomics and Metabolomics Using BioContainers and Workflows Engines
Pérez‐Riverol, Y., & Moreno, P. A. (2019) · PROTEOMICS · doi:10.1002/pmic.201900147 · Cited by 41
Ten quick tips for building FAIR workflows
de Visser, C., Johansson, L. F., Kulkarni, P., Mei, H., Neerincx, P. B. T., van der Velde, K. J., Horvatovich, P. L., van Gool, A. J., Swertz, M. A., Hoen, P. A. C. ’., et al. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011369 · Cited by 43
“For example, recommendations have been introduced regarding research data discovery [7], reproducible computational research [8], and the inter-operability of individual computational tools [9].”
BackgroundA long journey into reproducible computational neuroscience
Topalidou, M., Leblois, A., Boraud, T., & Rougier, N. P. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00030 · Cited by 40
“In a previous modeling study, Leblois et al. (2006) demonstrated an action selection mechanism in cortico-basal ganglia loops based on a competition between the positive feedback, direct pathway through the striatum and the negative feedback, hyperdirect pathway through the subthalamic nucleus.”
BackgroundBioinformatics can boost metabolomics research
Meier, R., Ruttkies, C., Treutler, H., & Neumann, S. (2017) · Journal of Biotechnology · doi:10.1016/j.jbiotec.2017.05.018 · Cited by 44
KniMet: a pipeline for the processing of chromatography–mass spectrometry metabolomics data
Liggi, S., Hinz, C., Hall, Z., Santoru, M. L., Poddighe, S., Fjeldsted, J. C., Atzori, L., & Griffin, J. L. (2018) · Metabolomics · doi:10.1007/s11306-018-1349-5 · Cited by 43
“Hence, utilization of standardized procedures is fundamental for reliable and reproducible results (Meier et al. 2017; Rocca-Serra et al. 2016; Sandve et al. 2013).”
BackgroundDigan, W. E., Névéol, A., Neuraz, A., Wack, M., Baudoin, D., Burgun, A., & Rance, B. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa261 · Cited by 52
“After filtering first on titles, then on abstracts, and using full texts for snowballing, we identified 8 articles of interest.(2,9,10,13,15,24,30,31) Figure 1 illustrates the distributions of the selected articles over the spectrum of research fields and according to the level of analysis proposed in the work (founda…”
MethodPowerSystems.jl — A power system data management package for large scale modeling
Lara, J. D., Barrows, C., Thom, D., Krishnamurthy, D., & Callaway, D. S. (2021) · SoftwareX · doi:10.1016/j.softx.2021.100747 · Cited by 53
Reproducible research: a minority opinion
Drummond, C. (2017) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2017.1413140 · Cited by 39
osfr: An R Interface to the Open Science Framework
Wolen, A. R., Hartgerink, C. H. J., Hafen, R., Richards, B., Soderberg, C. K., & York, T. P. (2020) · The Journal of Open Source Software · doi:10.21105/joss.02071 · Cited by 41
Toward Long-Term and Archivable Reproducibility
Akhlaghi, M., Infante-Sáinz, R., Roukema, B. F., Khellat, M., Valls-Gabaud, D., & Baena-Gallé, R. (2021) · Computing in Science & Engineering · doi:10.1109/mcse.2021.3072860 · Cited by 44
Fergusson, D. A., Avey, M. T., Barron, C. C., Bocock, M., Biefer, K., Boet, S., Bourque, S. L., Conic, I., Chen, K., Dong, Y., et al. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215221 · Cited by 40
“It has been suggested that this poor level of translation may be due to a lack of reproducibility within basic science studies [4, 5].”
BackgroundMachine learning in predictive biocatalysis: A comparative review of methods and applications
Tripathi, N., Hérisson, J., & Faulon, J. (2025) · Biotechnology Advances · doi:10.1016/j.biotechadv.2025.108698 · Cited by 40
Network‐Based Approaches for Pathway Level Analysis
Nguyen, T., Mitrea, C., & Drăghici, S. (2018) · Current Protocols in Bioinformatics · doi:10.1002/cpbi.42 · Cited by 41
“First, software availability and version control are crucial for reproducing the experimental results that were used to assess the performance of the approach (Sandve, Nekrutenko, Taylor, & Hovig, 2013).”
MethodIntegrative Cancer Pharmacogenomics to Infer Large-Scale Drug Taxonomy
El-Hachem, N., Gendoo, D. M. A., Ghoraie, L. S., Safikhani, Z., Smirnov, P., Chung, C., Deng, K., Fang, A., Birkwood, E., Ho, C., et al. (2017) · Cancer Research · doi:10.1158/0008-5472.can-17-0096 · Cited by 41
“(29) in terms of code availability and replicability of results.”
BackgroundReproducible Extraction of Cross-lingual Topics (rectr)
Chan, C., Zeng, J., Weßler, H., Jungblut, M., Welbers, K., Bajjalieh, J., van Atteveldt, W., & Althaus, S. L. (2020) · Communication Methods and Measures · doi:10.1080/19312458.2020.1812555 · Cited by 39
Reproducibility and Replicability in the Context of the Contested Identities of Geography
Sui, D. Z., & Kedron, P. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806024 · Cited by 37
Kaushik, G. N., Ivkovic, S., Simonovic, J., Tijanić, N., Davis‐Dusenbery, B. N., & Kural, D. (2016) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813207813_0016 · Cited by 36
rCASC: reproducible classification analysis of single-cell sequencing data
Alessandrì, L., Cordero, F., Beccuti, M., Arigoni, M., Olivero, M., Romano, G., Rabellino, S., Licheri, N., Libero, G. D., Pace, L., et al. (2019) · GigaScience · doi:10.1093/gigascience/giz105 · Cited by 38
“Indeed, it was developed following the best-practice rules for re-producible computational research, proposed in 2013 by Sandve et al. [6].”
Summers, A. Z., Gilmer, J., Iacovella, C. R., Cummings, P. T., & McCabe, C. (2020) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.9b01183 · Cited by 39
Ten simple rules to make your research more sustainable
Ligozat, A., Névéol, A., Daly, B., & Frenoux, E. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008148 · Cited by 41
“Furthermore, reusing existing material can also bring added value by demonstrating its reproducibility, documenting a reuse case from the perspective of users with a different background or experimental set-up [33, 34].”
A Primer on Infectious Disease Bacterial Genomics
Lynch, T., Petkau, A., Knox, N., Graham, M., & Domselaar, G. V. (2016) · Clinical Microbiology Reviews · doi:10.1128/cmr.00001-16 · Cited by 53
“, raw, intermediate, and result data) and propose some best-practice guidance to develop policies for the management, analysis, and sharing of data within HTS projects (2, 3).”
BackgroundChallenges with Reproducibility
Bajpai, V. K., Kühlewind, M., Ott, J., Schönwälder, J., Sperotto, A., & Trammell, B. (2017) · doi:10.1145/3097766.3097767 · Cited by 36
“Sandve et al. in [20] list down ten simple rules for reproducible research.”
BackgroundNine quick tips for analyzing network data
Miele, V., Matias, C., Robin, S. S., & Dray, S. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007434 · Cited by 36
“That said, we strongly suggest that you learn programming and scripting your analysis (in agreement with papers in the "Ten Simple Rules" collection about computing skills and reproducibility [15, 16]).”
BackgroundOpenSAFELY: A platform for analysing electronic health records designed for reproducible research
Nab, L., Schaffer, A. L., Hulme, W., DeVito, N. J., Dillingham, I., Wiedemann, M., Andrews, C. D., Curtis, H., Fisher, L., Green, A., et al. (2024) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5815 · Cited by 34
“18 Reshaping a healthcare research database into an analysis-ready dataset also often involves codelists: a list of codes for clinical events or demographic characteristics used to map EHR data to variables used in an analysis.”
Psychology’s Replication Crisis as Scientific Opportunity: A Précis for Policymakers
Rodgers, J. L., & Shrout, P. E. (2017) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732217749254 · Cited by 40
Denaxas, S., Direk, K., González-Izquierdo, A., Pikoula, M., Cakiroglu, S. A., Moore, J. H., Hemingway, H., & Smeeth, L. L. (2017) · BioData Mining · doi:10.1186/s13040-017-0151-7 · Cited by 35
A multi-omics digital research object for the genetics of sleep regulation
Jan, M., Gobet, N., Diessler, S., Franken, P., & Xenarios, I. (2019) · Scientific Data · doi:10.1038/s41597-019-0171-x · Cited by 36
TDAstats: R pipeline for computing persistent homology in topological data analysis
Wadhwa, R., Williamson, D. F. K., Dhawan, A., & Scott, J. G. (2018) · The Journal of Open Source Software · doi:10.21105/joss.00860 · Cited by 34
A Minimum Standard for Publishing Computational Results in the Weather and Climate Sciences
Irving, D. (2015) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-15-00010.1 · Cited by 34
“There are numerous authors who suggest that the best practices adopted by IS2015 (e.g., scripting, version control) save time in the long run (e.g., Sandve et al. 2013; Wilson et al. 2014), which means that once researchers have learned and adopted these practices they may actually save time.”
Background
Metadata
- Authors
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E.
- Affiliations
- University of Oslo (NO)
- Pennsylvania State University (US)
- Emory University (US)
- Oslo University Hospital (NO)
- Corresponding author
- Sandve, Geir Kjetil
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-10-24
- Volume
- 9
- Issue
- 10
- Pages
- e1003285
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3812051
- OpenAlex
- W2036318837
- Cited by
- 880
- References
- 27
- Citations by year
- 2026: 176 · 2025: 60 · 2024: 52 · 2023: 50 · 2022: 58 · 2021: 67 · 2020: 63 · 2019: 88 · 2018: 76 · 2017: 64 · 2016: 56 · 2015: 42 · 2014: 24 · 2013: 3
- FWCI
- 91.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (0850103)
- Pennsylvania Department of Health (HG004909)
- University of Pennsylvania
- Pennsylvania State University
- National Institutes of Health (HG006620)
- Huck Institutes of the Life Sciences
- Institute for Computational and Data Sciences, Pennsylvania State University
- National Human Genome Research Institute (HG005133, U41HG006620, R21HG005133, R01HG004909, HG006620, HG004909)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducible research, computational reproducibility, research practices, bioinformatics analysis, research transparency, version control, software dependency management, workflow management, research data management, research culture, research evaluation, research dissemination, scholarly communication
- MeSH terms
- Reproducibility of Results, Computational Biology, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.