Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Journal policy on research data and code availability is an important part of the ongoing shift toward publishing reproducible computational science. This article extends the literature by studying journal data sharing policies by year (for both 2011 and 2012) for a referent set of 170 journals. We make a further contribution by evaluating code sharing policies, supplemental materials policies, and open access status for these 170 journals for each of 2011 and 2012. We build a predictive model of open data and code policy adoption as a function of impact factor and publisher and find higher impact journals more likely to have open data and code policies and scientific societies more likely to have open data and code policies than commercial publishers. We also find open data policies tend to lead open code policies, and we find no relationship between open data and code policies and either supplemental material policies or open access journal status. Of the journals in this study, 38% had a data policy, 22% had a code policy, and 66% had a supplemental materials policy as of June 2012. This reflects a striking one year increase of 16% in the number of data policies, a 30% increase in code policies, and a 7% increase in the number of supplemental materials policies. We introduce a new dataset to the community that categorizes data and code sharing, supplemental materials, and open access policies in 2011 and 2012 for these 170 journals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Policies that mandate sharing of analysis scripts have already been implemented at some journals, for example in economics and political science [19,45,46].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An increasing number of publishers and funding bodies require authors or grant holders to make their data publicly available [247].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [247]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 436
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 1×
“…has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated with…”
BackgroundScientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
Evolution of Translational Omics: Lessons Learned and the Path Forward
Micheel, C., Nass, S. J., & Omenn, G. S. (2013) · Cited by 365
Public Availability of Published Research Data in High-Impact Journals
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024357 · Cited by 363 · cited 1×
“…attention has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated…”
BackgroundPublish your computer code: it is good enough
Barnes, N. (2010) · Nature · doi:10.1038/467753a · Cited by 315
Accessible Reproducible Research
Mesirov, J. P. (2010) · Science · doi:10.1126/science.1179653 · Cited by 307
Reproducible Research in Computational Harmonic Analysis
Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2009) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274
Self-citation
Research works printed but not linked: 2
- Reproducible Research. Special Issue, Computing in Science and Engineering 14: 4 11–56. http://ieeexplore.ieee.org/xpl/tocresult.jsp?reload=true&isnumber=6241356&punumber=5992 (last accessed May 8, 2013).
- Editorial (1998) The gatekeepers. Nature Structural Biology 5: 3 164. http://www.nature.com/nsmb/wilma/v5n3.892130820.html (last accessed May 8, 2013).
Cited by
Items 201–276 of 276
ProvCaRe: A Large-Scale Semantic Provenance Resource for Scientific Reproducibility
Liu, C., Kim, M. J., Rueschman, M., & Sahoo, S. S. (2020) · Advanced information and knowledge processing · doi:10.1007/978-3-030-67681-0_5 · Cited by 1
Weber, N., & Thomer, A. K. (2019) · IGI Global eBooks · doi:10.4018/978-1-5225-8903-7.ch024 · Cited by 2
spectre: an R package to estimate spatially‐explicit community composition using sparse data
Simpkins, C., Hanß, S., Spangenberg, M. C., Salecker, J., Hesselbarth, M. H. K., & Wiegand, K. (2022) · Ecography · doi:10.1111/ecog.06272 · Cited by 2
Biobanks in open-access scientific research
Valverde, A., González, J. A., Sevilla, F., Calderón-Ferrey, M., Jiménez, G., Monge-Ureña, S., Araya‐Zúñiga, I., & González, R. (2025) · Ethics Medicine and Public Health · doi:10.1016/j.jemep.2025.101222 · Cited by 1
Silles, C. A. (2014) · Kent Academic Repository (University of Kent) · Cited by 1
Transcriptome computational workbench (TCW): analysis of single and comparative transcriptomesPreprint
Soderlund, C. A. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/733311 · Cited by 1
Mininet-WiFi: plataforma de emulação para redes sem fio definidas por software
Fontes, R. D. R. (2018) · doi:10.47749/t/unicamp.2018.1051315 · Cited by 2
Simulation Studies of Social Systems -- Telling the Story Based on ProvenancePreprint
Reinhardt, O., Prike, T., Hinsch, M., Bijak, J., Uhrmacher, A. M., & Wilsdorf, P. (2022) · doi:10.36227/techrxiv.20209844.v1 · Cited by 1
A conceptual review of uses and meanings of reproducibility and replicationPreprint
Ulpts, S., & Schneider, J. W. (2025) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/entu4_v3 · Cited by 1
Knowledge and attitudes among life scientists towards reproducibility within journal articles.Preprint
Samota, E. K. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.6084/m9.figshare.c.4436912 · Cited by 2
PyPackIT: Automated Research Software Engineering for Scientific Python Applications on GitHubPreprint
Ariamajd, A., de Castro, R. L.-R., & Volkamer, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.04921 · Cited by 1
Better Web Measurements: Enhancing Security and Privacy Research
Demir, N. (2023) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000163503 · Cited by 3
Visualizing and Forecasting Box-Office Revenues: A Case Study of the James Bond Movie Series
Petrosyan, V. (2014) · Digital Commons - USU (Utah State University) · doi:10.26076/49ec-6583
Como tornar mais confiável a pesquisa publicada
Ioannidis, J. P. A. (2015) · Acimed · doi:10.36512/rcics.v26i2.791
Integrity, Reproducibility, and the Changing Technological Environment for Research
Stodden, V. (2015) · 2015 AAAS Annual Meeting (12-16 February 2015)
Self-citationIrving, D. (2016) · Minerva Access (University of Melbourne)
Método de extração de coortes em bases de dados assistenciais para estudos da doença cardiovascular
Abrahão, M. T. F. (2016) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.5.2016.tde-04082016-160129 · Cited by 3
Les dades de la recerca: de la foscor a la claror
Juan, I. L. I. (2013) · RECERCAT (Consorci de Serveis Universitaris de Catalunya)
Algorithms for Reconstruction of Undersampled Atomic Force Microscopy Images
Oxvig, C. S. (2017) · VBN Forskningsportal (Aalborg Universitet) · doi:10.5278/vbn.phd.engsci.00158
Connecting Researchers to Repositories IMLS Project Report
Brandt, D. S., Carlson, J. R., & Lake, S. (2017) · Purdue e-Pubs (Purdue University System)
Metajelo: a Metadata Package for Journals to Support External Linked Objects
Vilhuber, L., & Lagoze, C. (2021) · International Journal of Digital Curation · doi:10.2218/ijdc.v16i1.600
Schiessl, I. T., & Moreno, F. P. (2019) · Pesquisa Brasileira em Ciência da Informação e Biblioteconomia · doi:10.22478/ufpb.1981-0695.2019v14n4.49357 · Cited by 2
Preregistration, Data Sharing, and Other Salutary Behaviors
Bausell, R. B. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197536537.003.0011
Bayesian two-interval testPreprint
Meyer, N., & Sauleau, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01271
Peer Review #1 of "MLitB: machine learning in the browser (v0.1)"
(2015) · PeerJ Computer Science · doi:10.7287/peerj-cs.11v0.1/reviews/1
Peer Review #2 of "MLitB: machine learning in the browser (v0.2)"
(2015) · PeerJ Computer Science · doi:10.7287/peerj-cs.11v0.2/reviews/2
Peer Review #2 of "MLitB: machine learning in the browser (v0.1)"
(2015) · PeerJ Computer Science · doi:10.7287/peerj-cs.11v0.1/reviews/2
Reproducibility Policies in Cardiology Journals: The REPLICA Cross-Sectional StudyPreprint
Helal, L. C. A. S., Ferrari, F., Rice, D. B., Ahmadzai, N., Skidmore, B., Umpierre, D., & Moher, D. (2022) · medRxiv · doi:10.1101/2022.08.04.22278423
Recent Questions in Responsible Conduct of Research
Wood, L. B. (2014) · The Journal of Philosophy Science & Law · doi:10.5840/jpsl20141411
SHAH, S. A. H., & Petzold, F. (2023) · Architecture and Planning Journal (APJ) · doi:10.54729/2789-8547.1235
Kutnyánszky, A., Szabó, G., Tomcsik, A., & Verle, I. (2023) · Magyar Tudomány · doi:10.1556/2065.184.2023.7.5 · Cited by 1
Is a mathematical model equivalent to its computer implementation?
Hiesmayr, B. C., & Hütt, M. ‐. (2024) · Proceedings of the European Academy of Sciences and Arts · doi:10.4081/peasa.26
Untangling Cables: A Case Study of the Life & Afterlife of Digital Devices in Academic Research
Muntean, R., Hennessy, K., Mills, C., & Antle, A. N. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661540
Andries, C., Quindroit, P., Popoff, B., Oudarrour, I., & Lamer, A. (2024) · Studies in health technology and informatics · doi:10.3233/shti240429 · Cited by 1
Curating models from BioModels: Developing a workflow for creating OMEX files
Xu, J. H., & Smith, L. P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0314875 · Cited by 2
Rethlefsen, M. L. (2024) · Research Publications (Maastricht University) · doi:10.26481/dis.20250121mr
Alternate view for investigating data science knowledge gaps: A U.S. perspective
Grant, D. A., Yeo, B., & Jacob, B. (2025) · Information Development · doi:10.1177/02666669241307022
Digital Open Science – Teaching digital tools for reproducible and transparent researchPreprint
Toelch, U., & Ostwald, D. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/fbvep_v1
A conceptual review of uses and meanings of reproducibility and replicationPreprint
Ulpts, S., & Schneider, J. W. (2025) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/entu4_v2
Selva, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.15893
L’économie à l’ère de la science ouverte : un nouvel élan pour la reproductibilité
Bayle, G., Dubois, D., & Willinger, M. (2026) · Revue économique · doi:10.3917/reco.765.0733 · Cited by 1
Longitudinal dataset of journal data sharing policies across 22 disciplines for 2014 2019 and 2023
Ikeuchi, U. (2025) · Scientific Data · doi:10.1038/s41597-025-06434-2
Osakpolor, S. E. (2023) · Publication Server of Kaiserslautern University of Technology (Kaiserslautern University of Technology) · doi:10.26204/kluedo/7323
Design of a machine-learning-based CAD in colonoscopy
Domínguez, R. (2024) · Investigo Institutional repository of UVigo (Universidade de Vigo)
Comparative analysis of Registered Reports and the standard research literature
O'Mahony, A. (2023) · ORCA Online Research @Cardiff (Cardiff University) · Cited by 3
Processes of change in cognitive therapy for posttraumatic stress disorder
Wiedemann, M. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-vg9w0xvro · Cited by 1
Data sharing intentions of neuroscience researchers: A consequentialist theory perspective
Wu, Z., Huang, L., Wang, M., & He, X. (2026) · Library & Information Science Research · doi:10.1016/j.lisr.2026.101398
García‐Rodríguez, M., Rodeiro-Iglesias, J., & Añel, J. A. (2026) · Geoscientific model development · doi:10.5194/gmd-19-7687-2026
Ikeuchi, U., Kazuhiro, H., & Shinichi, A. (n.d.) · Institutional Repositories DataBase (IRDB)
Essays on female labour supply and fertility
coelho, B. A. R. (2018) · theses.fr (ABES) · doi:10.70675/acfe3129zb963z433dzb16dz4e495fa6d1bf
Gonzalez, J. A. H. (2025) · theses.fr (ABES) · doi:10.70675/293bd8fezc1f7z40e8z9102z055d3165d9ec
Journal research data policies in materials science
Hörmann, L., Myneni, H., Al‐Hamd, R. K. S., Batalović, K., Bonfanti, S., Grasselli, F., Gražulis, S., Koç, B., Konstantinou, K., Lončarić, I., et al. (2026) · Digital Discovery · doi:10.1039/d6dd00111d · Cited by 1
Ramírez, A. (2026) · Revista de Filosofía de la Universidad de Costa Rica · doi:10.15517/revfil.2026.6920
O Fórum SOL de Ciência Aberta e o Desafio da Reprodutibilidade
Viterbo, J., Araujo, A., & Kon, F. (2026) · Computação Brasil. · doi:10.5753/compbr.2026.57.9305
Research Data-Sharing Practices Through the Lens of Data Availability Statements
Tsunoda, H., Sun, Y., Nishizawa, M., Liu, X.-M., Amano, K., & Kawamura, S. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846017
FAIRsharing record for: Ontology of Biological and Clinical Statistics
FAIRsharing Team (2018) · FAIRsharing.org · doi:10.25504/fairsharing.5p12xh
Metadata
- Authors
- Stodden, V., Guo, P., & Ma, Z.
- Affiliations
- Columbia University (US)
- Corresponding author
- Stodden, Victoria
- ORCID
- Stodden, Victoria
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-06-21
- Volume
- 8
- Issue
- 6
- Pages
- e67111
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3689732
- OpenAlex
- W2031934506
- Cited by
- 257
- References
- 25
- Citations by year
- 2026: 9 · 2025: 12 · 2024: 11 · 2023: 11 · 2022: 19 · 2021: 23 · 2020: 33 · 2019: 25 · 2018: 26 · 2017: 30 · 2016: 18 · 2015: 21 · 2014: 15 · 2013: 2
- FWCI
- 54.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Columbia Academic Commons (Columbia University)
- Funders
- National Science Foundation (1153384)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducible research, open data, open code, open access journals, journal impact factor
- MeSH terms
- Models, Theoretical, Periodicals as Topic, Reproducibility of Results, Computational Biology, Codes of Ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


