Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Journal policy on research data and code availability is an important part of the ongoing shift toward publishing reproducible computational science. This article extends the literature by studying journal data sharing policies by year (for both 2011 and 2012) for a referent set of 170 journals. We make a further contribution by evaluating code sharing policies, supplemental materials policies, and open access status for these 170 journals for each of 2011 and 2012. We build a predictive model of open data and code policy adoption as a function of impact factor and publisher and find higher impact journals more likely to have open data and code policies and scientific societies more likely to have open data and code policies than commercial publishers. We also find open data policies tend to lead open code policies, and we find no relationship between open data and code policies and either supplemental material policies or open access journal status. Of the journals in this study, 38% had a data policy, 22% had a code policy, and 66% had a supplemental materials policy as of June 2012. This reflects a striking one year increase of 16% in the number of data policies, a 30% increase in code policies, and a 7% increase in the number of supplemental materials policies. We introduce a new dataset to the community that categorizes data and code sharing, supplemental materials, and open access policies in 2011 and 2012 for these 170 journals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Policies that mandate sharing of analysis scripts have already been implemented at some journals, for example in economics and political science [19,45,46].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An increasing number of publishers and funding bodies require authors or grant holders to make their data publicly available [247].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [247]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 436
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 1×
“…has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated with…”
BackgroundScientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
Evolution of Translational Omics: Lessons Learned and the Path Forward
Micheel, C., Nass, S. J., & Omenn, G. S. (2013) · Cited by 365
Public Availability of Published Research Data in High-Impact Journals
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024357 · Cited by 363 · cited 1×
“…attention has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated…”
BackgroundPublish your computer code: it is good enough
Barnes, N. (2010) · Nature · doi:10.1038/467753a · Cited by 315
Accessible Reproducible Research
Mesirov, J. P. (2010) · Science · doi:10.1126/science.1179653 · Cited by 307
Reproducible Research in Computational Harmonic Analysis
Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2009) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274
Self-citation
Research works printed but not linked: 2
- Reproducible Research. Special Issue, Computing in Science and Engineering 14: 4 11–56. http://ieeexplore.ieee.org/xpl/tocresult.jsp?reload=true&isnumber=6241356&punumber=5992 (last accessed May 8, 2013).
- Editorial (1998) The gatekeepers. Nature Structural Biology 5: 3 164. http://www.nature.com/nsmb/wilma/v5n3.892130820.html (last accessed May 8, 2013).
Cited by
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Donoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695
Metadata
- Authors
- Stodden, V., Guo, P., & Ma, Z.
- Affiliations
- Columbia University (US)
- Corresponding author
- Stodden, Victoria
- ORCID
- Stodden, Victoria
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-06-21
- Volume
- 8
- Issue
- 6
- Pages
- e67111
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3689732
- OpenAlex
- W2031934506
- Cited by
- 257
- References
- 25
- Citations by year
- 2026: 9 · 2025: 12 · 2024: 11 · 2023: 11 · 2022: 19 · 2021: 23 · 2020: 33 · 2019: 25 · 2018: 26 · 2017: 30 · 2016: 18 · 2015: 21 · 2014: 15 · 2013: 2
- FWCI
- 54.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Columbia Academic Commons (Columbia University)
- Funders
- National Science Foundation (1153384)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducible research, open data, open code, open access journals, journal impact factor
- MeSH terms
- Models, Theoretical, Periodicals as Topic, Reproducibility of Results, Computational Biology, Codes of Ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals. https://app.cochat.ai/papers/10.1371/journal.pone.0067111
- The paper
- Stodden, V., Guo, P., & Ma, Z. (2013). Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals. PLoS ONE. https://doi.org/10.1371/journal.pone.0067111
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


