CoChat

Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals

Open access
Stodden, Victoria, Guo, Peixuan, Ma, Zhaokun · 2013 · PLoS ONE

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Abstract

Journal policy on research data and code availability is an important part of the ongoing shift toward publishing reproducible computational science. This article extends the literature by studying journal data sharing policies by year (for both 2011 and 2012) for a referent set of 170 journals. We make a further contribution by evaluating code sharing policies, supplemental materials policies, and open access status for these 170 journals for each of 2011 and 2012. We build a predictive model of open data and code policy adoption as a function of impact factor and publisher and find higher impact journals more likely to have open data and code policies and scientific societies more likely to have open data and code policies than commercial publishers. We also find open data policies tend to lead open code policies, and we find no relationship between open data and code policies and either supplemental material policies or open access journal status. Of the journals in this study, 38% had a data policy, 22% had a code policy, and 66% had a supplemental materials policy as of June 2012. This reflects a striking one year increase of 16% in the number of data policies, a 30% increase in code policies, and a 7% increase in the number of supplemental materials policies. We introduce a new dataset to the community that categorizes data and code sharing, supplemental materials, and open access policies in 2011 and 2012 for these 170 journals.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0067111

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Changes in Data Sharing Policy, 2011–2012.
Figure 1 Changes in Data Sharing Policy, 2011–2012.CC BY · PubMed Central
Figure 2 Changes in Code Sharing Policy, 2011–2012.
Figure 2 Changes in Code Sharing Policy, 2011–2012.CC BY · PubMed Central
Figure 3 Changes in Supplemental Materials Policy, 2011–2012.
Figure 3 Changes in Supplemental Materials Policy, 2011–2012.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 25 references

Research works printed but not linked: 2

  1. Reproducible Research. Special Issue, Computing in Science and Engineering 14: 4 11–56. http://ieeexplore.ieee.org/xpl/tocresult.jsp?reload=true&isnumber=6241356&punumber=5992 (last accessed May 8, 2013).
  2. Editorial (1998) The gatekeepers. Nature Structural Biology 5: 3 164. http://www.nature.com/nsmb/wilma/v5n3.892130820.html (last accessed May 8, 2013).

Cited by

See all 276

Metadata

Authors
Stodden, V., Guo, P., & Ma, Z.
Affiliations
Columbia University (US)
Corresponding author
Stodden, Victoria
Venue
PLoS ONE
Published
2013-06-21
Volume
8
Issue
6
Pages
e67111
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2031934506
Cited by
257
References
25
Citations by year
2026: 9 · 2025: 12 · 2024: 11 · 2023: 11 · 2022: 19 · 2021: 23 · 2020: 33 · 2019: 25 · 2018: 26 · 2017: 30 · 2016: 18 · 2015: 21 · 2014: 15 · 2013: 2
FWCI
54.56
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Columbia Academic Commons (Columbia University)
Funders
National Science Foundation (1153384)
Topics
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
reproducible research, open data, open code, open access journals, journal impact factor
MeSH terms
Models, Theoretical, Periodicals as Topic, Reproducibility of Results, Computational Biology, Codes of Ethics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals. https://app.cochat.ai/papers/10.1371/journal.pone.0067111
The paper
Stodden, V., Guo, P., & Ma, Z. (2013). Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals. PLoS ONE. https://doi.org/10.1371/journal.pone.0067111

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.