CoChat
Sign Up
Cited by 369Research Data Management PracticesScientific Computing and Data ManagementAcademic Publishing and Open Access

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

An empirical analysis of journal policy effectiveness for computational reproducibility

Open access
Stodden, Victoria, Seiler, Jennifer, Ma, Zhaokun·2018·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1708290115PMC PMC5856507
Open this paper in CoChat — free accountOpen PDF

Abstract

A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpublication by the authors upon request. We assess the effectiveness of such a policy by ( i ) requesting data and code from authors and ( ii ) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1708290115

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 26 references

Research works printed but not linked: 2

  1. ; American Association for the Advancement of Science Science journals: Editorial Policies. Available at www.sciencemag.org/authors/science-journals-editorial-policies. Accessed September 11 2011. (2011).
  2. V Stodden JM Borwein DH Bailey “Setting the default to reproducible’ in Computational Science Research. SIAM News . Available at https://sinews.siam.org/Aboutthe-Author/setting-the-default-to-reproducible-in-computational-science-research. Accessed February 16 2018. (2013).

Cited by

Sorted by citations

Items 301–369 of 369

Related

See all

Metadata

Authors
Stodden, V., Seiler, J., & Ma, Z.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Columbia University (US)
Corresponding author
Stodden, Victoria
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2018-03-12
Volume
115
Issue
11
Pages
2584-2589
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1708290115
PMC
PMC5856507
OpenAlex
W2793993424
Cited by
369
References
26
Citations by year
2026: 45 · 2025: 34 · 2024: 36 · 2023: 53 · 2022: 40 · 2021: 50 · 2020: 36 · 2019: 48 · 2018: 26
FWCI
64.95
Type
journal article
Language
en
Open access
Licence
www.pnas.org/site/aboutpnas/licenses.xhtml (vor, 184 d)
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
  • Alfred P Sloan Foundation (PG004545)
  • National Science Foundation (1153384)
Topics
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
computational reproducibility, editorial policy, data availability, code availability, replication, research artifacts, scientific journals, author guidelines

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review