The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
An empirical analysis of journal policy effectiveness for computational reproducibility
Open accessAbstract
A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpublication by the authors upon request. We assess the effectiveness of such a policy by ( i ) requesting data and code from authors and ( ii ) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Indeed, the reproducibility of published computational research has been reported as similarly dismal (~25%).4”
From Reproducibility of Quantitative Systems Pharmacology Models: Current Challenges and Future Opportunities (2019) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 473
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 464
Self-citationStatistical Analyses and Reproducible Research
Gentleman, R. C., & Lang, D. T. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x178663 · Cited by 364
Gilbert, K. J., Andrew, R. L., Bock, D. G., Franklin, M. T., Kane, N. C., Moore, J., Moyers, B. T., Renaut, S., Rennison, D. J., Veen, T., et al. (2012) · Molecular Ecology · doi:10.1111/j.1365-294x.2012.05754.x · Cited by 290
Reproducible Research in Computational Harmonic Analysis
Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. (2008) · Computing in Science & Engineering · doi:10.1109/mcse.2009.15 · Cited by 274
Self-citationStodden, V., Guo, P., & Ma, Z. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067111 · Cited by 257
Self-citationIs Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'
Chang, A. C., & Li, P. (2015) · doi:10.2139/ssrn.2669564 · Cited by 217
Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say "Usually Not"
Chang, A. C., & Li, P. (2015) · Finance and Economics Discussion Series · doi:10.17016/feds.2015.083 · Cited by 216
Making Data Maximally Available
Hanson, B., Sugden, A. M., & Alberts, B. (2011) · Science · doi:10.1126/science.1203354 · Cited by 172
The Legal Framework for Reproducible Scientific Research: Licensing and Copyright
Stodden, V. (2008) · Computing in Science & Engineering · doi:10.1109/mcse.2009.19 · Cited by 157
Self-citationComputing environments for reproducibility: Capturing the “Whole Tale”
Brinckman, A., Chard, K., Gaffney, N. I., Hategan, M., Jones, M. B., Kowalik, K., Kulasekaran, S., Ludäscher, B., Mecum, B., Nabrzyski, J., et al. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2017.12.029 · Cited by 151
Self-citationQuintana, D. S. (2020) · eLife · doi:10.7554/elife.53275 · Cited by 150
Privacy, Big Data, and the Public Good
(2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781107590205 · Cited by 145
Carata, L., Akoush, S., Balakrishnan, N., Bytheway, T., Sohan, R., Seltzer, M., & Hopper, A. (2014) · Communications of the ACM · doi:10.1145/2596628 · Cited by 85
Creating and sharing reproducible research code the workflowr way
Blischak, J., Carbonetto, P. S., & Stephens, M. (2019) · F1000Research · doi:10.12688/f1000research.20843.1 · Cited by 76
Definitions of dataset in the scientific and technical literature
Renear, A. H., Sacchi, S., & Wickett, K. M. (2010) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504701240 · Cited by 65
Certificates of Confidentiality: Protecting Human Subject Research Data in Law and Practice
Wolf, L. E., Patel, M. J., Tarver, B. A. W., Austin, J. L., Dame, L. A., & Beskow, L. M. (2015) · The Journal of Law Medicine & Ethics · doi:10.1111/jlme.12302 · Cited by 41
Enabling Reproducibility in Big Data Research: Balancing Confidentiality and Scientific Transparency
Stodden, V. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781107590205.007 · Cited by 26
Self-citationOpen Science is for Aging Research, Too
Isaacowitz, D. M., & Lind, M. (2019) · Innovation in Aging · doi:10.1093/geroni/igz028 · Cited by 25
Differential Privacy: A Cryptographic Approach to Private Data Analysis
Dwork, C. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781107590205.018 · Cited by 20
Carata, L., Akoush, S., Balakrishnan, N., Bytheway, T., Sohan, R., Seltzer, M., & Hopper, A. (2014) · Queue · doi:10.1145/2602649.2602651 · Cited by 17
Reproducing Epidemiologic Research and Ensuring Transparency
Coughlin, S. S. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx065 · Cited by 10
Opinion: set the default to "open"
Bailey, D. H., Borwein, J. M., & Stodden, V. (2013) · Notices of the American Mathematical Society · Cited by 9
Self-citationPerturbation Methods for Protecting Numerical Data: Evolution and Evaluation
Sarathy, R., & Muralidhar, K. (2012) · Handbook of statistics · doi:10.1016/b978-0-44-451875-0.00019-1 · Cited by 7
Research works printed but not linked: 2
- ; American Association for the Advancement of Science Science journals: Editorial Policies. Available at www.sciencemag.org/authors/science-journals-editorial-policies. Accessed September 11 2011. (2011).
- V Stodden JM Borwein DH Bailey “Setting the default to reproducible’ in Computational Science Research. SIAM News . Available at https://sinews.siam.org/Aboutthe-Author/setting-the-default-to-reproducible-in-computational-science-research. Accessed February 16 2018. (2013).
Cited by
Sorted by citations
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
“Other factors, such as the lack of available code and data, can also lead to irreproducibility, and there are several studies investigating these shortcomings.(50,51) We discuss our choice of terminology in detail in the supplemental experimental procedures (section S1).”
BackgroundO’Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G. B., Moher, D., et al. (2021) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12721 · Cited by 825
The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support
Razavi, S., Jakeman, A. J., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Piano, S. L., Iwanaga, T., Becker, W., et al. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104954 · Cited by 703
Jermyn, A. S., Bauer, E. B., Schwab, J., Farmer, R., Ball, W. H., Bellinger, E. P., Dotter, A., Joyce, M., Marchant, P., Mombarg, J. S. G., et al. (2023) · The Astrophysical Journal Supplement Series · doi:10.3847/1538-4365/acae8d · Cited by 736
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Metadata
- Authors
- Stodden, V., Seiler, J., & Ma, Z.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Columbia University (US)
- Corresponding author
- Stodden, Victoria
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2018-03-12
- Volume
- 115
- Issue
- 11
- Pages
- 2584-2589
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5856507
- OpenAlex
- W2793993424
- Cited by
- 369
- References
- 26
- Citations by year
- 2026: 45 · 2025: 34 · 2024: 36 · 2023: 53 · 2022: 40 · 2021: 50 · 2020: 36 · 2019: 48 · 2018: 26
- FWCI
- 64.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Alfred P Sloan Foundation (PG004545)
- National Science Foundation (1153384)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- computational reproducibility, editorial policy, data availability, code availability, replication, research artifacts, scientific journals, author guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
