The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of Psychological Science
Open accessPublished version of a preprint
Abstract
In April 2019, Psychological Science published its first issue in which all Research Articles received the Open Data badge. We used that issue to investigate the effectiveness of this badge, focusing on the adherence to its aim at Psychological Science : sharing both data and code to ensure reproducibility of results. Twelve researchers of varying experience levels attempted to reproduce the results of the empirical articles in the target issue (at least three researchers per article). We found that all 14 articles provided at least some data and six provided analysis code, but only one article was rated to be exactly reproducible, and three were rated as essentially reproducible with minor deviations. We suggest that researchers should be encouraged to adhere to the higher standard in force at Psychological Science . Moreover, a check of reproducibility during peer review may be preferable to the disclosure method of awarding badges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Whether shared data are findable, accessible, interoperable, and reusable (FAIR) is yet another question, and some evidence, at least in the field of psychology, suggests that this is often not the case [49,50].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [50]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Other researchers then accessed those open datasets and tried to reproduce the results reported in each paper, but had a low rate of success [50].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · cited 1×
“Authors can also ensure machine-actionable reusability of their data by following the findable, accessible, interoperable, and reusable (FAIR) guidelines (Wilkinson et al., 2016).”
MethodReproducibility and Replicability in Science
Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, et al. (2019) · doi:10.17226/25303 · Cited by 924 · cited 1×
“…for successful reproduction in advance, for a study to count as reproducible, its results should at least be reproducible by a competent external researcher (National Academies of Sciences, Engineering, and Medicine, 2019), such as a PhD student with some experience and training in a similar field.”
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700 · cited 1×
“Kidwell et al. (2016) found that introducing badges at Psychological Science led to an increase in sharing, which indicates the superficial success of this policy— particularly compared with other initiatives (see Rowhani-Farid & Barnett, 2018, and Rowhani-Farid et al., 2020, who found lower and no…”
ResultData reuse and the open data citation advantage
Piwowar, H. A., & Vision, T. J. (2013) · PeerJ · doi:10.7717/peerj.175 · Cited by 603
Enhancing reproducibility for computational methods
Stodden, V., McNutt, M., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 411
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369 · cited 1×
“Stodden et al. (2018) assessed the effectiveness of a policy of mandatory sharing on request at the journal Science and found that, despite this policy, they received data for only 44% of articles.”
BackgroundThe citation advantage of linking publications to research data
Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K., & McGillivray, B. (2020) · PLOS ONE · doi:10.1371/journal.pone.0230416 · Cited by 335
Data Sharing in Psychology: A Survey on Barriers and Preconditions
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917751886 · Cited by 283 · cited 1×
“Moreover, the provision of further incentives, in particular by funding agencies and institutions, may help make data sharing more common and effective (Houtkoop et al., 2018).”
BackgroundEich, E. (2014) · Psychological Science · doi:10.1177/0956797613512465 · Cited by 255 · cited 1×
“the badges in 2014 (Eich, 2014), and, in April 2019, published its first issue in which all 14 Research Articles received Open Data badges (Volume 30, Issue 4).”
BackgroundGabelica, M., Bojčić, R., & Puljak, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.05.019 · Cited by 249 · cited 1×
“5 Regardless, whether authors fill in their disclosure items appropriately should continue to be monitored—a recent study found low adherence even to mandatory data availability statements in biomedical research manuscripts (Gabelica et al., 2022).”
Background
Cited by
The dire disregard of measurement invariance testing in psychological science.
Maassen, E., D'Urso, E. D., van Assen, M. A. L. M., Nuijten, M. B., Roover, K. D., & Wicherts, J. M. (2023) · Psychological Methods · doi:10.1037/met0000624 · Cited by 151
Open Science 2.0: Towards a truly collaborative research ecosystem
Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023) · PLOS Biology · doi:10.1371/journal.pbio.3002362 · Cited by 72
Computational reproducibility of Jupyter notebooks from biomedical publications
Samuel, S., & Mietchen, D. (2024) · GigaScience · doi:10.1093/gigascience/giad113 · Cited by 91
Rethlefsen, M. L., Brigham, T., Price, C. L., Moher, D., Bouter, L. M., Kirkham, J. J., Schroter, S., & Zeegers, M. P. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111229 · Cited by 77
Transparency Is Now the Default at Psychological Science
Hardwicke, T. E., & Vazire, S. (2023) · Psychological Science · doi:10.1177/09567976231221573 · Cited by 57
“Most pertinently, three studies have reported difficulties reproducing the results of articles published in Psychological Science (Crüwell et al., 2023; Hardwicke et al., 2021; Homewood, 2023).”
Metadata
- Authors
- Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., Zaneva, M., & Brown, N. J. L.
- Affiliations
- University of Cambridge (GB)
- Australian National University (AU)
- Temple University (US)
- University of Sussex (GB)
- Ruhr University Bochum (DE)
- University of Vienna (AT)
- University Hospital Heidelberg (DE)
- Centre National de la Recherche Scientifique (FR)
- University of Sheffield (GB)
- University of Auckland (NZ)
- University of Oxford (GB)
- Linnaeus University (SE)
- Corresponding author
- Crüwell, Sophia
- ORCID
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2023-02-02
- Volume
- 34
- Issue
- 4
- Pages
- 512-522
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W4319062942
- Cited by
- 54
- References
- 43
- Citations by year
- 2026: 15 · 2025: 9 · 2024: 16 · 2023: 14
- FWCI
- 12.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (unspecified)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DiVA (Linkoping University) (submittedVersion, other-oa)
- ResearchSpace (University of Auckland) (submittedVersion)
- Funders
- Economic and Social Research Council (Data Analytics & Society Centre for Doctoral Train)
- Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalen (1706dgn006)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducibility, research data sharing, reproducibility assessment, peer review, open science policy
- MeSH terms
- Humans, Periodicals as Topic, Psychology, Research, Reproducibility of Results, Information Dissemination, Editorial Policies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of Psychological Science. https://app.cochat.ai/papers/10.1177/09567976221140828
- The paper
- Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., Zaneva, M., & Brown, N. J. L. (2023). What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of Psychological Science. Psychological Science. https://doi.org/10.1177/09567976221140828
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.