Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency
Open accessPublished version of a preprint
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Beginning January 2014, Psychological Science gave authors the opportunity to signal open data and materials if they qualified for badges that accompanied published articles. Before badges, less than 3% of Psychological Science articles reported open data. After badges, 23% reported open data, with an accelerating trend; 39% reported open data in the first half of 2015, an increase of more than an order of magnitude from baseline. There was no change over time in the low rates of data sharing among comparison journals. Moreover, reporting openness does not guarantee openness. When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned. Open materials also increased to a weaker degree, and there was more variability among comparison journals. Badges are simple, effective signals to promote open practices and improve preservation of data and materials by using independent repositories.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Poor distribution of labor and lack of task specialization may be part of the reason data and code sharing remain rare [48,54]”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [54]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, simply making data available does not guarantee that the data have utility: Kidwell et al. also observed that a substantial proportion (50/111, 45%) of reportedly available data was in fact not available or was incomplete, incorrect, or had insufficient documentation.”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, in an observational study examining the impact of a voluntary ‘open badges’ scheme at the journal Psychological Science, Kidwell et al. [10] found that the proportion of articles reporting available data increased from around 2.5% during the pre-policy period to 39.5% by the end of the post-policy period.”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reproducibility badges adopted by several journals45 recognize stronger forms of reproducibility without penalizing authors who cannot yet reach them.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They then studied what happened when the journal Psychological Science introduced the open data badge and found a substantial increase in the percentage of publications reporting open data [54].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [54]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Spence, M. (1973) · The Quarterly Journal of Economics · doi:10.2307/1882010 · Cited by 19,518
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Signaling Theory: A Review and Assessment
Connelly, B. L., Certo, S. T., Ireland, R. D., & Reutzel, C. R. (2011) · Journal of Management · doi:10.1177/0149206310388419 · Cited by 6,735
The Sociology of Science: Theoretical and Empirical Investigations
Kaplan, N., Merton, R. K., & Storer, N. W. (1974) · Technology and Culture · doi:10.2307/3102264 · Cited by 4,005
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Self-citationTHE SOCIOLOGY OF SCIENCE:: Theoretical and Empirical Investigations
Merton, R. Κ., & Storer, N. W. (2017) · Cited by 1,462
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Self-citationThe poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543 · cited 1×
“The key problem with “upon request” policies is that requests come after the researcher has completed the research, published the article, and lost or forgot detailed materials and data [9,11,16].”
BackgroundPromoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
Self-citationThe Availability of Research Data Declines Rapidly with Article Age
Vines, T. H., Albert, A. Y. K., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Gilbert, K. J., Moore, J.-S., Renaut, S., & Rennison, D. J. (2014) · Current Biology · doi:10.1016/j.cub.2013.11.014 · Cited by 482 · cited 3×
“Moreover, simple specifications of what is required to earn a badge is effective in promoting use of independent repositories that have stronger assurance of availability and preservation than ad hoc methods such as posting data on personal websites [11].”
BackgroundMethodReproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310 · cited 1×
“The result is that a tiny minority of data from published articles are shared [7,20].”
BackgroundPublic Data Archiving in Ecology and Evolution: How Well Are We Doing?
Roche, D. G., Kruuk, L. E. B., Lanfear, R., & Binning, S. A. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002295 · Cited by 303
Effects and Functions in the Evolution of Signaling Systems
Otte, D. (1974) · Annual Review of Ecology and Systematics · doi:10.1146/annurev.es.05.110174.002125 · Cited by 286
Normative Dissonance in Science: Results from a National Survey of U.S. Scientists
Anderson, M. S., Martinson, B. C., & De Vries, R. (2007) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2007.2.4.3 · Cited by 208 · cited 1×
“However, even though researchers endorse openness and transparency [1], with many demands on their time, researchers are not likely to share data and materials unless they have incentives to do so.”
BackgroundAre We Wasting a Good Crisis? The Availability of Psychological Research Data after the Storm
Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015) · Collabra · doi:10.1525/collabra.13 · Cited by 152
The Dawn of Open Access to Phylogenetic Data
Magee, A. F., May, M. R., & Moore, B. R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0110268 · Cited by 68 · cited 1×
“The key problem with “upon request” policies is that requests come after the researcher has completed the research, published the article, and lost or forgot detailed materials and data [9,11,16].”
BackgroundSizing the Problem of Improving Discovery and Access to NIH-Funded Data: A Preliminary Study
Read, K., Sheehan, J. R., Huerta, M. F., Knecht, L. S., Mork, J. G., Humphreys, B. L., & NIH Big Data Annotator Group (2015) · PLoS ONE · doi:10.1371/journal.pone.0132735 · Cited by 36 · cited 2×
“The result is that a tiny minority of data from published articles are shared [7,20].”
BackgroundSummary report of journal operations, 2013.
(2014) · American Psychologist · doi:10.1037/a0036753 · Cited by 5
Research works printed but not linked: 1
- RK Merton, 1942, Science and technology in a democratic order, J of Legal and Pol Soc, 1, 115
Cited by
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Experimental Physiology · doi:10.1113/ep088870 · Cited by 3,497
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationThe preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
“It is possible that such incentives will become nearly as effective as badges for open data, which were associated with more than a 10-fold increase in data sharing in an initial test (58).”
BackgroundSelf-citationThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x20943823 · Cited by 1,443
“The bioscience community has introduced a range of initiatives to address the problem, from open access and open practices to enable the scrutiny of all aspects of the research [4, 5] through to study preregistration to shift the focus towards robust methods rather than the novelty of the results [6, 7], as well as re…”
Metadata
- Authors
- Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S., & Nosek, B. A.
- Affiliations
- Center for Open Science (US)
- University of Belgrade (RS)
- University of California, Riverside (US)
- University College London (GB)
- Max Planck Institute for Behavioral Economics (DE)
- Mississippi State University (US)
- University of Vienna (AT)
- Corresponding author
- Kidwell, Mallory C.
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-05-12
- Volume
- 14
- Issue
- 5
- Pages
- e1002456
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4865119
- OpenAlex
- W2390706404
- Cited by
- 700
- References
- 19
- Citations by year
- 2026: 31 · 2025: 35 · 2024: 39 · 2023: 71 · 2022: 110 · 2021: 87 · 2020: 98 · 2019: 80 · 2018: 70 · 2017: 54 · 2016: 25
- FWCI
- 170.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- WU Research (publishedVersion)
- Phaidra (Universität Wien) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Faculty of Philosophy (University of Belgrade) (submittedVersion, cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- The Laura and John Arnold Foundation
- The John Templeton Foundation
- National Institute on Aging
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- open badges, open data, open practices, research transparency, data sharing, open materials, data preservation, scholarly publishing, author incentives
- MeSH terms
- Cost-Benefit Analysis, Psychology, Publishing, Serial Publications, Internet, Information Dissemination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.