Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency
Open accessPublished version of a preprint
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Beginning January 2014, Psychological Science gave authors the opportunity to signal open data and materials if they qualified for badges that accompanied published articles. Before badges, less than 3% of Psychological Science articles reported open data. After badges, 23% reported open data, with an accelerating trend; 39% reported open data in the first half of 2015, an increase of more than an order of magnitude from baseline. There was no change over time in the low rates of data sharing among comparison journals. Moreover, reporting openness does not guarantee openness. When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned. Open materials also increased to a weaker degree, and there was more variability among comparison journals. Badges are simple, effective signals to promote open practices and improve preservation of data and materials by using independent repositories.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Poor distribution of labor and lack of task specialization may be part of the reason data and code sharing remain rare [48,54]”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [54]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, simply making data available does not guarantee that the data have utility: Kidwell et al. also observed that a substantial proportion (50/111, 45%) of reportedly available data was in fact not available or was incomplete, incorrect, or had insufficient documentation.”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, in an observational study examining the impact of a voluntary ‘open badges’ scheme at the journal Psychological Science, Kidwell et al. [10] found that the proportion of articles reporting available data increased from around 2.5% during the pre-policy period to 39.5% by the end of the post-policy period.”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reproducibility badges adopted by several journals45 recognize stronger forms of reproducibility without penalizing authors who cannot yet reach them.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They then studied what happened when the journal Psychological Science introduced the open data badge and found a substantial increase in the percentage of publications reporting open data [54].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [54]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Spence, M. (1973) · The Quarterly Journal of Economics · doi:10.2307/1882010 · Cited by 19,518
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Signaling Theory: A Review and Assessment
Connelly, B. L., Certo, S. T., Ireland, R. D., & Reutzel, C. R. (2011) · Journal of Management · doi:10.1177/0149206310388419 · Cited by 6,735
The Sociology of Science: Theoretical and Empirical Investigations
Kaplan, N., Merton, R. K., & Storer, N. W. (1974) · Technology and Culture · doi:10.2307/3102264 · Cited by 4,005
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Self-citationTHE SOCIOLOGY OF SCIENCE:: Theoretical and Empirical Investigations
Merton, R. Κ., & Storer, N. W. (2017) · Cited by 1,462
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Self-citationThe poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543 · cited 1×
“The key problem with “upon request” policies is that requests come after the researcher has completed the research, published the article, and lost or forgot detailed materials and data [9,11,16].”
BackgroundPromoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
Self-citation
Research works printed but not linked: 1
- RK Merton, 1942, Science and technology in a democratic order, J of Legal and Pol Soc, 1, 115
Cited by
Items 101–200 of 736
Promoting Open Science: A Holistic Approach to Changing Behaviour
Robson, S. G., Baum, M. A., Beaudry, J. L., Beitner, J., Brohmer, H., Chin, J. M., Jaśko, K., Kouros, C. D., Laukkonen, R., Moreau, D., et al. (2021) · Collabra Psychology · doi:10.1525/collabra.30137 · Cited by 86
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201856 · Cited by 78
Self-citationRavaud, P., Créquit, P., Williams, H. C., Meerpohl, J. J., Craig, J. C., & Boutron, I. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.027 · Cited by 78
“stakeholders who make their data publicly available would be useful [13].”
BackgroundTransparent Social Inquiry: Implications for Political Science
Elman, C., Kapiszewski, D., & Lupia, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-091515-025429 · Cited by 83
Open Science 2.0: Towards a truly collaborative research ecosystem
Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023) · PLOS Biology · doi:10.1371/journal.pbio.3002362 · Cited by 72
Lei, N., Masanet, E. R., & Koomey, J. (2021) · Energy Policy · doi:10.1016/j.enpol.2021.112422 · Cited by 82
“Finally, researchers should consider publishing in journals that provide Open Science Badges [79], which can ensure persistent access to modeling files and increase data sharing opportunities.”
BackgroundStriving for transparent and credible research: practical guidelines for behavioral ecologists
Ihle, M., Winney, I. S., Krystalli, A., & Croucher, M. (2017) · Behavioral Ecology · doi:10.1093/beheco/arx003 · Cited by 75
“These Badges have been proven to incentivize scientists to make their work more transparent (Kidwell et al. 2016).”
BackgroundSharing Data and Materials in Psychological Science
Lindsay, D. S. (2017) · Psychological Science · doi:10.1177/0956797617704015 · Cited by 72
Perceived crisis and reforms: Issues, explanations, and remedies.
de Boeck, P. A. L., & Jeon, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000154 · Cited by 75
Integrating Qualitative Methods and Open Science: Five Principles for More Trustworthy Research*
Humphreys, L., Lewis, N. A., Sender, K., & Won, A. S. (2021) · Journal of Communication · doi:10.1093/joc/jqab026 · Cited by 69
Prosser, A. M. B., Hamshaw, R. J. T., Meyer, J., Bagnall, R., Blackwood, L. M., Huysamen, M., Jordan, A. L., Vasileiou, K., & Walter, Z. C. (2022) · British Journal of Social Psychology · doi:10.1111/bjso.12576 · Cited by 75
“When Psychological Science adopted badges in 2014, there was a marked increase in articles reporting data sharing, from 2.5% before to 22.8% in the following year (Kidwell et al., 2016).”
BackgroundInfluentialA new ecosystem for evidence synthesis
Nakagawa, S., Dunn, A. G., Lagisz, M., Bannach‐Brown, A., Grames, E. M., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Westgate, M. J., Arnold, P. A., et al. (2020) · Nature Ecology & Evolution · doi:10.1038/s41559-020-1153-2 · Cited by 69
A Standard for the Scholarly Citation of Archaeological Data as an Incentive to Data Sharing
Marwick, B., & Birch, S. E. P. (2018) · Advances in Archaeological Practice · doi:10.1017/aap.2018.3 · Cited by 71
“Grantees are expected to encourage and facilitate such sharing [2016].”
BackgroundMaking ERP research more transparent: Guidelines for preregistration
Paul, M., Govaart, G., & Schettino, A. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.02.016 · Cited by 71
Questionable Research Practices and Open Science in Quantitative Criminology
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · Journal of Quantitative Criminology · doi:10.1007/s10940-021-09525-6 · Cited by 82
“Similarly, journal policies encouraging or requiring open data seem to be having the desired effect, with articles published in those journals being more likely to have made their data available in a public repository (Hardwicke et al. 2018; Kidwell et al. 2016; c.f., Rowhani-Farid and Barnett 2018).”
BackgroundImproving Psychological Science through Transparency and Openness: An Overview
Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2018) · Perspectives on Behavior Science · doi:10.1007/s40614-018-00186-8 · Cited by 70
“These journal policies coincide with a sharp increase in the proportion of articles published that have open data (Kidwell et al., 2016).”
BackgroundTowse, J. N., Ellis, D. A., & Towse, A. S. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01486-1 · Cited by 69
WORCS: A workflow for open reproducible code in science
Lissa, C. J. V., Brandmaier, A. M., Brinkman, L., Lamprecht, A.-L., Peikert, A., Struiksma, M. E., & Vreede, B. M. I. (2021) · Data Science · doi:10.3233/ds-210031 · Cited by 70
Bakker, B. N., Jaidka, K., Dorr, T. G., Fasching, N., & Lelkes, Y. (2021) · Journal of Communication · doi:10.1093/joc/jqab031 · Cited by 69
Open Communication Science: A Primer on Why and Some Recommendations for How
Lewis, N. A. (2020) · Communication Methods and Measures · doi:10.1080/19312458.2019.1685660 · Cited by 59
“After badges, 23% reported open data, a trend that continues to increase at an accelerating rate – by the first half of 2015, that number was up to 39% (Kidwell et al., 2016).”
BackgroundExploratory and confirmatory research in the open science era
Nilsen, E. B., Bowler, D. E., & Linnell, J. D. C. (2020) · Journal of Applied Ecology · doi:10.1111/1365-2664.13571 · Cited by 72
Seven steps toward more transparency in statistical practice
Wagenmakers, E., Sarafoglou, A., Aarts, S., Albers, C. J., Algermissen, J., Bahník, Š., van Dongen, N. N. N., Hoekstra, R., Moreau, D., van Ravenzwaaij, D., et al. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01211-8 · Cited by 68
Motivating participation in open science by examining researcher incentives
Ali‐Khan, S. E., Harris, L. W., & Gold, E. R. (2017) · eLife · doi:10.7554/elife.29319 · Cited by 64
Toribio‐Flórez, D., Anneser, L., Lopes, F. N. D. O., Pallandt, M. M. T. A., Tunn, I., Windel, H., & on behalf of Max Planck PhDnet Open Science Group (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2020.586992 · Cited by 63
“…from funders, which might only lead researchers to show minimal compliance (Neylon, 2017), individual incentives for researchers should be introduced through, for example, professional recognition or the allocation of extra funding (Kidwell et al., 2016; Fecher et al., 2015; Ali-Khan et al., 2018).”
BackgroundTwelve key challenges in medical machine learning and solutions
Ellis, R. J., Sander, R. M., & Limon, A. (2022) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2022.100068 · Cited by 61
Biosecurity in an age of open science
Smith, J. A., & Sandbrink, J. B. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001600 · Cited by 64
“Many have been proposed, including changing hiring practices to support open data, open materials and preregistration [83–85], open science leaderboards [86], journal scores based on transparency [87], badges acknowledging open science on published papers [88], and assessment of open science PLOS BIOLOGY”
BackgroundAn Action Plan for Diamond Open Access
Borrell-Damián, L., & Rooryck, J. (2022) · doi:10.4000/books.oep.16147 · Cited by 60
Nguyen, V. M., Brooks, J. L., Young, N., Lennox, R. J., Haddaway, N., Whoriskey, F. G., Harcourt, R. G., & Cooke, S. J. (2017) · Canadian Journal of Fisheries and Aquatic Sciences · doi:10.1139/cjfas-2016-0261 · Cited by 60
Köhler, T., González‐Morales, M. G., Banks, G. C., O’Boyle, E. H., Allen, J. A., Sinha, R., Woo, S. E., & Gulick, L. M. V. (2020) · Industrial and Organizational Psychology · doi:10.1017/iop.2019.121 · Cited by 66
“These initiatives include evaluating the effectiveness of publication badges (Kidwell et al., 2016), the potential for bias in the peer review process (Emerson et al., 2010), evaluating journal policies related to peer review (Banks et al., 2016a), and compliance with those policies (Hardwicke et…”
BackgroundFlake, J. K. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1898962 · Cited by 58
Cashin, A. G., Bagg, M. K., Richards, G. C., Toomey, E. C., McAuley, J. H., & Lee, H. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111296 · Cited by 65
Dougherty, M. R. P., Slevc, L. R., & Grand, J. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691618810693 · Cited by 80
Mebane, C. A., Sumpter, J. P., Fairbrother, A., Augspurger, T. P., Canfield, T. J., Goodfellow, W. L., Guiney, P. D., LeHuray, A. P., Maltby, L., Mayfield, D. B., et al. (2019) · Integrated Environmental Assessment and Management · doi:10.1002/ieam.4119 · Cited by 62
“In at least one journal, this approach appeared to markedly improve the sharing and preservation of data through linked, independent repositories (Kidwell et al. 2016; Munaf o et al. 2017).”
Backgroundde Oliveira, G. S., & Baggs, E. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009303538 · Cited by 81
“One visible change has been the introduction of badges, which are awarded to published papers if the authors have preregistered their hypotheses before collecting the data, and if they havemade their data and researchmaterials openly available for their peers to examine (Kidwell et al., 2016).”
BackgroundThe Weak Spots in Contemporary Science (and How to Fix Them)
Wicherts, J. M. (2017) · Animals · doi:10.3390/ani7120090 · Cited by 59
“Journal publishers could demand more transparency from their authors [93] or could incentivize data sharing and other open practices by handing out badges, which appear to work quite well in promoting rigorous practices [94].”
BackgroundClinical Psychological Science: Then and Now
Lilienfeld, S. O. (2016) · Clinical Psychological Science · doi:10.1177/2167702616673363 · Cited by 55
“Following the lead of the APS journal Psychological Science, CPS is now formally embracing the core tenets of the movement toward open science, which underscore the importance of transparency and self-scrutiny in science.”
BackgroundInfluentialCrüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., et al. (2023) · Psychological Science · doi:10.1177/09567976221140828 · Cited by 54
“Kidwell et al. (2016) found that introducing badges at Psychological Science led to an increase in sharing, which indicates the superficial success of this policy— particularly compared with other initiatives (see Rowhani-Farid & Barnett, 2018, and Rowhani-Farid et al., 2020, who found lower and no…”
ResultTransparency Is Now the Default at Psychological Science
Hardwicke, T. E., & Vazire, S. (2023) · Psychological Science · doi:10.1177/09567976231221573 · Cited by 57
Self-citationReproducible research practices and transparency across linguistics
Bochyńska, A., Keeble, L., Halfacre, C. A. E., Casillas, J. V., Champagne, I.-A., Chen, K., Röthlisberger, M., Buchanan, E. M., & Roettger, T. B. (2023) · Glossa Psycholinguistics · doi:10.5070/g6011239 · Cited by 54
Song, H., Markowitz, D. M., & Taylor, S. H. (2022) · Journal of Communication · doi:10.1093/joc/jqac017 · Cited by 41
Personal Motivations and Systemic Incentives: Scientists on Questionable Research Practices
Bruton, S. V., Medlin, M. M., Brown, M., & Sacco, D. F. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00182-9 · Cited by 55
Taboos and Self-Censorship Among U.S. Psychology Professors
Clark, C. J., Fjeldmark, M., Lu, L., Baumeister, R. F., Ceci, S. J., Frey, K., Miller, G. F., Reilly, W., Tice, D. M., von Hippel, W., et al. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916241252085 · Cited by 56
“For example, many scholars have advocated for greater transparency and reproducibility in science (e.g., Kidwell et al., 2016; Nosek et al., 2015, 2022), but not all scholars have responded favorably to these norm changes (e.g., Bahlai et al., 2019; Whitaker & Guest, 2020).”
Developing an open science ‘mindset’
Hagger, M. S. (2021) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2021.2012474 · Cited by 56
“…that aim to improve replicability, reproducibility, credibility, and transparency in research (e.g. Aczel et al., 2020; Appelbaum et al., 2018; Kidwell et al., 2016; Morey et al., 2016; NIH, 2021a; Nosek et al., 2015; NSF, 2021a; Open Science MOOC, 2018; SIPS, 2021; The FOSTER consortium,…”
BackgroundToward Open and Reproducible Epidemiology
Mathur, M. B., & Fox, M. P. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad007 · Cited by 56
“Journals have introduced simple, effective incentives such as optional badge icons on papers to recognize openscience practices (15, 28).”
BackgroundPackaging data analytical work reproducibly using R (and friends)Preprint
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · doi:10.7287/peerj.preprints.3192v1 · Cited by 50
Pawel, S., Kook, L., & Reeve, K. A. (2023) · Biometrical Journal · doi:10.1002/bimj.202200091 · Cited by 53
“Similar initiatives have led to a large increase in the adoption of preregistered study protocols in the field of psychology (Kidwell et al., 2016).”
BackgroundPreregistration of Analyses of Preexisting Data
Mertens, G., & Krypotos, A. (2019) · Psychologica Belgica · doi:10.5334/pb.493 · Cited by 53
Stracke, C. M. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3_2 · Cited by 54
Radical Solutions and Open Science: An Open Approach to Boost Higher Education
Burgos, D. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3 · Cited by 51
Tamminen, K. A., & Poucher, Z. A. (2018) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2017.12.010 · Cited by 53
Liu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47
A Framework for Improving the Quality of Research in the Biological Sciences
Casadevall, A., Ellis, L. M., Davies, E. W., McFall-Ngai, M., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.01256-16 · Cited by 46
Schneider, J. M., Rosman, T., Kelava, A., & Merk, S. (2022) · Psychological Science · doi:10.1177/09567976221097499 · Cited by 51
“Beyond initial investigations of badges’ effectiveness in fostering data sharing and adherence to open-science standards (Kidwell et al., 2016), we now have evidence that badges have the potential to increase trust in scientists by their target audiences (scientists and student teachers).”
BackgroundMethodInfluentialData management and sharing: Practices and perceptions of psychology researchers
Borghi, J., & Gulick, A. E. V. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252047 · Cited by 51
“For example, evaluation of datasets made available under the Cognition data policy [23] and associated with articles given open science badges in Psychological Science [24] indicates that a non-trivial proportion were incomplete or not reusable in principle.”
MethodA practical guide for transparency in psychological sciencePreprint
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H. R., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/rtygm · Cited by 46
Self-citationBest Practices for Making Reproducible Biochemical Models
Porubsky, V. L., Goldberg, A. P., Rampadarath, A. K., Nickerson, D. P., Karr, J. R., & Sauro, H. M. (2020) · Cell Systems · doi:10.1016/j.cels.2020.06.012 · Cited by 45
“Some journals now test the functionality of submitted models and may award badges to manuscript that attach data, source code and model artifacts, or attach artifacts that pass reproducibility tests performed by the journal (Donoho, 2010; Kidwell et al., 2016; AJPS, 2016).”
BackgroundKey challenges facing data-driven multicellular systems biology
Macklin, P. (2019) · GigaScience · doi:10.1093/gigascience/giz127 · Cited by 47
“While badges can help [31, 32], we must ensure that data users can easily cite all these contributions in papers, that impact metrics re ect the breadth of contributions, and that tenure and other career processes truly value all contributions to community knowledge resources.”
BackgroundBadges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.2 · Cited by 53
A step-by-step guide on preregistration and effective data sharing for psychopathology research.
Krypotos, A.-M., Klugkist, I., Mertens, G., & Engelhard, I. M. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000424 · Cited by 41
Publish or perish, but do not forget your software artifacts
Heumüller, R., Nielebock, S., Krüger, J., & Ortmeier, F. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09851-6 · Cited by 60
Open research data repositories: Practices, norms, and metadata for sharing images
Hansson, K. E., & Dahlgren, A. N. (2021) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24571 · Cited by 43
“Increasing datafication of research practices and their infrastructure also creates new incentives to enable several researchers to share metadata and data (Kidwell et al., 2016; Nosek et al., 2015; Roche et al., 2014).”
BackgroundEstimating the reproducibility of social learning research published between 1955 and 2018
Minocher, R., Atmaca, S., Bavero, C., McElreath, R., & Beheim, B. (2021) · Royal Society Open Science · doi:10.1098/rsos.210450 · Cited by 26
Spitzer, L., & Mueller, S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281086 · Cited by 47
“Open science badges [75] that are displayed prominently in research articles and signal to readers which open science techniques are associated with this article (e.”
BackgroundHarper, L. M., & Kim, Y. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.03.001 · Cited by 46
“4%) from the first half of 2014 to the first half of 2015 (Kidwell et al., 2016).”
BackgroundMarwick, B. (2017) · ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) · doi:10.17605/osf.io/3d6xx · Cited by 41
Gamification Reloaded: Current and Future Trends in Gamification Science
Mazarakis, A. (2021) · i-com · doi:10.1515/icom-2021-0025 · Cited by 47
“Kidwell et al. showed that badges could motivate scientists to share data [57].”
BackgroundTransparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
Improving norms in research culture to incentivize transparency and rigor
Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 41
Guidelines to improve internationalization in the psychological sciences
Puthillam, A., Doble, L. J. M., Santos, J. J. I. D., Elsherif, M. M., Steltenpohl, C. N., Moreau, D., Pownall, M., Silverstein, P., Anand-Vembar, S., & Kapoor, H. (2023) · Social and Personality Psychology Compass · doi:10.1111/spc3.12847 · Cited by 45
Toward open science in PLS-SEM: Assessing the state of the art and future perspectives
Adler, S. J., Sharma, P. N., & Radomir, L. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114291 · Cited by 44
Future Challenges and Opportunities in Academic Publishing
Siler, K. (2017) · The Canadian Journal of Sociology · doi:10.29173/cjs28140 · Cited by 39
“In a strong endorsement of Merton’s (1968) thesis that reward structures underpin scientific behaviors, Kidwell et al. (2016) found that the introduction of data sharing ‘badges’ adorning article webpages substantially increased data sharing at Psychological Science.”
BackgroundPractical tools and strategies for researchers to increase replicability
Nuijten, M. B. (2018) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.14054 · Cited by 45
“Several journals now require or reward open data (with extra recognition for papers that share their data), and these policies are strongly related to increased data availability.(43,44) Ideally, when data are shared, there should be a quality check to ensure they are relevant, complete, and usable.”
BackgroundExamining the Reproducibility of Meta-Analyses in Psychology: A Preliminary ReportPreprint
Lakens, D., Page‐Gould, E., van Assen, M. A. L. M., Spellman, B., Schönbrodt, F. D., Hasselman, F., Corker, K. S., Grange, J., Sharples, A. E., Cavender, C., et al. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/xfbjf · Cited by 42
What Crowdsourcing Can Offer to Cross-Cultural Psychological Science
Cuccolo, K., Irgens, M. S., Zlokovich, M. S., Grahe, J. E., & Edlund, J. E. (2020) · Cross-Cultural Research · doi:10.1177/1069397120950628 · Cited by 39
“The literature also supports open science initiatives as an avenue for improving the scientific quality of research (Asendorpf et al., 2013; Kidwell et al., 2016; Nosek et al., 2018), scientific efficiency, and democratizing knowledge (Arza & Fressoli, 2017).”
BackgroundPublishers’ Responsibilities in Promoting Data Quality and Reproducibility
Hrynaszkiewicz, I. (2019) · Handbook of experimental pharmacology · doi:10.1007/164_2019_290 · Cited by 43
“Data sharing also increased in the journal compared to other psychology journals (Munafò et al. 2017; Kidwell et al. 2016).”
BackgroundReproducibility and Replicability in the Context of the Contested Identities of Geography
Sui, D. Z., & Kedron, P. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806024 · Cited by 37
Open science and reform practices in organizational behavior research over time (2011 to 2019)
Tenney, E. R., Costa, E., Allard, A., & Vazire, S. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.10.015 · Cited by 37
Open Science Needs No Martyrs, but We Must Recognize the Need for Reform
Susi, T. (2022) · doi:10.4000/books.oep.16156 · Cited by 36
A Call for Open Science in Giftedness Research
McBee, M. T., Makel, M. C., Peters, S. J., & Matthews, M. S. (2018) · Gifted Child Quarterly · doi:10.1177/0016986218784178 · Cited by 40
“Put simply, badges are a form of recognition for research that has met standards for open science, and there is empirical research supporting that they serve to increase the use of open science practices and to increase the availability and accuracy of data (Kidwell et al., 2016).”
BackgroundKing, K. M., Pullmann, M. D., Lyon, A. R., Dorsey, S., & Lewis, C. C. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000417 · Cited by 38
“…2010), and editors and editorial boards have tried to shape how authors report studies, such as banning or requiring more stringent p values (Finch et al., 2004; Ioannidis, 2018), or implementing incentives to promote specific practices such as data or materials sharing (Kidwell et al., 2016).”
BackgroundThe credibility crisis in research: Can economics tools help?
Gall, T., Ioannidis, J. P. A., & Maniadis, Z. (2017) · PLoS Biology · doi:10.1371/journal.pbio.2001846 · Cited by 38
Evans, T. R., Branney, P., Clements, A. J., & Hatton, E. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1969233 · Cited by 39
“Whilst initiatives like journal badges and preregistration challenges are changing preregistration norms within academia (e.g., Kidwell et al. 2016), there are currently few direct benefits for other stakeholders to support preregistration.”
BackgroundThe open science initiative of the Empirical Software Engineering journal
Méndez, D., Monperrus, M., FELDT, R. H., & Zimmermann, T. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09712-x · Cited by 35
Feger, S. S., Dallmeier-Tiessen, S., Woźniak, P. W., & Schmidt, A. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300690 · Cited by 34
“[28] found a significant increase in data sharing of submissions to the Psychological Science journal that adapted to those badges.”
BackgroundInfluentialTeaching Good Research Practices: Protocol of a Research Master Course
Sarafoglou, A., Hoogeveen, S., Matzke, D., & Wagenmakers, E. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719858807 · Cited by 38
Verify original results through reanalysis before replicating
Nuijten, M. B., Bakker, M., Maassen, E., & Wicherts, J. M. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000791 · Cited by 34
A guide to open science practices for animal research
Diederich, K., Schmitt, K. D., Schwedhelm, P., Bert, B., & Heinl, C. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001810 · Cited by 35
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/umq8d · Cited by 37
Spitschan, M., Schmidt, M. H., & Blume, C. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16111.1 · Cited by 32
Open Science in Kenya: Where Are We?
Mwangi, K. W., Mainye, N., Ouso, D. O., Esoh, K. K., Muraya, A. W., Mwangi, C. K., Naitore, C., Karega, P., Kibet, G., Musundi, S., et al. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.669675 · Cited by 35
“The research data management policies also need to focus on incentivizing researchers to share their data by considering issues that affect data sharing (Fecher et al., 2015; Kidwell et al., 2016; Ali-Khan et al., 2017).”
BackgroundTool-Driven Revolutions in Archaeological Science
Schmidt, S. C., & Marwick, B. (2020) · Journal of Computer Applications in Archaeology · doi:10.5334/jcaa.29 · Cited by 39
“Simple forms of signalling, such as badges on papers to indicate that the authors have made code and data available, may also help to shift norms and raise awareness of these desirable behaviours (Kidwell et al. 2016).”
BackgroundMoving Transparent Statistics Forward at CHI
Kay, M., Haroz, S., Guha, S., Dragicevic, P., & Wacharamanotham, C. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3027084 · Cited by 30
“…made in some fields that would not be feasible at CHI, at least on a short term; on the other hand, badge systems like the OSF badges [1] have been proposed in other fields, have been found to be effective at increasing transparency in published work [12], and could in principle be feasible at CHI.”
BackgroundPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
How open is innovation research? – An empirical analysis of data sharing among innovation scholars
Barczak, G. J., Hopp, C., Kaminski, J., Piller, F. T., & Pruschak, G. (2021) · Industry and Innovation · doi:10.1080/13662716.2021.1967727 · Cited by 32
Biomedical publishing: Past historic, present continuous, future conditional
Sever, R. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002234 · Cited by 32
“content that represent another step in this direction [51,52], while PLOS and other publishers have also indicated their intent to explore badging [53].”
BackgroundA Unified Framework to Quantify the Credibility of Scientific FindingsPreprint
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/uwmr8 · Cited by 29
Dudda, L., Kormann, E., Kozula, M., DeVito, N. J., Klebel, T., Dewi, A. P. M., Spijker, R., Stegeman, I., Van den Eynden, V., Ross-Hellauer, T., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242057 · Cited by 28
“Rewards and incentives ( n = 9): nine studies investigated the effect of badges on open science or data sharing [56,65,65,74,74,81,102,104].”
Wang, S. V., & Pottegård, A. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwae087 · Cited by 29
The Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 32
Research Preregistration as a Teaching and Learning Tool in Undergraduate Psychology Courses
Blincoe, S., & Buchert, S. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719875844 · Cited by 30
“It is still early to measure many of the effects of these incentives, but one result of a January 2014 implementation of open science badges at Psychological Science was an improved rate of data sharing (Kidwell et al., 2016).”
Background
Metadata
- Authors
- Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S., & Nosek, B. A.
- Affiliations
- Center for Open Science (US)
- University of Belgrade (RS)
- University of California, Riverside (US)
- University College London (GB)
- Max Planck Institute for Behavioral Economics (DE)
- Mississippi State University (US)
- University of Vienna (AT)
- Corresponding author
- Kidwell, Mallory C.
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-05-12
- Volume
- 14
- Issue
- 5
- Pages
- e1002456
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4865119
- OpenAlex
- W2390706404
- Cited by
- 700
- References
- 19
- Citations by year
- 2026: 31 · 2025: 35 · 2024: 39 · 2023: 71 · 2022: 110 · 2021: 87 · 2020: 98 · 2019: 80 · 2018: 70 · 2017: 54 · 2016: 25
- FWCI
- 170.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- WU Research (publishedVersion)
- Phaidra (Universität Wien) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Faculty of Philosophy (University of Belgrade) (submittedVersion, cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- The Laura and John Arnold Foundation
- The John Templeton Foundation
- National Institute on Aging
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- open badges, open data, open practices, research transparency, data sharing, open materials, data preservation, scholarly publishing, author incentives
- MeSH terms
- Cost-Benefit Analysis, Psychology, Publishing, Serial Publications, Internet, Information Dissemination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.