Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency
Open accessPublished version of a preprint
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Beginning January 2014, Psychological Science gave authors the opportunity to signal open data and materials if they qualified for badges that accompanied published articles. Before badges, less than 3% of Psychological Science articles reported open data. After badges, 23% reported open data, with an accelerating trend; 39% reported open data in the first half of 2015, an increase of more than an order of magnitude from baseline. There was no change over time in the low rates of data sharing among comparison journals. Moreover, reporting openness does not guarantee openness. When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned. Open materials also increased to a weaker degree, and there was more variability among comparison journals. Badges are simple, effective signals to promote open practices and improve preservation of data and materials by using independent repositories.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Poor distribution of labor and lack of task specialization may be part of the reason data and code sharing remain rare [48,54]”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [54]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, simply making data available does not guarantee that the data have utility: Kidwell et al. also observed that a substantial proportion (50/111, 45%) of reportedly available data was in fact not available or was incomplete, incorrect, or had insufficient documentation.”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, in an observational study examining the impact of a voluntary ‘open badges’ scheme at the journal Psychological Science, Kidwell et al. [10] found that the proportion of articles reporting available data increased from around 2.5% during the pre-policy period to 39.5% by the end of the post-policy period.”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reproducibility badges adopted by several journals45 recognize stronger forms of reproducibility without penalizing authors who cannot yet reach them.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They then studied what happened when the journal Psychological Science introduced the open data badge and found a substantial increase in the percentage of publications reporting open data [54].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [54]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Spence, M. (1973) · The Quarterly Journal of Economics · doi:10.2307/1882010 · Cited by 19,518
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Signaling Theory: A Review and Assessment
Connelly, B. L., Certo, S. T., Ireland, R. D., & Reutzel, C. R. (2011) · Journal of Management · doi:10.1177/0149206310388419 · Cited by 6,735
The Sociology of Science: Theoretical and Empirical Investigations
Kaplan, N., Merton, R. K., & Storer, N. W. (1974) · Technology and Culture · doi:10.2307/3102264 · Cited by 4,005
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Self-citationTHE SOCIOLOGY OF SCIENCE:: Theoretical and Empirical Investigations
Merton, R. Κ., & Storer, N. W. (2017) · Cited by 1,462
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Self-citationThe poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543 · cited 1×
“The key problem with “upon request” policies is that requests come after the researcher has completed the research, published the article, and lost or forgot detailed materials and data [9,11,16].”
BackgroundPromoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
Self-citation
Research works printed but not linked: 1
- RK Merton, 1942, Science and technology in a democratic order, J of Legal and Pol Soc, 1, 115
Cited by
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Experimental Physiology · doi:10.1113/ep088870 · Cited by 3,497
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationThe preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
“It is possible that such incentives will become nearly as effective as badges for open data, which were associated with more than a 10-fold increase in data sharing in an initial test (58).”
BackgroundSelf-citationThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x20943823 · Cited by 1,443
“The bioscience community has introduced a range of initiatives to address the problem, from open access and open practices to enable the scrutiny of all aspects of the research [4, 5] through to study preregistration to shift the focus towards robust methods rather than the novelty of the results [6, 7], as well as re…”
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · British Journal of Pharmacology · doi:10.1111/bph.15193 · Cited by 1,081
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
“One element that could help nudging researchers towards properly powered studies may be the inclusion of power in the badge system used in some journals to promote good research practices (Chambers, 2013; Lindsay, 2015, 2017; Kidwell et al., 2016; Nosek & Lakens, 2014).”
BackgroundPsychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
“Initial evidence from a single adopting journal, Psychological Science, and a sample of comparison journals suggests that this simple incentive increases data sharing rates from less than 3% to more than 38% (Kidwell et al., 2016).”
BackgroundSelf-citationReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationWicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
“These transparent practices have many benefits and are currently gaining traction (e.g., Eich, 2014; Kidwell et al., 2016), but are often insufficiently detailed, documented or structured to allow other researchers to reproduce and replicate results (e.”
BackgroundThe Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support
Razavi, S., Jakeman, A., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Lo Piano, S., Iwanaga, T., Becker, W., et al. (2021) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104954 · Cited by 628
“Some journals now award “ open code badges ” (Kidwell et al., 2016) to highlight publications with accessible code and data.”
The ARRIVE guidelines 2.0: updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · The Journal of Physiology · doi:10.1113/jp280389 · Cited by 593
“…introduced a range of initiatives to address the problem, from open access and open practices to enable the scrutiny of all aspects of the research (Kidwell et al. 2016, Else, 2018) through to study preregistration to shift the focus towards robust methods rather than the novelty of the results…”
BackgroundOpen science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“The bounds of the confidence intervals shown for traditional literature were based on estimates (5% and 20%, respectively) of null findings that have been previously reported for traditional literature [46,47].”
BackgroundMethodResultInfluentialNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
“…design replications (Brandt et al. 2014, Schmidt 2009), special issues dedicated to replications (Nosek & Lakens 2013, Pashler & Wagenmakers 2012), multilab replication efforts (Ebersole et al. 2016, Klein et al. 2014), and the creation of registered replication reports (Simons et al. 2014).”
BackgroundWhat You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · BMC Veterinary Research · doi:10.1186/s12917-020-02451-y · Cited by 496
“The bioscience community has introduced a range of initiatives to address the problem, from open access and open practices to enable the scrutiny of all aspects of the research [4, 5] through to study preregistration to shift the focus towards robust methods rather than the novelty of the results [6, 7], as well as re…”
Registration of systematic reviews in PROSPERO: 30,000 records and counting
Page, M. J., Shamseer, L., & Tricco, A. C. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0699-4 · Cited by 509
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Assessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462
“The journal Psychological Science has evaluated whether digital badges result in more data sharing [48].”
BackgroundData sharing practices and data availability upon request differ across scientific disciplines
Tedersoo, L., Küngas, R., Oras, E., Köster, K., Eenmaa, H., Leijen, Ä., Pedaste, M., Raju, M., Astapova, A., Lukner, H., et al. (2021) · Scientific Data · doi:10.1038/s41597-021-00981-0 · Cited by 387
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Implications of the Credibility Revolution for Productivity, Creativity, and Progress
Vazire, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617751884 · Cited by 359
“One major theme of the credibility revolution is the need for greater transparency in the research and publication process (Kidwell et al., 2016; Lupia & Elman, 2014; Nosek et al., 2015; Spellman, Gilbert, & Corker, 2017; Vazire, 2017).”
BackgroundInvestigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Self-citationMoshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., Grahe, J. E., McCarthy, R. J., Musser, E. D., Antfolk, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918797607 · Cited by 377
Self-citationIntegrating explanation and prediction in computational social science
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03659-0 · Cited by 371
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
“Furthermore, even when data are shared, inadequate documentation and formatting can render them unusable [10].”
BackgroundMethodInfluentialSelf-citationData Sharing in Psychology: A Survey on Barriers and Preconditions
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917751886 · Cited by 283
“In terms of effectiveness, Kidwell et al. (2016) found that after the premier journal Psychological Science had adopted the open data badges, data sharing rates increased dramatically.”
BackgroundOur path to better science in less time using open data science tools
Lowndes, J. S. S., Best, B. D., Scarborough, C., Afflerbach, J. C., Frazier, M. R., O’Hara, C. C., Jiang, N., & Halpern, B. S. (2017) · Nature Ecology & Evolution · doi:10.1038/s41559-017-0160 · Cited by 286
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · BMJ Open Science · doi:10.1136/bmjos-2020-100115 · Cited by 273
The Open Innovation in Science research field: a collaborative conceptualisation approach
Beck, S., Bergenholtz, C., Bogers, M., Brasseur, T.-M., Conradsen, M. L., Di Marco, D., Distel, A. P., Dobusch, L., Dörler, D., Effert, A., et al. (2022) · Industry and Innovation · doi:10.1080/13662716.2020.1792274 · Cited by 254
“…have found that researchers are more likely to share their research data when a badge indicating whether results have been reproduced or replicated is published along with their article (Kidwell et al. 2016), although the evidence here is mixed (Rowhani-Farid, Aldcroft, and Barnett 2020).”
BackgroundMarsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
A design framework and exemplar metrics for FAIRness
Wilkinson, M. D., Sansone, S.-A., Schultes, E., Doorn, P., Bonino da Silva Santos, L. O., & Dumontier, M. (2018) · Scientific Data · doi:10.1038/sdata.2018.118 · Cited by 211
Misrepresentation and distortion of research in biomedical literature
Boutron, I., & Ravaud, P. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710755115 · Cited by 273
“The use of a badge to acknowledge open practices has been effective in changing researchers’ behavior (86).”
BackgroundWallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006930 · Cited by 269
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra: Psychology · doi:10.1525/collabra.158 · Cited by 233
“…and scientific journals have endorsed transparent and reproducible research practices through the TOP guidelines (Nosek et al., 2015)5 and major psychology journals have begun implementing policy changes that encourage or mandate sharing (see e.g., Kidwell et al., 2016; Nuijten et al., 2017).”
BackgroundSelf-citationAn Agenda for Open Science in Communication
Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., Lukito, J., Bier, L. M., Zhang, R., Johnson, B. K., et al. (2021) · Journal of Communication · doi:10.1093/joc/jqz052 · Cited by 209
A Unified Framework to Quantify the Credibility of Scientific Findings
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918787489 · Cited by 214
“conduct a fair replication accessible in a public repository (Kidwell et al., 2016).”
BackgroundRecommendations for Increasing the Transparency of Analysis of Preexisting Data Sets
Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919848684 · Cited by 203
“The current set of badges—for open data, open materials, and preregistration (Kidwell et al., 2016)—have been adopted by a number of psychology journals, including Psychological Science (Eich, 2014) and Advances in Methods and Practices in Psychological Science.”
BackgroundMethodNaudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A. (2018) · BMJ · doi:10.1136/bmj.k400 · Cited by 227
Data sharing in PLOS ONE: An analysis of Data Availability Statements
Federer, L. M., Belter, C. W., Joubert, D. J., Livinski, A., Lu, Y.-L., Snyders, L. N., & Thompson, H. (2018) · PLOS ONE · doi:10.1371/journal.pone.0194768 · Cited by 208
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
“It is known, for example, that specific journal policies can be associated with increases in data and materials availability [6,27,28].”
BackgroundSelf-citationPsychology's Replication Crisis and Clinical Psychological Science
Tackett, J. L., Brandes, C. M., King, K. M., & Markon, K. E. (2019) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-050718-095710 · Cited by 198
“Since implementing badges in psychology journals, data sharing has risen from approximately 3% to 40% of articles in participating journals (with no change in nonparticipants), and open materials have likewise increased, though to a lesser extent (Kidwell et al. 2016).”
BackgroundTransparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201
“Interestingly, an incentive to archive in the form of a badge may 240 be similarly effective [48] as requiring archiving [49] and could therefore eliminate much of the 241 controversy regarding archiving [e.”
BackgroundA methodological synthesis of self-paced reading in second language research
MARSDEN, E., THOMPSON, S., & PLONSKY, L. (2018) · Applied Psycholinguistics · doi:10.1017/s0142716418000036 · Cited by 180
“…8 begin to recognize authors for fully open methodological transparency, by, for example, adopting the Centre for Open Science badge scheme (Blohowiak et al., 2016), which has been shown to increase the long-term availability of materials and data (Kidwell et al., 2016).”
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
“Such systematic shifts in policies help to avoid collective-action problems such that only a subset of scientists engage in best practices that increase research quality but may also reduce productivity, which risks placing them at a professional disadvantage (Kidwell et al., 2016).”
BackgroundSelf-citationHardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187
“Journal policies that use badges to encourage data sharing (Kidwell et al., 2016) or mandate data sharing (Hardwicke et al., 2018; Nuijten et al., 2017) have been associated with marked increases in data availability in the journals that adopted them.”
BackgroundSelf-citationAssessing data availability and research reproducibility in hydrology and water resources
Stagge, J. H., Rosenberg, D. E., Abdallah, A. M., Akbar, H., Attallah, N. A., & James, R. (2019) · Scientific Data · doi:10.1038/sdata.2019.30 · Cited by 165
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
A consensus-based transparency checklist
Aczel, B., Szaszi, B., Sarafoglou, A., Kekecs, Z., Kucharský, Š., Benjamin, D., Chambers, C. D., Fisher, A., Gelman, A., Gernsbacher, M. A., et al. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0772-6 · Cited by 152
The Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 241
Reproducible and replicable pain research: a critical review
Lee, H., Lamb, S. E., Bagg, M. K., Toomey, E., Cashin, A. G., & Moseley, G. L. (2018) · Pain · doi:10.1097/j.pain.0000000000001254 · Cited by 150
“Some are relatively straightforward changes, for example journals educating readers on the pertinent issues, ‘badging’ papers with optimal transparency [31], publishing registered reports, and mandating pre-registration for all study designs.”
BackgroundStrengthening the Practice of Exercise and Sport-Science Research
Halperin, I., Vigotsky, A. D., Foster, C., & Pyne, D. B. (2017) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.2017-0322 · Cited by 160
“10 For example, since 2014, when Psychological Science announced it would award badges for data-sharing behaviors, the average data-sharing rate increased tenfold to 38% from 2013 to 2015.”
BackgroundInfluentialThe dire disregard of measurement invariance testing in psychological science.
Maassen, E., D'Urso, E. D., van Assen, M. A. L. M., Nuijten, M. B., Roover, K. D., & Wicherts, J. M. (2023) · Psychological Methods · doi:10.1037/met0000624 · Cited by 151
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
“However, the extent to which badges can be used to increase open scholarship behaviours remains unclear; while one study 72 reports increased data sharing rates among articles published in Psychological Science with badges, a recent randomized control trial shows no evidence for badges increasing…”
BackgroundWhat incentives increase data sharing in health and medical research? A systematic review
Rowhani-Farid, A., Allen, M. J., & Barnett, A. G. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0028-9 · Cited by 146
“A limitation of the badge study was that it did not use a randomized parallel group design; notwithstanding, it was the only incentive that was tested in the health and medical research community, with pre- and postincentive empirical data [15].”
BackgroundMethodInfluentialPromoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Adelson, J. L., Barton, E., Bradshaw, C., Bryant, B. R., Bryant, D., Cook, B. G., Coyne, M., DeBettencourt, L., DeHaven, A. C., Dymond, S., et al. (2018) · Exceptional Children · doi:10.1177/0014402918793138 · Cited by 143
“Offering openscience badges has been associated with an increase in data sharing (Kidwell et al., 2016) in the absence of any mandate to do so.”
BackgroundInfluentialSelf-citationTowards wide-scale adoption of open science practices: The role of open science communities
Armeni, K., Brinkman, L., Carlsson, R., Eerland, A., Fijten, R., Fondberg, R., Heininga, V. E., Heunis, S., Koh, W. Q., Masselink, M., et al. (2021) · Science and Public Policy · doi:10.1093/scipol/scab039 · Cited by 140
Transparency of CHI Research Artifacts: Results of a Self-Reported Survey
Wacharamanotham, C., Eisenring, L., Haroz, S., & Echtler, F. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376448 · Cited by 118
A proposal for the future of scientific publishing in the life sciences
Stern, B. M., & O’Shea, E. K. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000116 · Cited by 114
Reproducible research and GIScience: an evaluation using AGILE conference papers
Nüst, D., Granell, C., Hofer, B., Konkol, M., Ostermann, F., Šileryté, R., & Cerutti, V. (2018) · PeerJ · doi:10.7717/peerj.5072 · Cited by 137
Packaging Data Analytical Work Reproducibly Using R (and Friends)
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · The American Statistician · doi:10.1080/00031305.2017.1375986 · Cited by 123
“A cross-discipline system of signaling open data and materials accompanying journal articles with badges is being used to incentivize openness in a dozen journals (Kidwell et al. 2016).”
BackgroundInterpersonal Movement Synchrony Responds to High- and Low-Level Conversational Constraints
Paxton, A., & Dale, R. A. C. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01135 · Cited by 127
“Current best practices for open science include the sharing of data and code in public repositories (see Nosek et al., 2015; Blohowiak et al., 2016; Gewin, 2016; Kidwell et al., 2016).”
BackgroundLearning From Testimony on Quantitative Research in Management
King, A. A., Goldfarb, B. D., & Simcoe, T. S. (2019) · Academy of Management Review · doi:10.2139/ssrn.3816190 · Cited by 135
Reproducible research practices are underused in systematic reviews of biomedical interventions
Page, M. J., Altman, D. G., Shamseer, L., McKenzie, J. E., Ahmadzai, N., Wolfe, D., Yazdi, F., Catalá-López, F., Tricco, A. C., & Moher, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.017 · Cited by 115
A methodological synthesis and meta-analysis of judgment tasks in second language research
Plonsky, L., Marsden, E. J., Crowther, D., Gass, S. M., & Spinner, P. (2019) · Second language Research · doi:10.1177/0267658319828413 · Cited by 112
“This situation may change as more journals encourage their authors towards greater transparency (by, for example, adopting the Centre for Open Science badge scheme (Blohowiak et al., 2016) shown to increase long-term availability of materials and data (Kidwell et al., 2016)).”
BackgroundOpen and reproducible neuroimaging: From study inception to publication
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119623 · Cited by 126
Reproducibility and replicability: opportunities and challenges for geospatial research
Kedron, P., Li, W., Fotheringham, S., & Goodchild, M. (2021) · International Journal of Geographical Information Science · doi:10.1080/13658816.2020.1802032 · Cited by 108
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108
Self-citationQuality Uncertainty Erodes Trust in Science
Vazire, S. (2017) · Collabra: Psychology · doi:10.1525/collabra.74 · Cited by 106
“…change is rare, and the most successful academic journals are unlikely to change on their own, for fear of harming their reputations (though see Kidwell et al., 2016 and Piwoward, Day, & Fridsma, 2007, for evidence that transparency does not seem to have a negative impact on traditional…”
BackgroundMetaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F. M., Chee, Y. E., Wintle, B. C., Burgman, M. A., Mccarthy, M. A., & Gordon, A. (2016) · BioScience · doi:10.1093/biosci/biw159 · Cited by 123
A Brief Guide to Evaluate Replications
LeBel, E. P., Vanpaemel, W., Cheung, I., & Campbell, L. (2019) · Meta-Psychology · doi:10.15626/mp.2018.843 · Cited by 107
“, with a codebook) on a public repository (criteria for earning open materials and open data badges, respectively; Kidwell et al., 2016) and following the relevant reporting standards for the type of study and methodology used (e.”
MethodArtino, A. R., Driessen, E. W., & Maggio, L. A. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002412 · Cited by 111
Data sharing and how it can benefit your scientific career
Popkin, G. (2019) · Nature · doi:10.1038/d41586-019-01506-x · Cited by 112
The five pillars of computational reproducibility: bioinformatics and beyond
Ziemann, M., Poulain, P., & Bora, A. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad375 · Cited by 121
“If a workflow is executed successfully and tests are passed, the repository and linked research article can be decorated with a ‘badge’ to indicate computational reproducibility, similar to badges used for other Open Science practices [144].”
MethodGiofrè, D., Cumming, G., Fresc, L., Boedker, I., & Tressoldi, P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0175583 · Cited by 98
“Recent research indicates that the implementation of these new guidelines was found to be very effective in promoting the use of Open Practices from January 2012 to May 2015 [25].”
MethodResultIs authorship sufficient for today’s collaborative research? A call for contributor roles
Vasilevsky, N., Hosseini, M., Teplitzky, S., Ilik, V., Mohammadi, E., Schneider, J., Kern, B., Colomb, J., Edmunds, S., Gutzman, K. E., et al. (2020) · Accountability in Research · doi:10.1080/08989621.2020.1779591 · Cited by 108
“Badges that acknowledge open science practices have been used by the Open Science Foundation to provide incentives for researchers (Kidwell et al. 2016).”
MethodUpon the Shoulders of Giants: Open-Source Hardware and Software in Analytical Chemistry
Dryden, M. D. M., Fobel, R., Fobel, C., & Wheeler, A. R. (2017) · Analytical Chemistry · doi:10.1021/acs.analchem.7b00485 · Cited by 97
Both Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 110
“, 2015), recognizing Open Practices by awarding electronic badges (Kidwell et al., 2016), allowing for submissions of Registered Reports (Chambers, 2019), and requiring that researchers disclose methodological and design specifics (LeBel et al.”
Martone, M. E., García, A., & VandenBos, G. R. (2018) · American Psychologist · doi:10.1037/amp0000242 · Cited by 99
“…the use of Open Science Foundation badges in its journal Psychological Science to recognize authors who share their data and materials (Eich, 2014) According to analyses, display of these badges is correlated with significantly increased sharing of data (Lindsay, 2017; Kidwell et al., 2016).”
BackgroundDid awarding badges increase data sharing inBMJ Open? A randomized controlled trial
Rowhani-Farid, A., Aldcroft, A., & Barnett, A. G. (2020) · Royal Society Open Science · doi:10.1098/rsos.191818 · Cited by 101
“These findings stand in contrast with those of Kidwell et al. [13] who found that Open Data Badges were associated with an increase in data sharing from 1.5% to 39% at Psychological Science [13].”
BackgroundMethodInfluentialSignaling the trustworthiness of science
Jamieson, K. H., McNutt, M., Kiermer, V., & Sever, R. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1913039116 · Cited by 89
Core competencies for scientific editors of biomedical journals: consensus statement
Moher, D., Galipeau, J., Alam, S., Barbour, V., Bartolomeos, K., Baskin, P., Bell-Syer, S., Cobey, K. D., Chan, L., Clark, J., et al. (2017) · BMC Medicine · doi:10.1186/s12916-017-0927-0 · Cited by 81
Information Visualization Evaluation Using Crowdsourcing
Borgo, R., Micallef, L., Bach, B., McGee, F., & Lee, B. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13444 · Cited by 102
Actionable recommendations for narrowing the science-practice gap in open science
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.02.007 · Cited by 95
The Pandemic as a Portal: Reimagining Psychological Science as Truly Open and Inclusive
Ledgerwood, A., Hudson, S. T. J., Lewis, N. A., Maddox, K. B., Pickett, C. L., Remedios, J. D., Cheryan, S., Diekman, A. B., Dutra, N. B., Goh, J. X., et al. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211036654 · Cited by 100
“…have conducted metascientific studies of both demographic representation (Roberts et al., 2020) and the adoption of open-science practices in journals (Kidwell et al., 2016), journals could conduct annual self-studies and use the results to guide editorial policies and reviewer guidelines.”
BackgroundMethodContainers for computational reproducibility
Moreau, D., Wiebels, K., & Boettiger, C. (2023) · Nature Reviews Methods Primers · doi:10.1038/s43586-023-00236-9 · Cited by 119
“Scientists often build upon workflows of each other and share data and code publicly 10,11 .”
BackgroundExploratory reports: A new article type for Cortex
McIntosh, R. D. (2017) · Cortex · doi:10.1016/j.cortex.2017.07.014 · Cited by 94
Publication Bias and Editorial Statement on Negative Findings
Blanco-Perez, C., & Brodeur, A. (2020) · The Economic Journal · doi:10.1093/ej/ueaa011 · Cited by 76
Credibility of preprints: an interdisciplinary survey of researchers
Soderberg, C. K., Errington, T. M., & Nosek, B. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.201520 · Cited by 98
Self-citationSpellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 92
Science as behaviour: Using a behaviour change approach to increase uptake of open science
Norris, E., & O’Connor, D. B. (2019) · Psychology & Health · doi:10.1080/08870446.2019.1679373 · Cited by 81
“Strategies used so far to help move researchers towards Open Science practices have largely focused on the provision of incentives such as journal badges recognizing pre-registration of research protocols, open data and open materials (Kidwell et al., 2016).”
BackgroundMethods to Address Confounding and Other Biases in Meta-Analyses: Review and Recommendations
Mathur, M. B., & VanderWeele, T. J. (2022) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-051920-114020 · Cited by 85
Has open data arrived at the British Medical Journal (BMJ) ? An observational study
Rowhani-Farid, A., & Barnett, A. G. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011784 · Cited by 84
Qualitative data sharing and synthesis for sustainability science
Alexander, S. M., Jones, K., Bennett, N. J., Budden, A. E., Cox, M. E., Crosas, M., Game, E. T., Geary, J., Hardy, D., Johnson, J. T., et al. (2019) · Nature Sustainability · doi:10.1038/s41893-019-0434-8 · Cited by 96
Wagge, J., Brandt, M. J., Lazarević, L. B., Legate, N., Christopherson, C. D., Wiggins, B., & Grahe, J. E. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00247 · Cited by 84
“Figure 1 highlights that students are, along the way, participating in some of the critical requirements of open science and transparent methodology: open methods, open data, and preregistration of hypotheses (Kidwell et al., 2016).”
BackgroundMethodInfluentialSelf-citationA survey on how preregistration affects the research workflow: better science but more work
Sarafoglou, A., Kovács, M., Bakos, B. E., Wagenmakers, E., & Aczél, B. (2022) · Royal Society Open Science · doi:10.1098/rsos.211997 · Cited by 89
“Online repositories have been created to store preregistrations (e.g. the Open Science Framework (OSF; https://osf.io) and AsPredicted.org), and several journals recognize preregistered studies with badges [10].”
How to Automatically Document Data With the codebook Package to Facilitate Data Reuse
Arslan, R. C. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919838783 · Cited by 89
Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79
“15 We note that Kidwell et al. (2016) found that some articles that did share data did not receive an Open Data Badge, but it was unclear why.”
BackgroundResultInfluentialGood Research Practice in Non-Clinical Pharmacology and Biomedicine
Bespalov, A., Michel, M. C., & Steckler, T. (2020) · Handbook of experimental pharmacology · doi:10.1007/978-3-030-33656-1 · Cited by 91
Metadata
- Authors
- Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S., & Nosek, B. A.
- Affiliations
- Center for Open Science (US)
- University of Belgrade (RS)
- University of California, Riverside (US)
- University College London (GB)
- Max Planck Institute for Behavioral Economics (DE)
- Mississippi State University (US)
- University of Vienna (AT)
- Corresponding author
- Kidwell, Mallory C.
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-05-12
- Volume
- 14
- Issue
- 5
- Pages
- e1002456
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4865119
- OpenAlex
- W2390706404
- Cited by
- 700
- References
- 19
- Citations by year
- 2026: 31 · 2025: 35 · 2024: 39 · 2023: 71 · 2022: 110 · 2021: 87 · 2020: 98 · 2019: 80 · 2018: 70 · 2017: 54 · 2016: 25
- FWCI
- 170.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- WU Research (publishedVersion)
- Phaidra (Universität Wien) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Faculty of Philosophy (University of Belgrade) (submittedVersion, cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- The Laura and John Arnold Foundation
- The John Templeton Foundation
- National Institute on Aging
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- open badges, open data, open practices, research transparency, data sharing, open materials, data preservation, scholarly publishing, author incentives
- MeSH terms
- Cost-Benefit Analysis, Psychology, Publishing, Serial Publications, Internet, Information Dissemination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.