Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The widespread reluctance to share published research data is often hypothesized to be due to the authors' fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically. METHODS AND FINDINGS: We related the reluctance to share research data for reanalysis to 1148 statistically significant results reported in 49 papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance. CONCLUSIONS: Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving policies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To avoid bias in the prevalence of reporting errors, we performed a validity study of statcheck, in which we compared statcheck’s results with the results of Wicherts, Bakker, and Molenaar (2011), who performed a manual search for and verification of reporting errors in a sample of 49 articles.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts, Bakker, and Molenaar (2011)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, Wicherts et al. included all results reported in the text or a table in the Results section of an article.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al., 2011
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Moreover, Wicherts et al. (2011) reported that a higher prevalence of reporting errors were associated with a failure to share data upon request.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al. (2011)
The cited source says what the claim attributes to it.
In the citing paper
“the current study counts p = .000 as incorrect, whereas this was not the case in for example Wicherts et al. (2011; see also Appendix A).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al. (2011)
The cited source says what the claim attributes to it.
In the citing paper
“Wicherts et al. (2011) included only t, F, and χ 2 values of which the reported p-value was smaller than .05”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al. (2011)
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Statistical analyses of research data are quite error prone [1,2,3], accounts of statistical results may be inaccurate [4], and decisions that researchers make during the analytical phase of a study may lean towards the goal of achieving a preferred (significant) result [5,6,7,8].”
BackgroundCohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
THE JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION
(1911) · Journal of the American Medical Association · doi:10.1001/jama.1911.04260070032017 · Cited by 3,361
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“This procedure is not without its critics [24,25], but remains to be used extensively in psychology [26] and related fields.”
BackgroundStatistical Methods for Research Workers.
Hotelling, H., & Fisher, R. A. (1930) · Journal of the American Statistical Association · doi:10.2307/2278215 · Cited by 1,529
International Journal of Methods in Psychiatric Research
(2006) · International Journal of Methods in Psychiatric Research · doi:10.1002/(issn)1557-0657 · Cited by 1,492
Publication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Cited by
Witwer, K. W., Buzás, E. I., Bemis, L. T., Bora, A., Lässer, C., Lötvall, J., Nolte‐‘t Hoen, E. N., Piper, M. G., Sivaraman, S., Skog, J., et al. (2013) · Journal of Extracellular Vesicles · doi:10.3402/jev.v2i0.20360 · Cited by 2,421
“Database submission Data submission is mandated by most journals (95,96) and independent studies have found that unavailability of data is associated with low data or study quality (94, 97,98); however, compliance remains poor (94,99,100).”
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
“Even diligent reviewers miss lots of errors (Schroter et al., 2008; Wicherts et al., 2011).”
Increasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Self-citationThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Self-citationReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
“Another solution is that the data are available for independent reanalysis after publication, although this is not always possible due to low sharing rates (Wicherts et al., 2011).”
BackgroundSelf-citationThe GRoLTS-Checklist: Guidelines for Reporting on Latent Trajectory Studies
van de Schoot, R., Sijbrandij, M., Winter, S. D., Depaoli, S., & Vermunt, J. K. (2017) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2016.1247646 · Cited by 721
“There is a growing awareness that open and transparent research is essential to maintaining and improving the quality of science (Asendorpf et al., 2013; Miguel et al., 2014; Nosek et al., 2015; Wicherts, 2013; Wicherts, Bakker, & Molenaar, 2011).”
Best practices in data analysis and sharing in neuroimaging using MRI
Nichols, T. E., Das, S., Eickhoff, S. B., Evans, A. C., Glatard, T., Hanke, M., Kriegeskorte, N., Milham, M. P., Poldrack, R. A., Poline, J.-B., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4500 · Cited by 714
Data reuse and the open data citation advantage
Piwowar, H. A., & Vision, T. J. (2013) · PeerJ · doi:10.7717/peerj.175 · Cited by 603
The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.3 · Cited by 560
Gorgolewski, K. J., Varoquaux, G., Rivera, G., Schwarz, Y., Ghosh, S. S., Maumet, C., Sochat, V. V., Nichols, T. E., Poldrack, R. A., Poline, J.-B., et al. (2015) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2015.00008 · Cited by 572
“We then filtered our set to maps contrasting “stop” and “go” conditions, which resulted in eight maps across four studies (see Table 1 ).”
BackgroundMethodNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
“The APA already says that data should be available upon request, but in practice, they are not (see e.g., Wicherts et al. 2006, 2011).”
BackgroundMethodSimpson's paradox in psychological science: a practical guide
Kievit, R. A., Frankenhuis, W. E., Waldorp, L. J., & Borsboom, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00513 · Cited by 542
“Based on repeated measures of individuals on these dimensions, they found that the factor structure that described the inter-individual differences (which in their sample could be described by two dimensions) did not accurately capture the dimensions along which the individuals in that sample…”
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
“To avoid bias in the prevalence of reporting errors, we performed a validity study of statcheck, in which we compared statcheck ’ s results with the results of Wicherts, Bakker, and Molenaar (2011), who performed a manual search for and verification of reporting errors in a sample of 49 articles.”
InfluentialSelf-citationThe past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
“Concerns about lack of reproducibility are not entirely surprising considering the relative lack of methodological transparency about the process of conducting empirical research that eventually leads to a published article (Banks et al., 2016a; Bedeian, Taylor, & Miller, 2010; John, Loewenstein, & Prelec, 2012; O’Boy…”
BackgroundStar Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
Making big data open: data sharing in neuroimaging
Poldrack, R. A., & Gorgolewski, K. J. (2014) · Nature Neuroscience · doi:10.1038/nn.3818 · Cited by 425
“Wicherts et al. 24 requested data for a large number of psychological studies and then compared papers for which data either were or were not voluntarily shared upon request.”
ResultKühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399
“Data sharing practices have been found to be somewhat lacking and have been put forward as one reason impeding scientific progress and replication [41].”
BackgroundData sharing practices and data availability upon request differ across scientific disciplines
Tedersoo, L., Küngas, R., Oras, E., Köster, K., Eenmaa, H., Leijen, Ä., Pedaste, M., Raju, M., Astapova, A., Lukner, H., et al. (2021) · Scientific Data · doi:10.1038/s41597-021-00981-0 · Cited by 387
Psychological interventions for parents of children and adolescents with chronic illness
Eccleston, C., Palermo, T. M., Fisher, E., & Law, E. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009660.pub2 · Cited by 369
What Drives Academic Data Sharing?
Fecher, B., Friesike, S., & Hebing, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118053 · Cited by 338
Toward open sharing of task-based fMRI data: the OpenfMRI project
Poldrack, R. A., Barch, D. M., Mitchell, J. P., Wager, T. D., Wagner, A. D., Devlin, J. T., Cumba, C., Koyejo, O., & Milham, M. P. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00012 · Cited by 342
“…to share; an analysis of psychology papers for which data were shared upon request vs. those that were not shared found that papers for which data were not shared had a higher rate of apparent errors in statistical reporting as well as having smaller effect sizes on average (Wicherts et al., 2011).”
BackgroundThe importance of open data and software: Is energy research lagging behind?
Pfenninger, S., DeCarolis, J., Hirth, L., Quoilin, S., & Staffell, I. (2017) · Energy Policy · doi:10.1016/j.enpol.2016.11.046 · Cited by 335
“Indeed, a reluctance to share data was shown to be associated with weaker evidence (Wicherts et al., 2011).”
BackgroundPublic Data Archiving in Ecology and Evolution: How Well Are We Doing?
Roche, D. G., Kruuk, L. E. B., Lanfear, R., & Binning, S. A. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002295 · Cited by 303
“Despite the obvious benefits of PDA for science, many researchers remain reluctant to share their data publicly [1,3,12,14–19].”
BackgroundData Sharing in Psychology: A Survey on Barriers and Preconditions
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917751886 · Cited by 283
“For example, Wicherts, Bakker & Molenaar found willingness to share research data was related to the internal consistency of statistical results and strength of evidence (2011).”
Background(2021) · Research Design in Clinical Psychology · doi:10.1017/9781108993647.010 · Cited by 276
Marsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
A peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266
“This issue of hidden self-serving biases was touched upon in recent work by Wicherts, Bakker, and Molenaar (2011) .”
ResultStodden, V., Guo, P., & Ma, Z. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067111 · Cited by 257
“…has focused on changes needed in scientific publishing and the role of journal open data requirements in fostering scientific reliability [4,5,6,7] but little research has focused on availability of the code needed to replicate computational findings, although this has been associated with…”
BackgroundLazic, S. E., & Essioux, L. (2013) · BMC Neuroscience · doi:10.1186/1471-2202-14-37 · Cited by 237
Marwick, B. (2017) · Journal of Archaeological Method and Theory · doi:10.1007/s10816-015-9272-9 · Cited by 179
“Making data and code available enables others to inspect these materials to evaluate the reliability of the publication, and to include the materials into other projects, and may lead to higher quality and more impactful published research (Gleditsch and Strand 2003; Piwowar et al. 2007; Wicherts et al. 2011).”
BackgroundResearch Design in Clinical Psychology
Kazdin, A. E. (2021) · doi:10.1017/9781108993647 · Cited by 240
Psychological interventions for parents of children and adolescents with chronic illness
Eccleston, C., Fisher, E., Law, E., Bartlett, J., & Palermo, T. M. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009660.pub3 · Cited by 216
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213
Big data from small data: data-sharing in the 'long tail' of neuroscience
Ferguson, A. R., Nielson, J. L., Cragin, M. H., Bandrowski, A. E., & Martone, M. E. (2014) · Nature Neuroscience · doi:10.1038/nn.3838 · Cited by 213
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 206
Ernst, A. F., & Albers, C. J. (2017) · PeerJ · doi:10.7717/peerj.3323 · Cited by 194
Bakker, M., & Wicherts, J. M. (2014) · Psychological Methods · doi:10.1037/met0000014 · Cited by 209
“The practical use of statistical tests in psychological research often deviates from how the use of these tests is described in the textbooks (Bakker et al., 2012; John et al., 2012; Simmons et al., 2011; Wagenmakers et al., 2011; Wicherts et al., 2011).”
BackgroundSelf-citationPsychological interventions for parents of children and adolescents with chronic illness
Law, E., Fisher, E., Eccleston, C., & Palermo, T. M. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009660.pub4 · Cited by 207
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 198
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202
The academic, economic and societal impacts of Open Access: an evidence-based review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.1 · Cited by 182
“For example, the willingness to publicly share data has been associated with fewer statistical errors in the final research article (Wicherts et al., 2011).”
Lazic, S. E., & Essioux, L. (2016) · Cited by 182
Registered Reports: Realigning incentives in scientific publishing
Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015) · Cortex · doi:10.1016/j.cortex.2015.03.022 · Cited by 173
“…2015), hindsight bias in which researchers present a hypothesis derived from data as a priori (John et al., 2012; Kerr, 1998), lack of data sharing (Wicherts et al., 2011; Wicherts et al., 2006), failure to consider statistical power (Bezeau and Graves 2001; Button et al., 2013; Cohen, 1962), and…”
Questionable research practices among italian research psychologists
Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172792 · Cited by 162
Self-citationLaw, E. F., Fisher, E., Fales, J., Noel, M., & Eccleston, C. (2014) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsu032 · Cited by 159
A Practical Guide for Improving Transparency and Reproducibility in Neuroimaging Research
Gorgolewski, K. J., & Poldrack, R. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002506 · Cited by 160
An A–Z of Applied Linguistics Research Methods
Loewen, S., & Plonsky, L. (2016) · doi:10.1007/978-1-137-40322-3 · Cited by 156
Are We Wasting a Good Crisis? The Availability of Psychological Research Data after the Storm
Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015) · Collabra · doi:10.1525/collabra.13 · Cited by 152
Answers to 18 Questions About Open Science Practices
Banks, G. C., Field, J. G., Oswald, F. L., O’Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9547-8 · Cited by 178
“Journals affiliated with the APA have authors sign a contract that they will make their data available to peers to verify the findings (Wicherts et al., 2011).”
How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?
Kepes, S., & McDaniel, M. A. (2013) · Industrial and Organizational Psychology · doi:10.1111/iops.12045 · Cited by 124
Multisector Dynamics: Advancing the Science of Complex Adaptive Human‐Earth Systems
Reed, P. M., Hadjimichael, A., Moss, R. H., Brelsford, C. M., Burleyson, C., Cohen, S., Dyreson, A. R., Gold, D., Gupta, R. S., Keller, K. M., et al. (2022) · Earth s Future · doi:10.1029/2021ef002621 · Cited by 138
“Reproducibility makes it easier to repeat and confirm the findings of others (McNutt, 2014; Pfenninger et al., 2017; Wicherts et al., 2011).”
ResultNieuwland, M. S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.11.019 · Cited by 121
Archiving Primary Data: Solutions for Long-Term Studies
Mills, J. A., Teplitsky, C., Arroyo, B., Charmantier, A., Becker, P. H., Birkhead, T. R., Bize, P., Blumstein, D. T., Bonenfant, C., Boutin, S., et al. (2015) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.07.006 · Cited by 121
“economically [6] and permits the essential verification of existing results [7].”
BackgroundProsocial motives underlie scientific censorship by scientists: A perspective and research agenda
Clark, C. J., Jussim, L. J., Frey, K., Stevens, S. T., al‐Gharbi, M., Aquino, K. F., Bailey, J. M., Barbaro, N., Baumeister, R. F., Bleske-Rechek, A. L., et al. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2301642120 · Cited by 129
“For example, peer reviewers evaluate research more favorably when findings support their prior beliefs, theoretical orientations, and political views (61–63). credulity among scholars toward favorable outcomes (66).”
BackgroundScientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPreprint
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/k4bgq · Cited by 115
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108
“Some evidence, for example, implies that data availability is modestly associated with a lower prevalence of statistical reporting inconsistencies [20].”
Backgroundvan der Zee, T., & Reich, J. (2018) · AERA Open · doi:10.1177/2332858418787466 · Cited by 134
Open and reproducible neuroimaging: From study inception to publication
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119623 · Cited by 126
Outlier Removal and the Relation with Reporting Errors and Quality of Psychological Research
Bakker, M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103360 · Cited by 101
“[13] found that the reluctance to share data was related to weaker evidence, as the mean p value within articles was higher on average (closer to .”
BackgroundMethodResultInfluentialSelf-citationGiofrè, D., Cumming, G., Fresc, L., Boedker, I., & Tressoldi, P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0175583 · Cited by 98
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 116
“Secondly, the willingness to share research data has been related to the strength of the statistical significance and a higher prevalence of reporting statistical errors [79].”
BackgroundProgress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95
Bootstrapping in Applied Linguistics: Assessing its Potential Using Shared Data
Plonsky, L., Egbert, J., & Laflair, G. T. (2014) · Applied Linguistics · doi:10.1093/applin/amu001 · Cited by 89
“Based on our results, we agree with Wicherts et al.’s (2006) recommendation that”
BackgroundResultInfluentialMartone, M. E., García, A., & VandenBos, G. R. (2018) · American Psychologist · doi:10.1037/amp0000242 · Cited by 101
Kepes, S., Bennett, A. A., & McDaniel, M. A. (2014) · Academy of Management Learning and Education · doi:10.5465/amle.2013.0193 · Cited by 110
Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project
Rohrer, J. M., Tierney, W., Uhlmann, E. L., DeBruine, L. M., Heyman, T., Jones, B. C., Schmukle, S. C., Silberzahn, R., Willén, R. M., Carlsson, R., et al. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620964106 · Cited by 88
“…Chang & Li, 2018; Hard-wicke et al., 2018), write persuasive e-mails to get the original authors to share the underlying data (often to no avail; Wicherts et al., 2011), recalculate statistics because reported values are not always accurate (e.g., Nuijten et al., 2016), or apply advanced…”
BackgroundPoldrack, R. A. (2019) · Neuron · doi:10.1016/j.neuron.2018.11.030 · Cited by 84
Wilson, M. L. L., Resnick, P., Coyle, D., & Chi, E. H. (2013) · CHI '13 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/2468356.2479636 · Cited by 84
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 92
The what, why, and how of born-open data
Rouder, J. N. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0630-z · Cited by 92
Reproducibility of individual effect sizes in meta-analyses in psychology
Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., Olsson-Collentine, A., & Wicherts, J. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0233107 · Cited by 82
“Based on the finding that reluctance to share data is associated with more reporting errors in primary studies ([25]; but see also [26]), one would expect meta-analyses accompanied by open data to be of higher (reporting) quality.”
BackgroundSelf-citationActionable recommendations for narrowing the science-practice gap in open science
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.02.007 · Cited by 80
Interpretations and methods: Towards a more effectively self-correcting social psychology
Jussim, L. J., Crawford, J. T., Anglin, S. M., Stevens, S. T., & Duarte, J. L. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.003 · Cited by 89
Publish (your data) or (let the data) perish! Why not publish your data too?
Wicherts, J. M., & Bakker, M. (2012) · Intelligence · doi:10.1016/j.intell.2012.01.004 · Cited by 95
“…(2012), doi:10.1016/j.intell.2012.01.004 they may result in the correction of potential errors in the analyses and in the reporting of results (which are quite common; Bakker & Wicherts, 2011; Wicherts et al., 2011) and may even help prevent such errors in the first place (Wicherts, 2011).”
BackgroundInfluentialSelf-citationJournal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79
“10 It is possible that our sample contained articles that contained reporting inconsistencies because those inconsistencies were the topic of investigation (Bakker & Wicherts, 2014; Veldkamp et al., 2014; Wicherts et al., 2011).”
BackgroundResultInfluentialSelf-citationSijtsma, K. (2015) · Psychometrika · doi:10.1007/s11336-015-9446-0 · Cited by 96
Memory Detection 2.0: The First Web-Based Memory Detection Test
Kleinberg, B., & Verschuère, B. (2015) · PLoS ONE · doi:10.1371/journal.pone.0118715 · Cited by 82
“The recent crisis in psychological research [20][21][22] has raised renewed attention to power and replicability.”
BackgroundHeterogeneity in European Research Integrity Guidance: Relying on Values or Norms?
Godecharle, S., Nemery, B., & Dierickx, K. (2014) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/1556264614540594 · Cited by 82
Krawczyk, M., & Reuben, E. (2012) · Accountability in Research · doi:10.1080/08989621.2012.678688 · Cited by 72
“Another way of interpreting this tendency is to suppose that some working papers may be of inferior quality and authors are afraid of verification (Wicherts et al., 2011).”
BackgroundSakaluk, J. K. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.013 · Cited by 71
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114876 · Cited by 69
“Recent results however suggest that reported results from t, F, and x(2) tests in the scientific literature are characterized by a great deal of errors [11, 12, 13, 14, 15].”
BackgroundResultSelf-citationSolutions to the Credibility Crisis in Management Science
Byington, E. K., & Felps, W. (2016) · Academy of Management Learning and Education · doi:10.5465/amle.2015.0035 · Cited by 77
Marsden, E. J., Morgan‐Short, K., Trofimovich, P., & Ellis, N. C. (2018) · Language Learning · doi:10.1111/lang.12284 · Cited by 71
Replication and Reproducibility in Cross-Cultural Psychology
Milfont, T. L., & Klein, R. A. (2018) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117744892 · Cited by 73
Science Deserves Better: The Imperative to Share Complete Replication Files
Dafoe, A. (2014) · Political Science and Politics · doi:10.1017/s104909651300173x · Cited by 56
Hensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 78
Artner, R., Verliefde, T., Steegen, S., Gomes, S. L. B. T., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2021) · Psychological Methods · doi:10.1037/met0000365 · Cited by 88
The Dawn of Open Access to Phylogenetic Data
Magee, A. F., May, M. R., & Moore, B. R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0110268 · Cited by 62
“prevalence of apparent errors in the reporting of statistical results [30].”
BackgroundMethodInfluentialvan Witteloostuijn, A. C. (2016) · Cross Cultural & Strategic Management · doi:10.1108/ccsm-03-2016-0084 · Cited by 71
Letting the daylight in: Reviewing the reviewers and other ways to maximize transparency in science
Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00020 · Cited by 60
“In another study it was found that researchers who report such erroneous results are less likely to share their data for reanalysis (Wicherts et al., 2011).”
BackgroundSelf-citationAre you 110% sure? Modeling of fractions and proportions in strategy and management research
Villadsen, A. R., & Wulff, J. N. (2019) · Strategic Organization · doi:10.1177/1476127019854966 · Cited by 78
“Indeed, it has been shown that studies conducted by authors who share data are more likely to replicate (Wicherts et al., 2011).”
BackgroundLaurinavichyute, A., Yadav, H., & Vasishth, S. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104332 · Cited by 86
When Data Sharing Gets Close to 100%: What Human Paleogenetics Can Teach the Open Science Movement
Anagnostou, P., Capocasa, M., Milia, N., Sanna, E., Battaggia, C., Luzi, D., & Destro Bisol, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121409 · Cited by 63
“Nevertheless, all empirical studies conducted so far clearly show that when we move the focus from specific projects to the wider scale of research fields, data sharing turns out to be far from being common practice [14, 23–34].”
BackgroundFive common pitfalls in eye-tracking research
Godfroid, A., & Hui, B. (2020) · Second language Research · doi:10.1177/0267658320921218 · Cited by 69
“The sharing of primary data is still rather limited in our field (e.g. Marsden and Plonsky, 2018; Plonsky et al., 2015), as it has been in psychology (e.g. Wicherts et al., 2006, 2011).”
BackgroundWater, Water, Everywhere: Defining and Assessing Data Sharing in Academia
Tuyl, S. V., & Whitmire, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147942 · Cited by 68
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska-Sampaio, A. P., & Carneiro, C. F. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 49
Metadata
- Authors
- Wicherts, J. M., Bakker, M., & Molenaar, D.
- Affiliations
- University of Amsterdam (NL)
- Corresponding author
- Wicherts, Jelte M.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-11-02
- Volume
- 6
- Issue
- 11
- Pages
- e26828
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3206853
- OpenAlex
- W2072620887
- Cited by
- 424
- References
- 50
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 18 · 2023: 26 · 2022: 23 · 2021: 54 · 2020: 34 · 2019: 27 · 2018: 37 · 2017: 33 · 2016: 40 · 2015: 41 · 2014: 33 · 2013: 27 · 2012: 15
- FWCI
- 69.64
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- Tilburg University Research Portal (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (451-07-016)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Sociology and Political Science › Data Analysis and Archiving
- Keywords
- research data sharing, data archiving, reanalysis, statistical reporting errors, statistical significance, quality of evidence, null hypothesis, psychology journals, data verification
- MeSH terms
- Cooperative Behavior, Humans, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.