Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The widespread reluctance to share published research data is often hypothesized to be due to the authors' fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically. METHODS AND FINDINGS: We related the reluctance to share research data for reanalysis to 1148 statistically significant results reported in 49 papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance. CONCLUSIONS: Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving policies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To avoid bias in the prevalence of reporting errors, we performed a validity study of statcheck, in which we compared statcheck’s results with the results of Wicherts, Bakker, and Molenaar (2011), who performed a manual search for and verification of reporting errors in a sample of 49 articles.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts, Bakker, and Molenaar (2011)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, Wicherts et al. included all results reported in the text or a table in the Results section of an article.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al., 2011
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Moreover, Wicherts et al. (2011) reported that a higher prevalence of reporting errors were associated with a failure to share data upon request.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al. (2011)
The cited source says what the claim attributes to it.
In the citing paper
“the current study counts p = .000 as incorrect, whereas this was not the case in for example Wicherts et al. (2011; see also Appendix A).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al. (2011)
The cited source says what the claim attributes to it.
In the citing paper
“Wicherts et al. (2011) included only t, F, and χ 2 values of which the reported p-value was smaller than .05”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Wicherts et al. (2011)
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Statistical analyses of research data are quite error prone [1,2,3], accounts of statistical results may be inaccurate [4], and decisions that researchers make during the analytical phase of a study may lean towards the goal of achieving a preferred (significant) result [5,6,7,8].”
BackgroundCohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
THE JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION
(1911) · Journal of the American Medical Association · doi:10.1001/jama.1911.04260070032017 · Cited by 3,361
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“This procedure is not without its critics [24,25], but remains to be used extensively in psychology [26] and related fields.”
BackgroundStatistical Methods for Research Workers.
Hotelling, H., & Fisher, R. A. (1930) · Journal of the American Statistical Association · doi:10.2307/2278215 · Cited by 1,529
International Journal of Methods in Psychiatric Research
(2006) · International Journal of Methods in Psychiatric Research · doi:10.1002/(issn)1557-0657 · Cited by 1,492
Publication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Educational and Psychological Measurement
(1982) · Serials Review · doi:10.1080/00987913.1982.10763392 · Cited by 1,410
The Journal of Allergy and Clinical Immunology
(1999) · Journal of Allergy and Clinical Immunology · doi:10.1016/s0091-6749(99)70212-5 · Cited by 1,093
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090 · cited 11×
“A further four papers published in the two journals from which data were requested in 2005 were not included in the follow-up because they did not involve NHST or did not contain significant results on the basis of t, F, of x2 tests.”
ResultInfluentialPublication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013 · cited 1×
“05, are considered by many researchers [27,28] and reviewers [29] to lend support to the hypothesized effects.”
BackgroundStatistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Sharing Detailed Research Data Is Associated with Increased Citation Rate
Piwowar, H. A., Day, R. S., & Fridsma, D. B. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000308 · Cited by 881 · cited 2×
“However, it is likely that similar problems play a role in the widespread reluctance to share data in other scientific fields [13,14,15,16,17,18,19,20].”
BackgroundCalibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 854 · cited 1×
“05 (which have a low chance of occurring regardless of actual effect sizes [35]), were indeed more common in papers from which no data were shared (16.”
Perspectives on Psychological Science
Blase, A. E. K. A. S. L. (2011) · APS observer · Cited by 628
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543 · cited 4×
“In the summer of 2005, Wicherts and colleagues [12] contacted the corresponding authors of 141 papers that were published in the second half of 2004 in one of four high-ranked journals published by the APA: Journal of Personality and Social Psychology (JPSP), Developmental Psychology (DP), Journal of Consulting and Cl…”
BackgroundInfluentialSelf-citationStatistical Methods for Research Workers.
Jowett, G. H., & Fisher, R. A. (1956) · Applied Statistics · doi:10.2307/2985747 · Cited by 516
Data Withholding in Academic Genetics: Evidence From a National Survey
Campbell, E. G., Clarridge, B. R., Gokhale, M., Birenbaum, L., Hilgartner, S., Holtzman, N. A., & Blumenthal, D. (2002) · JAMA · doi:10.1001/jama.287.4.473 · Cited by 489 · cited 2×
“However, it is likely that similar problems play a role in the widespread reluctance to share data in other scientific fields [13,14,15,16,17,18,19,20].”
BackgroundScientist as Subject: The Psychological Imperative.
Lewis, L. S., & Mahoney, M. J. (1978) · Contemporary Sociology · doi:10.2307/2064480 · Cited by 412
Empirical Study of Data Sharing by Authors Publishing in PLoS Journals
Savage, C. J., & Vickers, A. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0007078 · Cited by 411 · cited 2×
“However, it is likely that similar problems play a role in the widespread reluctance to share data in other scientific fields [13,14,15,16,17,18,19,20].”
BackgroundThe (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 5×
“As discussed elsewhere [33], likely causes include (1) errors in the retrieval and copying of test statistics, degrees of freedom, and/or p-value (e.”
BackgroundResultInfluentialSelf-citationStrong claims and weak evidence: Reassessing the predictive validity of the IAT.
Blanton, H., Jaccard, J., Klick, J., Mellers, B., Mitchell, G., & Tetlock, P. E. (2009) · Journal of Applied Psychology · doi:10.1037/a0014665 · Cited by 295 · cited 1×
“Psychological research data are often amenable to alternative methods of analysis [6,22,30] that may affect what can be concluded from them (at least within the rules of NHST).”
MethodSelective Reporting Biases in Cancer Prognostic Factor Studies
Kyzas, P. A., Loizou, K. T., & Ioannidis, J. P. A. (2005) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/dji184 · Cited by 231 · cited 2×
“However, it is likely that similar problems play a role in the widespread reluctance to share data in other scientific fields [13,14,15,16,17,18,19,20].”
BackgroundWolins, L. (1962) · American Psychologist · doi:10.1037/h0038819 · Cited by 226 · cited 2×
“Statistical analyses of research data are quite error prone [1,2,3], accounts of statistical results may be inaccurate [4], and decisions that researchers make during the analytical phase of a study may lean towards the goal of achieving a preferred (significant) result [5,6,7,8].”
BackgroundVerifying the Solution from a Nonlinear Solver: A Case Study
McCullough, B. D., & Vinod, H. D. (2003) · American Economic Review · doi:10.1257/000282803322157133 · Cited by 220 · cited 2×
“However, it is likely that similar problems play a role in the widespread reluctance to share data in other scientific fields [13,14,15,16,17,18,19,20].”
BackgroundThe Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 185 · cited 1×
“05, are considered by many researchers [27,28] and reviewers [29] to lend support to the hypothesized effects.”
BackgroundIncongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181
Replication Standards for Quantitative Social Science: Why Not Sociology?
Freese, J. (2007) · Sociological Methods & Research · doi:10.1177/0049124107306659 · Cited by 177 · cited 3×
“Because existing guidelines on data sharing offer little promise for improvement [40], progress in psychological science and related fields would benefit from having research data itself be part of the process of replication [15,16], notably by the establishment by journals, professional organizations, and granting bo…”
BackgroundMcCullough, B. D., McGeary, K. A., & Harrison, T. D. (2006) · Journal of Money, Credit, and Banking · doi:10.1353/mcb.2006.0061 · Cited by 173 · cited 2×
“Changing policies require better educational training in data management and data archiving, which is currently suboptimal in many fields [36,37,38,39].”
BackgroundPitfalls in Human Research: Ten Pivotal Points
Eldred, S. H. (1977) · American Journal of Psychiatry · doi:10.1176/ajp.134.7.826 · Cited by 153
Sharing Publication-Related Data and Materials: Responsibilities of Authorship in the Life Sciences
(2003) · Plant Physiology · doi:10.1104/pp.900068 · Cited by 139
Scientist As Subject: The Psychological Imperative
Scipio, W. J. D. (1978) · American Journal of Psychiatry · doi:10.1176/ajp.135.1.142 · Cited by 132 · cited 1×
“Statistical analyses of research data are quite error prone [1,2,3], accounts of statistical results may be inaccurate [4], and decisions that researchers make during the analytical phase of a study may lean towards the goal of achieving a preferred (significant) result [5,6,7,8].”
BackgroundThe Historical Growth of Statistical Significance Testing in Psychology–and Its Future Prospects
Hubbard, R., & Ryan, P. A. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970808 · Cited by 118 · cited 1×
“This procedure is not without its critics [24,25], but remains to be used extensively in psychology [26] and related fields.”
BackgroundProbability as certainty: Dichotomous thinking and the misuse ofp values
Hoekstra, R., Finch, S. J., Kiers, H. A. L., & Johnson, A. (2006) · Psychonomic Bulletin & Review · doi:10.3758/bf03213921 · Cited by 112 · cited 1×
“05, are considered by many researchers [27,28] and reviewers [29] to lend support to the hypothesized effects.”
BackgroundSharing Detailed Research Data Is Associated with Increased Citation RatePreprint
Piwowar, H., & Piwowar, H. (2007) · Nature Precedings · doi:10.1038/npre.2007.361 · Cited by 105
Strasak, A. M., Zaman, Q., Marinell, G., Pfeiffer, K. P., & Ulmer, H. (2007) · The American Statistician · doi:10.1198/000313007x170242 · Cited by 103
Pitfalls in Human Research: Ten Pivotal Points.
Alexander, C. N., & Barber, T. X. (1978) · Contemporary Sociology · doi:10.2307/2065953 · Cited by 82 · cited 1×
“Statistical analyses of research data are quite error prone [1,2,3], accounts of statistical results may be inaccurate [4], and decisions that researchers make during the analytical phase of a study may lean towards the goal of achieving a preferred (significant) result [5,6,7,8].”
BackgroundHubbard, R., & Ryan, P. A. (2000) · Educational and Psychological Measurement · doi:10.1177/0013164400605001 · Cited by 64
Jackson, M., & Hill, P. J. (1995) · Journal of Theoretical Politics · doi:10.1177/0951692895007002004 · Cited by 63
Data Sharing in Medical Research: An Empirical Investigation
Reidpath, D. D., & Allotey, P. A. (2001) · Bioethics · doi:10.1111/1467-8519.00220 · Cited by 62 · cited 2×
“However, it is likely that similar problems play a role in the widespread reluctance to share data in other scientific fields [13,14,15,16,17,18,19,20].”
BackgroundInconsistencies between reported test statistics and p‐values in two psychiatry journals
Berle, D., & Starcevic, V. (2007) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.225 · Cited by 38
Private archives and public needs.
Ceci, S. J., & Walker, E. (1983) · American Psychologist · doi:10.1037/0003-066x.38.4.414 · Cited by 37 · cited 1×
“The unwillingness to share data of published research has been documented in a number of fields [13,14,15,16, 17,18,19,20] and is often ascribed in part to the fear among authors that independent reanalysis will expose statistical or analytical errors in their work [21] and will produce conclusions that differ from th…”
BackgroundPethokoukis, J. M. (2003) · PubMed · Cited by 21
Statistical errors in immunologic research
Murphy, J. R. (2004) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2004.09.023 · Cited by 19 · cited 1×
“Statistical analyses of research data are quite error prone [1,2,3], accounts of statistical results may be inaccurate [4], and decisions that researchers make during the analytical phase of a study may lean towards the goal of achieving a preferred (significant) result [5,6,7,8].”
BackgroundReplication Data Sets and Favored-Hypothesis Bias
Firebaugh, G. (2007) · Sociological Methods & Research · doi:10.1177/0049124107306663 · Cited by 19 · cited 3×
“Because existing guidelines on data sharing offer little promise for improvement [40], progress in psychological science and related fields would benefit from having research data itself be part of the process of replication [15,16], notably by the establishment by journals, professional organizations, and granting bo…”
BackgroundFriedlander, F. (1964) · American Psychologist · doi:10.1037/h0038977 · Cited by 12 · cited 2×
“Psychological research data are often amenable to alternative methods of analysis [6,22,30] that may affect what can be concluded from them (at least within the rules of NHST).”
BackgroundMethodSharing: guidelines go one step forwards, two steps back
Wicherts, J. M., & Bakker, M. (2009) · Nature · doi:10.1038/4611053c · Cited by 5 · cited 1×
“Because existing guidelines on data sharing offer little promise for improvement [40], progress in psychological science and related fields would benefit from having research data itself be part of the process of replication [15,16], notably by the establishment by journals, professional organizations, and granting bo…”
BackgroundSelf-citation(2006) · Nature · doi:10.1038/444653b · Cited by 5
Cited by
Witwer, K. W., Buzás, E. I., Bemis, L. T., Bora, A., Lässer, C., Lötvall, J., Nolte‐‘t Hoen, E. N., Piper, M. G., Sivaraman, S., Skog, J., et al. (2013) · Journal of Extracellular Vesicles · doi:10.3402/jev.v2i0.20360 · Cited by 2,421
“Database submission Data submission is mandated by most journals (95,96) and independent studies have found that unavailability of data is associated with low data or study quality (94, 97,98); however, compliance remains poor (94,99,100).”
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
“Even diligent reviewers miss lots of errors (Schroter et al., 2008; Wicherts et al., 2011).”
Increasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Self-citation
Metadata
- Authors
- Wicherts, J. M., Bakker, M., & Molenaar, D.
- Affiliations
- University of Amsterdam (NL)
- Corresponding author
- Wicherts, Jelte M.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-11-02
- Volume
- 6
- Issue
- 11
- Pages
- e26828
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3206853
- OpenAlex
- W2072620887
- Cited by
- 424
- References
- 50
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 18 · 2023: 26 · 2022: 23 · 2021: 54 · 2020: 34 · 2019: 27 · 2018: 37 · 2017: 33 · 2016: 40 · 2015: 41 · 2014: 33 · 2013: 27 · 2012: 15
- FWCI
- 69.64
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- Tilburg University Research Portal (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (451-07-016)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Sociology and Political Science › Data Analysis and Archiving
- Keywords
- research data sharing, data archiving, reanalysis, statistical reporting errors, statistical significance, quality of evidence, null hypothesis, psychology journals, data verification
- MeSH terms
- Cooperative Behavior, Humans, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.