PsychDisclosure.org: Grassroots Support for Reforming Reporting Standards in Psychology
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
There is currently an unprecedented level of doubt regarding the reliability of research findings in psychology. Many recommendations have been made to improve the current situation. In this article, we report results from PsychDisclosure.org , a novel open-science initiative that provides a platform for authors of recently published articles to disclose four methodological design specification details that are not required to be disclosed under current reporting standards but that are critical for accurate interpretation and evaluation of reported findings. Grassroots sentiment—as manifested in the positive and appreciative response to our initiative—indicates that psychologists want to see changes made at the systemic level regarding disclosure of such methodological details. Almost 50% of contacted researchers disclosed the requested design specifications for the four methodological categories (excluded subjects, nonreported conditions and measures, and sample size determination). Disclosed information provided by participating authors also revealed several instances of questionable editorial practices, which need to be thoroughly examined and redressed. On the basis of these results, we argue that the time is now for mandatory methods disclosure statements for all psychology journals, which would be an important step forward in improving the reliability of findings in psychology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 4×
“There is currently an unprecedented level of doubt regarding the reliability of research findings reported in psychology (Pashler & Wagenmakers, 2012) and neuroscience (Button et al., 2013) journals, which has”
InfluentialSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Cognitive consequences of forced compliance.
Festinger, L., & Carlsmith, J. M. (1959) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0041593 · Cited by 2,945 · cited 1×
“To reach back across the decades, one paper that was apparently published under norms of full disclosure was Festinger and Carlsmith’s (1959) classic induced-compliance experiment in support of cognitive dissonance theory.”
Platt, J. R. (1964) · Science · doi:10.1126/science.146.3642.347 · Cited by 2,907 · cited 1×
“The latter goal can be achieved by using methodologically sound manipulations and measures (LeBel & Peters, 2011) and stronger forms of hypothesis testing (Fiedler, Kutzner, & Krueger, 2012; Meehl, 1967, 1978; Platt, 1964).”
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Self-citationTheory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012) · doi:10.2139/ssrn.2160588 · Cited by 680
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · cited 2×
“Recently, there has been a growing concern regarding the non-replicability of research findings in psychology (Ioannidis, 2012; Pashler & Harris, 2012) and neuroscience (Button et al.”
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · cited 2×
“Because of the highly competitive nature of peerreviewed publishing, esthetics of data perfection often seem to prevail to the detriment of an overall view of the evidence (Giner-Sorolla, 2012).”
Self-citationAnnouncement: Reducing our irreproducibility
(2013) · Nature · doi:10.1038/496398a · Cited by 243
Cited by
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716
“All articles make explicit all conditions, measures, data exclusions, and how samples sizes were determined (LeBel et al., 2013; Simmons, Nelson, & Simonsohn, 2012).”
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
The Ordinal Effects of Ostracism: A Meta-Analysis of 120 Cyberball Studies
Hartgerink, C. H. J., van Beest, I., Wicherts, J. M., & Williams, K. D. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127002 · Cited by 558
Pre-registration in social psychology—A discussion and suggested template
van 't Veer, A. E., & Giner-Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 439
Self-citation
Metadata
- Authors
- LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2013-07
- Volume
- 8
- Issue
- 4
- Pages
- 424-432
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2141646831
- Cited by
- 103
- References
- 16
- FWCI
- 16.19
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.