What's Wrong with Research Literatures? And how to Make Them Right
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Meta-analysis is now the accepted procedure for summarizing research literatures in areas of applied psychology. Because of the bias for publishing statistically significant findings, while usually rejecting nonsignificant results, our research literatures yield misleading answers to important quantitative questions (e.g., How much better is the average psychotherapy patient relative to a comparable group of untreated controls? How much more aggressive are children who watch a great deal of violent TV than children who watch little or no violence on TV?). While all such research literatures provide overly optimistic meta-analytic estimates, exactly how practically important are these overestimates? Three studies testing the literature on implementation intentions finds only slightly elevated effectiveness estimates. Conversely, in three studies another growing research literature (the efficacy of remote intercessory prayer) is found to be misleading and is in all likelihood not a real effect (i.e., our three studies suggest the literature likely consists of Type I errors). Rules of thumb to predict which research literatures are likely invalid are offered. Finally, revised publication and data analysis procedures to generate unbiased research literatures in the future are examined.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Meta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · cited 1×
“Thus, for example, Smith and Glass (1977) conclude that, “the typical therapy client is better off than 75% of untreated individuals” (p.”
BackgroundImplementation intentions and effective goal pursuit.
Gollwitzer, P. M., & Brandstätter, V. (1997) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.73.1.186 · Cited by 1,343
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“313) Gigerenzer’s (1993) summary reflects an enormous literature (e.g., Atkinson, Furlong, & Wampold, 1982; Chase & Chase, 1976; Greenwald, 1975; Kerr, Tolliver, & Petree, 1977; Mahoney, 1977; Sterling, 1959) that demonstrates editors’ and reviewers’ preference for significant results (and their bias against nonsignif…”
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013 · cited 1×
“313) Gigerenzer’s (1993) summary reflects an enormous literature (e.g., Atkinson, Furlong, & Wampold, 1982; Chase & Chase, 1976; Greenwald, 1975; Kerr, Tolliver, & Petree, 1977; Mahoney, 1977; Sterling, 1959) that demonstrates editors’ and reviewers’ preference for significant results (and their bias against nonsignif…”
Estimation of a Single Effect Size: Parametric and Nonparametric Methods
Hedges, L. V., & Olkin, I. (1985) · Statistical Methods for Meta-Analysis · doi:10.1016/b978-0-08-057065-5.50010-5 · Cited by 983
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 2×
“313) Gigerenzer’s (1993) summary reflects an enormous literature (e.g., Atkinson, Furlong, & Wampold, 1982; Chase & Chase, 1976; Greenwald, 1975; Kerr, Tolliver, & Petree, 1977; Mahoney, 1977; Sterling, 1959) that demonstrates editors’ and reviewers’ preference for significant results (and their bias against nonsignif…”
BackgroundDoes psi exist? Replicable evidence for an anomalous process of information transfer.
Bem, D. J., & Honorton, C. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.4 · Cited by 330
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.23 · Cited by 141
Cited by
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
“Yet, a lack of null and negative results has been noticed in innumerable fields (Song et al. 2010; Gerber and Malhotra 2008; Howard et al. 2009; Dwan et al. 2008; Jennions and Moller 2002).”
BackgroundKühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399
A Meta-Analytic Review of Stand-Alone Interventions to Improve Body Image
Alleva, J. M., Sheeran, P., Webb, T. L., Martijn, C., & Miles, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0139177 · Cited by 319
Researcher Effects on Mortality Salience Research: A Meta-Analytic Moderator Analysis
Yen, C.-L., & Cheng, C.-P. (2013) · Death Studies · doi:10.1080/07481187.2012.682290 · Cited by 30
Lukas, C. A., Trevisi Fuentes, H., & Berking, M. (2019) · Internet Interventions · doi:10.1016/j.invent.2019.100250 · Cited by 27
Metadata
- Authors
- Howard, G. S., Hill, T. L., Maxwell, S. E., Baptista, T. M., Farias, M. H., Coelho, C., Coulter-Kern, M., & Coulter-Kern, R.
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 146-166
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W1986173452
- Cited by
- 22
- References
- 11
- FWCI
- 2.79
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.