Do Research Literatures Give Correct Answers?
Abstract
The rationale for insisting on properly designed studies is to ensure the construction of research literatures that are not influenced by biases of any sort. Once a literature is established, however, whatever answers it gives (usually via meta-analysis) are accepted as valid. The results of our favored statistical techniques are understood best when significant relationships are present while the meaning of null results are unclear. Thus, significant findings are more likely to be published. This file drawer problem has been known for many years. However, the effect that the file drawer has on research literatures has never been probed empirically. A new method is described that tests the impact a file drawer has on a research literature. The results of the file drawer on four separate psychological research literatures are presented.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The original studies may have inflated effect sizes due to publication, selection, reporting, or other biases.
“An important qualification about this result is the possibility that the original studies have inflated effect sizes due to publication, selection, reporting, or other biases ( 9, 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [17]
Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results.
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [17]
Concerns about reproducibility are widespread across disciplines.
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [17]
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Implementation intentions: Strong effects of simple plans.
Gollwitzer, P. M. (1999) · American Psychologist · doi:10.1037/0003-066x.54.7.493 · Cited by 5,478 · see citing works
Dissertation Abstracts International
Kutler, S. I. (2017) · The SHAFR Guide Online · doi:10.1163/2468-1733_shafr_sim010020122 · Cited by 4,190 · see citing works
The Essential Tension: Selected Studies in Scientific Tradition and Change
Kuhn, T. S. (1977) · doi:10.7208/chicago/9780226217239.001.0001 · Cited by 3,861 · see citing works
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657 · see citing works
Quine, W. V. O. (1980) · Harvard University Press eBooks · doi:10.2307/j.ctv1c5cx5c · Cited by 2,439 · see citing works
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
Music and spatial task performance
Rauscher, F. H., Shaw, G. L., & Ky, C. N. (1993) · Nature · doi:10.1038/365611a0 · Cited by 1,216 · see citing works
The Handbook of Research Synthesis
Hanka, R. (1994) · BMJ · doi:10.1136/bmj.309.6952.488a · Cited by 1,064 · see citing works
McGuire, J. E. (1962) · Philosophical Books · doi:10.1111/j.1468-0149.1962.tb01333.x · Cited by 336 · see citing works
Does psi exist? Replicable evidence for an anomalous process of information transfer.
Bem, D. J., & Honorton, C. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.4 · Cited by 330 · see citing works
Prelude or requiem for the ‘Mozart effect’?
Chabris, C. F. (1999) · Nature · doi:10.1038/23608 · Cited by 280 · see citing works
Listening to Music Enhances Spatial-Temporal Reasoning: Evidence for the "Mozart Effect"
Hetland, L. (2000) · Journal of Aesthetic Education · doi:10.2307/3333640 · Cited by 257 · see citing works
Toulmin, S. E., & Lindsay, R. B. (1964) · Physics Today · doi:10.1063/1.3051484 · Cited by 249 · see citing works
Evidence‐based sample size calculations based upon updated meta‐analysis
Sutton, A. J., Cooper, N. J., Jones, D. R., Lambert, P. C., Thompson, J. R., & Abrams, K. R. (2006) · Statistics in Medicine · doi:10.1002/sim.2704 · Cited by 147 · see citing works
Does psi exist? Lack of replication of an anomalous process of information transfer.
Milton, J., & Wiseman, R. J. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.4.387 · Cited by 144 · see citing works
A Proposal for a New Editorial Policy in the Social Sciences
Walster, G. W., & Cleary, T. A. (1970) · The American Statistician · doi:10.1080/00031305.1970.10478884 · Cited by 144 · see citing works
Does psi exist? Comments on Milton and Wiseman's (1999) meta-analysis of Ganzfield research.
Storm, L., & Ertel, S. (2001) · Psychological Bulletin · doi:10.1037/0033-2909.127.3.424 · Cited by 52 · see citing works
On Knowing What We Do Not Know
Kromrey, J. D., & Rendina‐Gobioff, G. (2006) · Educational and Psychological Measurement · doi:10.1177/0013164405278585 · Cited by 51 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Appelt, K. C., Milch, K. F., Handgraaf, M. J. J., & Weber, E. U. (2011) · Judgment and Decision Making · doi:10.1017/s1930297500001455 · Cited by 267 · see citing works
Braun, S. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00773 · Cited by 229 · see citing works
Dalton, D., Aguinis, H., Dalton, C. M., Bosco, F., & Pierce, C. (2012) · Personnel Psychology · doi:10.1111/j.1744-6570.2012.01243.x · Cited by 145 · see citing works
Estimating the Reproducibility of Psychological Science
Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R. (2016) · doi:10.31219/osf.io/447b3 · Cited by 88 · see citing works
Effectiveness of mission statements in organizations – A review
Braun, S. H., Wesche, J. S., Frey, D., Weisweiler, S., & Peus, C. V. (2012) · Journal of Management & Organization · doi:10.5172/jmo.2012.18.4.430 · Cited by 61 · see citing works
Effectiveness of Mission Statements in Organizations – A Review
Braun, S. H., Wesche, J. S., Frey, D., Weisweiler, S., & Peus, C. V. (2012) · Journal of Management & Organization · doi:10.1017/s1833367200000687 · Cited by 92 · see citing works
Lubke, G. H., & Campbell, I. (2016) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2016.1141355 · Cited by 36 · see citing works
What's Wrong with Research Literatures? And how to Make Them Right
Howard, G., Hill, T., Maxwell, S., Baptista, T., Farias, M., Coelho, C., Coulter-Kern, M., & Coulter-Kern, R. G. (2009) · Review of General Psychology · doi:10.1037/a0015319 · Cited by 24 · see citing works
Self-citationEditorial: Announcing preregistered reports.
Vonk, J. M., & Krause, M. A. (2018) · Animal Behavior and Cognition · doi:10.26451/abc.05.02.00.2018 · Cited by 20 · see citing works
Fernández‐Castilla, B., Said‐Metwaly, S., Kreitchmann, R. S., & Noortgate, W. V. D. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02373-9 · Cited by 14 · see citing works
Maag, J. W., & Losinski, M. L. (2015) · Advances in Social Sciences Research Journal · doi:10.14738/assrj.24.1044 · Cited by 7 · see citing works
Cromer, L. D. (2013) · Journal of the Scholarship of Teaching and Learning · Cited by 7 · see citing works
The perceived role of null results in school psychology research and publication
Iwachiw, J. S., Button, A. L., & Atlas, J. G. (2019) · School Psychology International · doi:10.1177/0143034319851230 · Cited by 6 · see citing works
Lartey, J. K. S., Almeida, S., & Paloyo, A. R. (2024) · Management Review Quarterly · doi:10.1007/s11301-024-00443-y · Cited by 5 · see citing works
A Prospective Study of Coping and Adjustment in Adolescents With Craniofacial Conditions
Roberts, R., & Shute, R. H. (2012) · Children s Health Care · doi:10.1080/02739615.2012.657031 · Cited by 5 · see citing works
Attractiveness and self-esteem: A test of sociometer theory
Bale, C. (2010) · CLOK (University of Central Lancashire) · Cited by 4 · see citing works
Friedman, H. L. (2010) · EXPLORE · doi:10.1016/j.explore.2010.03.003 · Cited by 3 · see citing works
Gayer, C. C. (2013) · UKnowledge (University of Kentucky) · Cited by 2 · see citing works
Effectiveness of Mission Statements in Organizations – A Review
Braun, S. H., Wesche, J. S., Frey, D., Weisweiler, S., & Peus, C. V. (2012) · Journal of Management & Organization · doi:10.5172/jmo.2012.1761 · Cited by 1 · see citing works
ORMA: A strategy to reduce Psychology's replication problems
Howard, G. S., & Maxwell, S. E. (2022) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2022.100991
Self-citation
- Authors
- Howard, G. S., Lau, M. Y., Maxwell, S. E., Venter, A., Lundy, R., & Sweeny, R. M.
- Affiliations
- University of Notre Dame (US)
- Corresponding author
- Howard, George S.
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 116-121
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2003439165
- Cited by
- 21
- References
- 21
- Citations by year
- 2024: 2 · 2022: 1 · 2019: 1 · 2018: 1 · 2017: 1 · 2016: 2 · 2015: 2 · 2013: 2 · 2012: 5
- FWCI
- 3.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Review of General Psychology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- file drawer problem, publication bias, research literature, meta-analysis, negative results, significance testing, publication rates, psychology research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
