Consequences of prejudice against the null hypothesis.
Abstract
The consequences of prejudice against accepting the null hypothesis were examined through (a) a mathematical model intended to stimulate the research-publication process and (b) case studies of apparent erroneous rejec-tions of the null hypothesis in published psychological research. The input parameters for the model characterize investigators ' probabilities of selecting a problem for which the null hypothesis is true, of reporting, following up on, or abandoning research when data do or do not reject the null hypothesis, and they characterize editors ' probabilities of publishing manuscripts concluding in favor of or against the null hypothesis. With estimates of the input parameters based on a questionnaire survey of a sample of social psychologists, the model output indicates a dysfunctional research-publication system. Particularly, the model indicates that there may be relatively few publications on problems for which the null hypothesis is (at least to a reasonable approximation) true, and of these, a high proportion will erroneously reject the null hypothesis. The case studies provide additional support for this conclusion. Accordingly, it is
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them.
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [11]
Positive results are more likely to be published than negative results.
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [11]
Theoretical analyses point to a system of incentives that prioritizes innovation over verification, leading to infrequent replication and behaviors that could reduce replicability.
“Theoretical analyses point to a system of incentives that prioritizes innovation over verification, leading to infrequent efforts to replicate findings and to behaviors that could reduce the replicability of published findings such as selective reporting, presenting exploratory findings as confirmatory tests, and failures of documentation, transparency, and sharing”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Greenwald, 1975
Research producing novel, statistically significant, or seemingly clean results is more likely to be published.
“Research that produces novel results, statistically significant results (that is, typically p < 0.05) and seemingly 'clean' results is more likely to be published 2,3 .”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [3]
The original studies may have inflated effect sizes due to publication, selection, reporting, or other biases.
“An important qualification about this result is the possibility that the original studies have inflated effect sizes due to publication, selection, reporting, or other biases ( 9, 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [16]
Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results.
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [16]
Low power of research designs combined with publication bias favoring positive results over negative results produces a literature with upwardly biased effect sizes.
“There is good evidence that low power of research designs combined with publication bias favoring positive results over negative results produces a literature with upwardly biased effect sizes ( 14, 16, 30, 31 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [16]
Concerns about reproducibility are widespread across disciplines.
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [16]
Cites
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,281 · see citing works
The Influence of Source Credibility on Communication Effectiveness
Hovland, C. I., & Weiss, W. (1951) · Public Opinion Quarterly · doi:10.1086/266350 · Cited by 3,543 · see citing works
Boneau, C. A., & Hays, W. L. (1966) · The American Journal of Psychology · doi:10.2307/1420739 · Cited by 1,897 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · see citing works
Experimental and quasi-experimental designs for research on teaching
Campbell, D. T. (1963) · Medical Entomology and Zoology · Cited by 1,524 · see citing works
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological bulletin · doi:10.1037/h0026141 · Cited by 1,363 · see citing works
Artifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · see citing works
The test of significance in psychological research.
Bakan, D. (1966) · Psychological Bulletin · doi:10.1037/h0020412 · Cited by 957 · see citing works
Applications of Information Theory to Psychology
Garner, W. R. (1959) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-195911000-00013 · Cited by 756 · see citing works
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 738 · see citing works
Research Methods in the Behavioral Sciences
Blankertz, D. F., Festinger, L., & Katz, D. M. (1954) · Journal of Marketing · doi:10.2307/1247742 · Cited by 713 · see citing works
Experiments on Mass Communication
Hovland, C. I., & Lumsdaine, A. A. (2017) · Princeton University Press eBooks · doi:10.1515/9781400886807 · Cited by 597 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524 · see citing works
Experimentation in social psychology.
Aronson, E., Wilson, T. D., & Brewer, M. B. (1998) · Cited by 519 · see citing works
The yin and yang of progress in social psychology: Seven koan.
McGuire, W. J. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0034345 · Cited by 490 · see citing works
"Reinstatement" of the communicator in delayed measurement of opinion change.
Kelman, H. C., & Hovland, C. I. (1953) · Journal of Abnormal & Social Psychology · doi:10.1037/h0061861 · Cited by 430 · see citing works
Experiments on mass communication.
indicated, N. A. (1949) · Journal of Consulting Psychology · doi:10.1037/h0049583 · Cited by 381 · see citing works
The learning and forgetting of controversial material.
Levine, J. M., & Murphy, G. (1943) · Journal of Abnormal & Social Psychology · doi:10.1037/h0062586 · Cited by 198 · see citing works
Testing the null hypothesis and the strategy and tactics of investigating theoretical models.
Grant, D. A. (1962) · Psychological Review · doi:10.1037/h0038813 · Cited by 169 · see citing works
A Proposal for a New Editorial Policy in the Social Sciences
Walster, G. W., & Cleary, T. A. (1970) · The American Statistician · doi:10.1080/00031305.1970.10478884 · Cited by 144 · see citing works
Data-Dredging Procedures in Survey Analysis
Selvin, H., & Stuart, A. (1966) · The American Statistician · doi:10.1080/00031305.1966.10480401 · Cited by 132 · see citing works
Bayesian Statistics for Social Scientists.
Hughes, M., & Phillips, L. D. (1975) · Operational Research Quarterly (1970-1977) · doi:10.2307/3007832 · Cited by 130 · see citing works
Persistence of induced opinion change and retention of the inducing message contents.
Watts, W. A., & McGuire, W. J. (1964) · Journal of Abnormal & Social Psychology · doi:10.1037/h0041081 · Cited by 129 · see citing works
Is it time to lay the sleeper effect to rest?
Gillig, P. M., & Greenwald, A. G. (1974) · Journal of Personality and Social Psychology · doi:10.1037/h0035744 · Cited by 99 · see citing works
Self-citationData-dredging procedures in survey analysis
Selvin, H. C., & Stuart, A. (2017) · Routledge eBooks · doi:10.4324/9781315128498-21 · Cited by 99 · see citing works
Attitude and selective learning: Where are the phenomena of yesteryear?
Greenwald, A. G., & Sakumura, J. S. (1967) · Journal of Personality and Social Psychology · doi:10.1037/h0025229 · Cited by 76 · see citing works
Self-citationBinder, A. (1963) · Psychological Review · doi:10.1037/h0039657 · Cited by 75 · see citing works
Experimental Social Psychology.
(1932) · Archives of Neurology And Psychiatry · doi:10.1001/archneurpsyc.1932.02230180241023 · Cited by 66 · see citing works
Belief, evaluation, and performance on an ESP task
Layton, B. D., & Turnbull, B. (1975) · Journal of Experimental Social Psychology · doi:10.1016/s0022-1031(75)80019-9 · Cited by 66 · see citing works
Cognitive versus affective factors in the learning and retention of controversial material.
Fitzgerald, D., & Ausbel, D. P. (1963) · Journal of Educational Psychology · doi:10.1037/h0045404 · Cited by 64 · see citing works
Salience Patterns, Source Credibility, and the Sleeper Effect
Schulman, G. I., & Worrall, C. (1970) · Public Opinion Quarterly · doi:10.1086/267813 · Cited by 64 · see citing works
Bloembergen, N. (1948) · Springer eBooks · doi:10.1007/978-94-017-6073-7_3 · Cited by 57 · see citing works
Tactical note on the relation between scientific and statistical hypotheses.
Edwards, W. (1965) · Psychological Bulletin · doi:10.1037/h0021967 · Cited by 52 · see citing works
Bayesian Statistical Inference for Psychological Research
Edwards, W., Lindman, H. R., & Savage, L. J. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-0919-5_34 · Cited by 52 · see citing works
Attitude as a determinant of learning and memory: A failure to confirm.
Waly, P., & Cook, S. W. (1966) · Journal of Personality and Social Psychology · doi:10.1037/h0023648 · Cited by 41 · see citing works
STATISTICS FOR SOCIAL SCIENTISTS
Immer, F. R. (1935) · Journal of Heredity · doi:10.1093/oxfordjournals.jhered.a104009 · Cited by 30 · see citing works
The influence of attitude on the recall of controversial material: A failure to confirm
Brigham, J. C., & Cook, S. W. (1969) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(69)90050-x · Cited by 27 · see citing works
A note on the inconclusiveness of accepting the null hypothesis.
Wilson, W. R., & Miller, H. L. (1964) · Psychological Review · doi:10.1037/h0046217 · Cited by 26 · see citing works
Effects of attitude on learning and memory: The influence of instruction-induced sets
Malpass, R. S. (1969) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(69)90036-5 · Cited by 25 · see citing works
A Proposal for a New Editorial Policy in the Social Sciences
Walster, G. W., & Cleary, T. A. (1970) · The American Statistician · doi:10.2307/2681924 · Cited by 16 · see citing works
Significance, nonsignificance, and interpretation of an ESP experiment
Greenwald, A. G. (1975) · Journal of Experimental Social Psychology · doi:10.1016/s0022-1031(75)80020-5 · Cited by 14 · see citing works
Self-citationQuasi-sensory communication: An investigation using semantic matching and accentuated affect.
McBain, W. N., Fox, W., Kimura, S., Nakanishi, M., & Tirado, J. (1970) · Journal of Personality and Social Psychology · doi:10.1037/h0028998 · Cited by 14 · see citing works
8 Prospective: Artifact and Control
Campbell, D. T. (2009) · doi:10.1093/acprof:oso/9780195385540.003.0008 · Cited by 14 · see citing works
Quasi-sensory communication: Still not proved.
Clement, D. E. (1972) · Journal of Personality and Social Psychology · doi:10.1037/h0032888 · Cited by 3 · see citing works
Sakurai, S. (2017) · CRC Press eBooks · doi:10.1201/9781315375168-13 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Efficacy of the Theory of Planned Behaviour: A meta‐analytic review
Armitage, C. J., & Conner, M. (2001) · British Journal of Social Psychology · doi:10.1348/014466601164939 · Cited by 9,970 · see citing works
“Greenwald, 1975; Rosenthal, 1984; White, 1982), although ndings are inconsistent. Related to this, the ‘ le drawer problem’ (Rosenthal, 1979) refers to the possibility that all published articles are the result of Type I errors, whereas all non-published (i.e. le drawer) studies are the remaining 95%. Reliability…”
MethodEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
“bias favoring positive results together produce a literature with upwardly biased effect sizes (14, 16, 33, 34).”
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
“Meehl (1990a) attributed to the distinguished philosopher Morris Raphael Cohen the saying "All logic texts are divided into two parts.”
BackgroundMethodRichardson, M., Abraham, C., & Bond, R. (2012) · Psychological Bulletin · doi:10.1037/a0026838 · Cited by 3,646 · see citing works
“Publication of statistically significant results is more probable (e.g., Greenwald, 1975), increasing the likelihood of Type I errors and an overestimation of the mean effect size in meta-analysis.”
BackgroundA manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · see citing works
“Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47,49,50 .”
Research design in clinical psychology
Kazdin, A. E. (1952) · Journal of Clinical Psychology · doi:10.1002/1097-4679(195201)8:1 · Cited by 2,153 · see citing works
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
“2McGuire (1983a) and Greenwald, Pratkanis, Leippe, and Baumgardner (1986) describe another means to the same end.”
BackgroundLipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · see citing works
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
“Note that—as in Experiment 1—this conclusion relies on interpreting a null effect as meaningful, which is possible when samples are adequately powered (Greenwald, 1975).”
BackgroundInfluentialOne Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558 · see citing works
“Journal referees may block publication of nonsignificant research findings (Greenwald, 1975).”
BackgroundPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
“Publication bias has been documented in various disciplines within the biomedical (3, 4, 5, 6, 7, 8, 9) and social sciences (10, 11, 12, 13, 14, 15, 16, 17).”
BackgroundThe preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Employee Turnover: A Meta-Analysis and Review with Implications for Research
Cotton, J. L., & Tuttle, J. M. (1986) · Academy of Management Review · doi:10.5465/amr.1986.4282625 · Cited by 1,493 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
“In general, one can distinguish between these two alternative accounts of a null finding by considering a different null hypothesis: not that the effect is zero, but that it is very small instead (see e.g., Cohen, 1988, pp. 16-17; Greenwald, 1975; Hodges & Lehmann, 1954; Serlin & Lapsley, 1985).”
Eagly, A. H., Ashmore, R. D., Makhijani, M. G., & Longo, L. C. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.110.1.109 · Cited by 1,875 · see citing works
“An additional consideration in evaluating meta-analytic methods is the extent to which the magnitude of the physical attractiveness stereotype may have been exaggerated in our review because of a publication bias in favor of significant findings (Greenwald, 1975; Lane & Dunlap, 1978).”
ResultNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
“In summary, the demands for novelty and positive results create incentives for (a) generating new ideas rather than pursuing additional evidence for or against ideas suggested previously; (b) reporting positive results and ignoring negative results (Fanelli, 2012; Greenwald, 1975; Ioannidis & Trikalinos, 2007; Rosenth…”
InfluentialThe Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · see citing works
Meta-Analysis in Clinical Research
L'ABBÉ, K. A., DETSKY, A. S., & O'ROURKE, K. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-107-2-224 · Cited by 1,267 · see citing works
The totalitarian ego: Fabrication and revision of personal history.
Greenwald, A. G. (1980) · American Psychologist · doi:10.1037/0003-066x.35.7.603 · Cited by 1,942 · see citing works
“…that the predisposition to confirm existing theoretical beliefs is pervasively characteristic of the research behavior of scientists, and I ( Greenwald, 1975a) have reported data showing that psychologists are strongly inclined to disregard research results inconsistent with their…”
BackgroundSelf-citationThe relative efficacy of avoidant and nonavoidant coping strategies: A meta-analysis.
Suls, J. M., & Fletcher, B. (1985) · Health Psychology · doi:10.1037//0278-6133.4.3.249 · Cited by 1,224 · see citing works
Field, A. P., & Gillett, R. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x502733 · Cited by 1,162 · see citing works
“This bias is not trivial: significant findings are estimated to be eight times more likely to be submitted than non-significant ones (Greenwald, 1975), studies with positive findings are around seven times more likely to be published than studies with results supporting the null hypothesis (Coursol…”
ResultShall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · see citing works
“…1993) 54% of the reviewers indicated they would prefer a study with new findings over a replication study.6 When asked what the problems with publishing replications were, comments such as “Not newsworthy” – “Waste of space” “. . . waste of resources” were given (see also Greenwald, 1978).”
BackgroundThe group polarization phenomenon.
Myers, D. G., & Lamm, H. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.4.602 · Cited by 1,102 · see citing works
“A further difficulty when trying to document the generality of a phenomenon is the likelihood that negative results have more often gone unreported than have rejections of the null hypothesis (Greenwald, 1975).”
BackgroundAttitudes and the Prediction of Behavior: A Meta-Analysis of the Empirical Literature
Kraus, S. J. (1995) · Personality and Social Psychology Bulletin · doi:10.1177/0146167295211007 · Cited by 1,613 · see citing works
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
“Consider the context, in which positive results are perceived as more publishable than negative results (Greenwald, 1975) and common analytic practices may inadvertently increase the likelihood of obtaining false positives (Simmons et al., 2011).”
BackgroundInfluentialIs psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,018 · see citing works
“Cook, Gruder, Henningan, and Faly (1979), Greenwald (1975), and Wilson and Shadish (2006) all offer valuable perspectives on broader issues involved in evaluating the veracity of null effects.”
BackgroundIntrinsic and extrinsic religiousness: Review and meta-analysis.
Donahue, M. J. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.48.2.400 · Cited by 948 · see citing works
Behavioral Development and Construct Validity: The Principle of Aggregation
Rushton, J. P., Brainerd, C. J., & Pressley, M. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.94.1.18 · Cited by 940 · see citing works
“The nongeneralizability of experimental findings to similar situations and, sometimes, their nonreplicability have been sources of concern (Greenwald, 1975).”
BackgroundHypothesis-testing processes in social interaction.
Snyder, M., & Swann, W. B. (1978) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.36.11.1202 · Cited by 937 · see citing works
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. V. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 936 · see citing works
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910 · see citing works
Gender and aggressive behavior: a meta-analytic review of the social psychological literature.
Eagly, A. H., & Steffen, V. J. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.100.3.309 · Cited by 872 · see citing works
Gender and the emergence of leaders: A meta-analysis.
Eagly, A. H., & Karau, S. J. (1991) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.60.5.685 · Cited by 832 · see citing works
Client Importance, Nonaudit Services, and Abnormal Accruals
Chung, H. H., & Kallapur, S. (2003) · The Accounting Review · doi:10.2308/accr.2003.78.4.931 · Cited by 824 · see citing works
One hundred years of forgetting: A quantitative description of retention.
Rubin, D. C., & Wenzel, A. E. (1996) · Psychological Review · doi:10.1037/0033-295x.103.4.734 · Cited by 820 · see citing works
“It has also encouraged researchers to ignore the important cases where there are no differences when conditions are changed (Greenwald, 1975;Rubin, 1989).”
BackgroundSmall Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742 · see citing works
“…than zero, and concluding that it is (basically) zero if we obtain a result that is statistically significantly smaller than that small effect (see, e.g., Cohen, 1988, pp. 16–17; Greenwald, 1975, pp. 16–19; Hodges & Lehmann, 1954; Kraemer, 1983; Rindskopf, 1997; Serlin & Lapsley, 1985, 1993).”
Scientific Guidelines for Conducting Integrative Research Reviews
Cooper, H. M. (1982) · Review of Educational Research · doi:10.3102/00346543052002291 · Cited by 785 · see citing works
“For instance, Greenwald (1975) found that about half of researchers who produced a rejection of the null hypothesis would submit a report for publication, while only 6 percent who failed to reject the null hypothesis would attempt to publish.”
BackgroundReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Fang, Y., Qureshi, I., Sun, H., McCole, P., Ramsey, E., & Lim, K. H. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.2.04 · Cited by 770 · see citing works
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
“Despite their theoretical and empirical value, the existing scientific culture provides few incentives for researchers to conduct replications or report negative results (Greenwald, 1975; Koole & Lakens, 2012).”
BackgroundÆgisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., et al. (2006) · The Counseling Psychologist · doi:10.1177/0011000005285875 · Cited by 737 · see citing works
“The direction of publication bias is that studies reporting significant results are more often published than studies presenting nonsignificant results (e.g., Greenwald, 1975; Rosnow & Rosenthal, 1989).”
ResultThe stability of behavior: II. Implications for psychological research.
Epstein, S. J. (1980) · American Psychologist · doi:10.1037/0003-066x.35.9.790 · Cited by 744 · see citing works
Publication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715 · see citing works
Ma, X., & Kishor, N. (1997) · Journal for Research in Mathematics Education · doi:10.2307/749662 · Cited by 731 · see citing works
“The reliability of a meta-analysis is sometimes a concern because of the suspicion that some researchers do not report statistically insignificant findings (Greenwald, 1975).”
BackgroundMethodNull hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090 · see citing works
“The replacement or complementation of NHST with the use of likelihood ratios and the estimation of posterior probabilities by application of Bayes's rule has been proposed by many writers (Edwards, 1965; Edwards et al., 1963; Gelman, Carlin, Stern, & Rubin, 1995; I. J. Good, 1981; Greenwald, 1975; Lindley, 1984; Rinds…”
MethodThe Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner‐Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & Veer, A. V. '. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 703 · see citing works
“Because effect sizes in the published literature are likely to be overestimates of the true effect size (Greenwald, 1975), researchers should err conservatively, toward higher levels of power.”
BackgroundPeters, G.-J. Y., Ruiter, R. A. C., & Kok, G. (2013) · Health Psychology Review · doi:10.1080/17437199.2012.703527 · Cited by 679 · see citing works
“Similarly, the significant outcome bias (SOB) has already caused most researchers to no longer even submit non-significant findings (Greenwald, 1975).”
BackgroundInfluentialEffects of Systematic Formative Evaluation: A Meta-Analysis
Fuchs, L. S., & Fuchs, D. (1986) · Exceptional Children · doi:10.1177/001440298605300301 · Cited by 669 · see citing works
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · see citing works
“In addition to publication bias at the journal level, publication bias also likely occurs at the individual scholar level, insofar as scholars decline to submit null results for publication either because such results conflict with beloved theoretical models in which they are invested or simply because they believe nu…”
BackgroundAn Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 605 · see citing works
“A high reproducibility estimate might boost confidence in conventional research and peerreview practices in the face of criticisms about inappropriate flexibility in design, analysis, and reporting that can inflate the rate of false positives (Greenwald, 1975; John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simo…”
BackgroundNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · see citing works
“This graph of sample size versus effectsize should be symmetrical in the absence of bias, but will tend to be skewed in its presence (Light and Pillemer (1984); see Figs 1 and 2 and further discussion in Section 5.”
BackgroundInfluentialNair, A. (2005) · Journal of Operations Management · doi:10.1016/j.jom.2005.11.005 · Cited by 598 · see citing works
Meta-analysis of comparative therapy outcome studies: a replication and refinement.
Shapiro, D. A., & Shapiro, D. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.92.3.581 · Cited by 590 · see citing works
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588 · see citing works
“For example, if researchers strongly believe that an effect is real and positive, statistically non-significant or negative estimates of that effect may never be submitted for publication or may be rejected by reviewers and editors (Greenwald, 1975; Sterling, Rosenbaum, & Weinkam, 1995; Rothstein et al., 2006; Ferguso…”
BackgroundLocus of control and academic achievement: A literature review.
Findley, M. J., & Cooper, H. M. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.44.2.419 · Cited by 568 · see citing works
Meta‐analysis of digit ratio 2D:4D shows greater sex difference in the right hand
Hönekopp, J., & Watson, S. J. (2010) · American Journal of Human Biology · doi:10.1002/ajhb.21054 · Cited by 566 · see citing works
“On a more general level we hope that we have made a case for considering the magnitude of effects in 2D:4D research; merely focusing on the statistical significance of effects may stall progress in 2D:4D research and in other areas [e.g., (Cohen, 1994; Greenwald, 1975; Hunter, 1997; Meehl, 1978)].”
BackgroundSample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542 · see citing works
How to Estimate and Interpret Various Effect Sizes.
Vacha‐Haase, T., & Thompson, B. (2004) · Journal of Counseling Psychology · doi:10.1037/0022-0167.51.4.473 · Cited by 538 · see citing works
“, nonsignificant results are filed and not even submitted for publication) and would perish (Greenwald, 1975; Rosenthal, 1979).”
BackgroundUnder what conditions does theory obstruct research progress?
Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.216 · Cited by 574 · see citing works
“…of research, greater rewards of research support, career advancement, and public recognition tend to fall to the researcher who investigates and confirms a new prediction than to the one who pursues a more familiar idea (see discussions by Armstrong, 1982; Fishman & Neigher, 1982; Greenwald, 1975).”
BackgroundInfluentialSelf-citationEagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 525 · see citing works
Psychological sources of ambiguity avoidance
Curley, S. P., Yates, J. F., & Abrams, R. A. (1986) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(86)90018-x · Cited by 523 · see citing works
“This situation, based upon an example first described by Keynes (1921), and again by Ellsberg (1961), demonstrates that two distinct types of uncertainty can be differentiated.”
MethodResultSocial facilitation: A meta-analysis of 241 studies.
Bond, C. F., & Titus, L. J. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.94.2.265 · Cited by 684 · see citing works
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461 · see citing works
“…and failure to publish studies with inadequate sample sizes (Newcombe, 1987), development of a more positive attitude toward nonsignificant results (Greenwald, 1975; Rennie & Flanagin, 1992), and improvement of the peer-review and publication processes (Higginson, 1987; Iyengar & Greenhouse, 1988;…”
BackgroundInfluentialDetecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
Beware of samples! A cognitive-ecological sampling approach to judgment biases.
Fiedler, K. (2000) · Psychological Review · doi:10.1037/0033-295x.107.4.659 · Cited by 460 · see citing works
On Average Deviation Indices for Estimating Interrater Agreement
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999) · Organizational Research Methods · doi:10.1177/109442819921004 · Cited by 462 · see citing works
“Computing the proportional agreement scale values as the average of the respective item values is consistent with how the James et al. (1984) r wg(J) indices are computed.”
MethodResultInfluentialThe actor-observer asymmetry in attribution: A (surprising) meta-analysis.
Malle, B. F. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.6.895 · Cited by 453 · see citing works
“It is commonly assumed that studies that contradict an established hypothesis or support it only weakly and without significance are less likely to be published than studies that support the established hypothesis (Begg & Berlin, 1988; Coursol & Wagner, 1986; Greenwald, 1975; Hedges, 1984).”
ResultDevelopment of self-esteem from age 4 to 94 years: A meta-analysis of longitudinal studies.
Orth, U., Erol, R. Y., & Luciano, E. C. (2018) · Psychological Bulletin · doi:10.1037/bul0000161 · Cited by 438 · see citing works
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597 · see citing works
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399 · see citing works
“Publication bias can occur at any stage of the publication process where a decision is made [18,19]: in the researcher’s decision to write up a manuscript [20]; in the decision to submit the manuscript to a journal [21,22]); in the decision of journal editors to send a paper out for review [23]; in the reviewer’s reco…”
BackgroundA general linear model for estimating effect size in the presence of publication bias
Vevea, J. L., & Hedges, L. (1995) · Psychometrika · doi:10.1007/bf02294384 · Cited by 367 · see citing works
Replications and Extensions in Marketing – Rarely Published But Quite Contrary
Hubbard, R., & Armstrong, J. (1994) · International Journal of Research in Marketing · doi:10.1016/0167-8116(94)90003-5 · Cited by 444 · see citing works
“…are biased in favor of the publication of statistically significant (p < 0.05) results, thus contaminating their respective literatures with Type I errors whose incidence exceeds that prescribed by nominal alpha levels such as 0.05 (Feige, 1975; Greenwald, 1975; Hubbard and Armstrong, 1992).”
BackgroundInfluential20 Stage Analysis of Reaction Processes
Sanders, A. F. (1980) · Advances in psychology · doi:10.1016/s0166-4115(08)61955-x · Cited by 407 · see citing works
Publication Bias in the Organizational Sciences
Kepes, S., Banks, G. C., McDaniel, M. A., & Whetzel, D. L. (2012) · Organizational Research Methods · doi:10.1177/1094428112452760 · Cited by 400 · see citing works
“As explained previously, published samples tend to report more significant and positive effect sizes than the literature as a whole (e.g., Greenwald, 1975; Hopewell, McDonald, Clarke, & Egger, 2007; Song et al., 2010).”
BackgroundSwedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works
“…such as selective reporting, presenting exploratory findings as confirmatory tests, and failures of documentation, transparency, and sharing (Casadevall and Fang, 2012; Gelman and Loken, 2013; Greenwald, 1975; Kimmelman et al., 2014; Makel et al., 2012; Nosek et al., 2012; Rosenthal, 1979).”
BackgroundLivingstone, S., & Helsper, E. J. (2006) · Journal of Communication · doi:10.1111/j.1460-2466.2006.00301.x · Cited by 393 · see citing works
“Although it can be suggested that journal publication is biased towards studies that show effects, Greenwald (1975) argues that this does not invalidate the conclusions drawn.”
BackgroundPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Bayesian Estimation and Inference
Zyphur, M. J., & Oswald, F. L. (2013) · Journal of Management · doi:10.1177/0149206313501200 · Cited by 392 · see citing works
“In any case, because of the focus on rejecting nulls with frequentist methods, theorizing null effects is almost never done, even though hypothesizing nulls can be useful (Greenwald, 1975; Kruschke, 2011; Schwab et al., 2011).”
BackgroundEffect sizes, confidence intervals, and confidence intervals for effect sizes
Thompson, B. (2007) · Psychology in the Schools · doi:10.1002/pits.20234 · Cited by 392 · see citing works
Meta-analysis of facial identification studies.
Shapiro, P. N., & Penrod, S. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.100.2.139 · Cited by 547 · see citing works
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M., & Lakens, D. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 359 · see citing works
“These include editors and reviewers selectively rejecting manuscripts with negative results (reviewer bias; Greenwald, 1975; Mahoney, 1977) and researchers deciding not to submit studies with negative results for publication (file-drawering; Rosenthal, 1979).”
BackgroundPeterson, R. S., & Behfar, K. J. (2003) · Organizational Behavior and Human Decision Processes · doi:10.1016/s0749-5978(03)00090-6 · Cited by 363 · see citing works
“Greenwald (1975) recommends testing a null range—rather than a null point to demonstrate why a .05 significance test warrants acceptance of the null hypothesis.”
BackgroundInfluentialScientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar‐Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 366 · see citing works
“, nonsignificant) results are much more likely to end up in the file drawer (Greenwald, 1975).”
BackgroundEffects of psychoeducational care for adult surgical patients: A meta-analysis of 191 studies
Devine, E. C. (1992) · Patient Education and Counseling · doi:10.1016/0738-3991(92)90193-m · Cited by 360 · see citing works
Skitka, L. J., Mullen, E., Griffin, T., Hutchinson, S. R., & Chamberlin, B. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.83.2.470 · Cited by 360 · see citing works
“…results of Study 2 therefore shift the burden of proof to those who characterize ideologically patterned attributions as rooted in underlying stable dispositional differences between liberals and conservatives (see Greenwald, 1975, on the power of null hypothesis results to shift burdens of proof).”
ResultPsychology Will Be a Much Better Science When We Change the Way We Analyze Data
Loftus, G. R. (1996) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep11512376 · Cited by 362 · see citing works
“Note that the logic of NHST demands that the equal signs in Equation 3 mean equal to an infinite number of decimal places . If one weakens this requirement such that "=" means "pretty much equal," then one must add additional assumptions specifying what is meant by "pretty much." Although the mathematical machinery fo…”
BackgroundAlbers, C. J., & Lakens, D. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.09.004 · Cited by 366 · see citing works
“…more likely to follow-up on initial studies that yielded higher e ff ect size estimates than initial studies that yielded lower e ff ect size estimates (cf. Greenwald, 1975), simply because these studies require less resources to observe a statistically signi fi cant result in the expected direction.”
ResultSex differences in influenceability.
Eagly, A. H. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.86 · Cited by 368 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
Communication, commitment, and cooperation in social dilemma.
Kerr, N. L., & Kaufman, C. M. (1994) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.66.3.513 · Cited by 358 · see citing works
Effects of video-game play on information processing: a meta-analytic investigation
Powers, K. L., Brooks, P. J., Aldrich, N. J., Palladino, M. A., & Alfieri, L. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0418-z · Cited by 348 · see citing works
“Due to concerns regarding the publication bias against null findings (e.g., Greenwald, 1975; Hubbard & Armstrong, 1997), we also compared effect sizes across published and unpublished studies, using journal impact factors to compare high-and low-impact publications.”
ResultThe Theories of Reasoned Action and Planned Behavior Applied to Exercise: A Meta-analytic Update
Downs, D. S., & Hausenblas, H. A. (2005) · Journal of Physical Activity and Health · doi:10.1123/jpah.2.1.76 · Cited by 357 · see citing works
Sensitivity Analysis for Publication Bias in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/rssc.12440 · Cited by 354 · see citing works
“, 2004; Coursol and Wagner, 1986) and selective submission of papers (Franco et al., 2014; Greenwald, 1975), as well as editors’ and reviewers’ biases (although some empirical work suggests that the latter may be a weaker influence than author decision making; see Lee et al.”
BackgroundSingle-sample tests for many correlations.
Larzelere, R. E., & Mulaik, S. A. (1977) · Psychological Bulletin · doi:10.1037/0033-2909.84.3.557 · Cited by 351 · see citing works
Cooper, H., & Patall, E. A. (2009) · Psychological Methods · doi:10.1037/a0015565 · Cited by 339 · see citing works
“Most notably, results of published data are generally more likely to reject the null hypothesis than unpublished data testing the same notion (Greenwald, 1975) and are more likely to exhibit larger effect sizes (Lipsey & Wilson, 1993).”
BackgroundResult
- Authors
- Greenwald, A. G.
- Affiliations
- The Ohio State University (US)
- Corresponding author
- Greenwald, Anthony G.
- Venue
- Psychological Bulletin
- Published
- 1975-01
- Volume
- 82
- Issue
- 1
- Pages
- 1-20
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2116618538
- Cited by
- 1,022
- References
- 47
- Citations by year
- 2026: 12 · 2025: 17 · 2024: 18 · 2023: 19 · 2022: 22 · 2021: 54 · 2020: 49 · 2019: 28 · 2018: 43 · 2017: 39 · 2016: 39 · 2015: 28 · 2014: 28 · 2013: 35 · 2012: 32
- FWCI
- 20.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- null hypothesis, publication bias, editorial decision-making, social psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
