Publication prejudices: An experimental study of confirmatory bias in the peer review system
The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“journal peer reviewers are more likely to recommend publication of reports with statistically positive results.”
From Reporting research results: a moral obligation for all researchers. (2007) · cites this paper as [7]
Experimental — accuracy not yet measuredIn the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
Citations
Cites
Social Theory and Social Structure
Ross, E. J., & Merton, R. K. (1958) · The American Catholic Sociological Review · doi:10.2307/3709394 · Cited by 10,315
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
(1975) · Medical Entomology and Zoology · Cited by 7,245
Social Theory and Social Structure.
Bock, K. E., & Merton, R. K. (1969) · American Sociological Review · doi:10.2307/2091967 · Cited by 6,660
Effects of externally mediated rewards on intrinsic motivation.
Deci, E. L. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030644 · Cited by 4,170
Lehman, H. (1972) · Philosophy of Science · doi:10.1086/288419 · Cited by 2,998
Cognitive Therapy and the Emotional Disorders
Ince, L. P. (1977) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.1977.31.4.633 · Cited by 2,892
Lepper, M. R., Greene, D., & Nisbett, R. E. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0035519 · Cited by 2,559
Lambert, N. M., Ketzenberger, K. E., & Kemp, C. A. (2014) · Encyclopedia of Special Education · doi:10.1002/9781118660584.ese0122 · Cited by 2,220
Social Stratification in Science
Cole, J. R., Cole, S., & deB. Beaver, D. (1974) · American Journal of Physics · doi:10.1119/1.1987897 · Cited by 1,135
Research works printed but not linked: 10
- Adams, J. A.Human memory. New York: McGraw-Hill, 1967.
- American Psychological Association.Standards for educational and psychological tests and manuals. Washington, D.C.: American Psychological Association, 1966.
- Bacon, F.Novum organum. New York: Bobbs-Merrill, 1960. (Originally published, 1621.)
- Dixon, B.What is science for? New York: Harper & Row, 1973.
- Lakatos, I., & Musgrave, A. (Eds.),Criticism and the growth of knowledge. London: Cambridge University Press, 1970.
- Mahoney, M. J., & DeMonbreun, B. G. Confirmatory bias in scientists and non-scientists.Cognitive Therapy and Research, 1977 (in press).
- Mahoney, M. J., & Kimper, T. P. From ethics to logic: A survey of scientists. In M. J. Mahoney,Scientist as subject. Cambridge, Massachusetts: Ballinger, 1976. Pp. 187?193.
- Neisser, V.Cognitive psychology. New York: Appleton-Century-Crofts, 1967.
- Popper, K. R.Objective knowledge: An evaluative approach. London: Oxford University Press, 1972.
- Weimer, W. B.Psychology and the conceptual foundations of science. Hillsdale, New Jersey; Lawrence Erlbaum Associates, 1977.
Cited by
Items 101–200 of 1,000
Top Management Team Diversity: A Systematic Review
Homberg, F., & Bui, H. (2013) · Group & Organization Management · doi:10.1177/1059601113493925 · Cited by 202
“Others argue that reviewers more often find methodological flaws in nonmainstream papers (Lawrence, 2003; Mahoney, 1977).”
BackgroundModels of Data: A Theory of How People Evaluate Data
Chinn, C. A., & Brewer, W. F. (2001) · Cognition and Instruction · doi:10.1207/s1532690xci1903_3 · Cited by 166
THE IMPORTANCE OF DATA-SELECTION CRITERIA: META-ANALYSES OF STREAM PREDATION EXPERIMENTS
Englund, G., Sarnelle, O., & Cooper, S. D. (1999) · Ecology · doi:10.1890/0012-9658(1999)080[1132:tiodsc]2.0.co;2 · Cited by 162
Testing for Publication Bias in Political Science
Gerber, A. S., Green, D. P., & Nickerson, D. (2001) · Political Analysis · doi:10.1093/oxfordjournals.pan.a004877 · Cited by 157
Barriers to scientific contributions: The author's formula
Armstrong, J. S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011201 · Cited by 152
Dickersin, K., & Chalmers, I. (2011) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2011.11k042 · Cited by 165
Carneiro, C. F. D., Queiroz, V. G. S., Moulin, T. C., Carvalho, C. A. M., Haas, C. B., Rayêe, D., Henshall, D. E., De-Souza, E. A., Amorim, F. E., Boos, F. Z., et al. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00101-3 · Cited by 146
“Additionally, traditional peer review has various drawbacks (Walker and da Silva, 2015), including reviewer bias (Mahoney, 1977; Murray et al., 2018), lack of agreement among reviewers (Rothwell and Martyn, 2000; Pier et al., 2018) and vulnerability to various forms of system gaming such as…”
BackgroundHuman subjects review, personal values, and the regulation of social science research.
Ceci, S. J., Peters, D. P., & Plotkin, J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.9.994 · Cited by 146
Osterloh, M., & Frey, B. S. (2015) · Evaluation Review · doi:10.1177/0193841x14524957 · Cited by 123
“Methodological shortcomings are identified by referees in 71% of articles that contradict mainstream thinking but only in 25% of articles supporting orthodox views (Mahoney 1977).”
BackgroundSignificance tests as sorcery: Science is empirical—significance tests are not
Lambdin, C. (2012) · Theory & Psychology · doi:10.1177/0959354311429854 · Cited by 143
“…research as a whole (Andreski, 1972).3 Further, a sound replication that does not achieve statistical significance is not likely to be published anyway, even though the results of a study—so long as its methodology is sound—should be irrelevant as to whether it gets published (Mahoney, 1977).”
BackgroundBaxt, W. G., Waeckerle, J. F., Berlin, J. A., & Callaham, M. L. (1998) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(98)70006-x · Cited by 141
Confirmational Response Bias Among Social Work Journals
Epstein, W. M. (1990) · Science, Technology, & Human Values · doi:10.1177/016224399001500102 · Cited by 141
Research on human judgment and its application to clinical practice.
Faust, D. (1986) · Professional Psychology Research and Practice · doi:10.1037/0735-7028.17.5.420 · Cited by 151
Peer review and the publication process
Ali, P. A., & Watson, R. A. (2016) · Nursing Open · doi:10.1002/nop2.51 · Cited by 154
“Evidence suggests that reviewers tend to accept papers that provide confirmatory results and reject those that do not confirm established theories (Mahoney 1977).”
BackgroundReplication Research, Publication Bias, and Applied Behavior Analysis
Tincani, M., & Travers, J. C. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00191-5 · Cited by 144
“ABA researchers may be disinclined to submit for publication research studies that did not replicate previous results because the probability of successful publication is lower due to reviewer and/or editor bias against such studies (Mahoney, 1977; Shadish et al., 2016).”
BackgroundInfluentialTraining data influence analysis and estimation: a survey
Hammoudeh, Z., & Lowd, D. (2024) · Machine Learning · doi:10.1007/s10994-023-06495-7 · Cited by 228
Miller, C. C. (2006) · Academy of Management Journal · doi:10.5465/amj.2006.21794661 · Cited by 134
“In one study, different versions of a paper were sent to 75 reviewers from the Journal of Applied Behavioral Analysis (Mahoney, 1977).”
BackgroundA kinked meta‐regression model for publication bias correction
Bom, P. R. D., & Rachinger, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1352 · Cited by 133
Peer Review for Journals as it Stands Today—Part 2
CAMPANARIO, J. M. (1998) · Science Communication · doi:10.1177/1075547098019004002 · Cited by 132
Rego, A., Marques, C. S., Leal, S. C. H., Sousa, F., & Cunha, M. P. E. (2010) · The International Journal of Human Resource Management · doi:10.1080/09585192.2010.488459 · Cited by 182
“In this way we address some researchers’ suggestion that efforts are needed to mitigate the effects of the confirmatory bias in science, which emphasizes data consistent with preliminary hypotheses while ignoring or discounting those that are inconsistent (Mahoney 1977; Greenwald, Pratkanis, Leippe and Baumgardner 198…”
BackgroundResearch Handbook of Expatriates
Hutchings, K., & Michailova, S. (2017) · Edward Elgar Publishing eBooks · doi:10.4337/9781784718183 · Cited by 146
Shadish, W. R., Zelinsky, N. A. M., Vevea, J. L., & Kratochwill, T. R. (2016) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.308 · Cited by 130
Reardon, D. C. (2018) · SAGE Open Medicine · doi:10.1177/2050312118807624 · Cited by 142
Advocacy and Objectivity in Science
Armstrong, J. S. (1979) · Management Science · doi:10.1287/mnsc.25.5.423 · Cited by 128
Clark, C. J., & Winegard, B. M. (2020) · Psychological Inquiry · doi:10.1080/1047840x.2020.1721233 · Cited by 128
“For instance, Mahoney (1977) and Koehler (1993) both found that reviewer assessments of scientific articles and pamphlets were more positive if they were consistent with the prior perspective of the reviewer (see Armstrong, 1997 for an overview of studies on peer review).”
BackgroundJermias, J. (2001) · Accounting Organizations and Society · doi:10.1016/s0361-3682(00)00008-8 · Cited by 128
Clinical judgment and decision making in neuropsychology
Wedding, D., & Faust, D. (1989) · Archives of Clinical Neuropsychology · doi:10.1016/0887-6177(89)90016-4 · Cited by 128
Feminism and psychology: Critiques of methods and epistemology.
Eagly, A. H., & Riger, S. (2014) · American Psychologist · doi:10.1037/a0037372 · Cited by 107
Mahoney, M. J., Kazdin, A. E., & Kenigsberg, M. (1978) · Cognitive Therapy and Research · doi:10.1007/bf01172515 · Cited by 125
“Recently, one of the first truly experimental analyses of variables influencing editorial decisions showed that certain aspects of the manuscript itself may dramatically affect probability of acceptance (Mahoney, 1977).”
BackgroundSelf-citationEditorial Judgments: A Praxeology of ‘Voting’ in Peer Review
Hirschauer, S. (2009) · Social Studies of Science · doi:10.1177/0306312709335405 · Cited by 148
When is statistical significance not significant?
Filho, D. B. F., Paranhos, R., da Rocha, E. C., Batista, M., Silva, J. A. D., Santos, M. L., & Marino, J. G. (2013) · Brazilian Political Science Review · doi:10.1590/s1981-38212013000100002 · Cited by 197
Anderson, H. R., Atkinson, R. W., Peacock, J. L., Sweeting, M. J., & Marston, L. (2005) · Epidemiology · doi:10.1097/01.ede.0000152528.22746.0f · Cited by 125
Hughes, T. P., Baird, A. H., Dinsdale, E. A., Harriott, V. J., Moltschaniwskyj, N. A., Pratchett, M. S., Tanner, J. E., & Willis, B. L. (2002) · Ecology · doi:10.1890/0012-9658(2002)083[0436:drvuma]2.0.co;2 · Cited by 125
The "I," the "we," and the "when": A meta-analysis of motivational primacy in self-definition.
Gaertner, L., Sedikides, C., Vevea, J. L., & Iuzzini, J. (2002) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.83.3.574 · Cited by 118
Bornmann, L., & Daniel, H. (2005) · Scientometrics · doi:10.1007/s11192-005-0214-2 · Cited by 146
Association Between Funding Source and Study Outcome in Orthopaedic Research
Leopold, S. S., Warme, W. J., Braunlich, E. F., & Shott, S. (2003) · Clinical Orthopaedics and Related Research · doi:10.1097/01.blo.0000093888.12372.d9 · Cited by 121
“As reported previously, scientists have considered the possibility of positive outcome bias as early as the seventeenth century,(25) and it has been reported in the nonsurgical specialties for more than 2 decades.(21,22)”
BackgroundHypotheses in Marketing Science: Literature Review and Publication Audit
Armstrong, J. S., Brodie, R. J., & Parsons, A. G. (2001) · Marketing Letters · doi:10.1023/a:1011169104290 · Cited by 121
Open exchange and epistemic progress.
Mahoney, M. J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.1.29 · Cited by 128
Self-citationProsocial motives underlie scientific censorship by scientists: A perspective and research agenda
Clark, C. J., Jussim, L. J., Frey, K., Stevens, S. T., al‐Gharbi, M., Aquino, K. F., Bailey, J. M., Barbaro, N., Baumeister, R. F., Bleske-Rechek, A. L., et al. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2301642120 · Cited by 129
Hergovich, A., Schott, R., & Burger, C. (2010) · Current Psychology · doi:10.1007/s12144-010-9087-5 · Cited by 117
“Mahoney (1977) conducted a study, where 67 reviewers assessed different versions of a report on a fictitious experiment.”
BackgroundResultCan psychology become a science?
Lilienfeld, S. O. (2010) · Personality and Individual Differences · doi:10.1016/j.paid.2010.01.024 · Cited by 129
“Mahoney (1977) asked 75 journal reviewers who held strong behavioral orientations to evaluate simulated manuscripts that featured identical research designs but different results.”
BackgroundSystematic Reviews With Meta-Analysis: Why, When, and How?
Crocetti, E. (2016) · Emerging Adulthood · doi:10.1177/2167696815617076 · Cited by 91
“…and in line with hypotheses are more likely to be submitted for publication and to be accepted than study results which are nonsignificant or significant but in contrast with hypotheses (Cooper, DeNeve, & Charlton, 1997; Greenwald, 1975; Krzyzanowska, Pintilie, &Tannock, 2003; Mahoney, 1977).”
ResultPublication bias in studies of an applied behavior‐analytic intervention: An initial analysis
Sham, E., & Smith, T. (2014) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.146 · Cited by 114
“In the only published study of publication bias in the ABA literature, Mahoney (1977) examined the role of reviewers. He commented that reviewers for ABA journals tend to be strong proponents of ABA interventions and thus may be more accepting of studies that show benefits than studies that do not (Mahoney, 1977). To…”
BackgroundResultInfluentialProfiles in Patient Safety: Confirmation Bias in Emergency Medicine
Pines, J. M. (2006) · Academic Emergency Medicine · doi:10.1197/j.aem.2005.07.028 · Cited by 89
The Scientific Impotence Excuse: Discounting Belief‐Threatening Scientific Abstracts
Munro, G. D. (2010) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2010.00588.x · Cited by 103
“…earlier suggests that people are quite adept at finding methodological flaws in research reports that are inconsistent with their existing beliefs (Cohen, Aronson, & Steele, 2000; Koehler, 1993; Lord, Ross, & Lepper, 1979; Mahoney, 1977; Munro & Ditto, 1997; Munro, Leary, & Lasane, 2004).”
BackgroundThe present and future of peer review: Ideas, interventions, and evidence
Aczel, B., Barwich, A.-S., Diekman, A. B., Fishbach, A., Goldstone, R. L., Gomez, P., Gundersen, O. E., von Hippel, P. T., Holcombe, A. O., Lewandowsky, S., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401232121 · Cited by 87
Cacioppo, J. T., Petty, R. E., & Sidera, J. A. (1982) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(82)90057-9 · Cited by 115
A Three-Tier Hierarchy of Self-Potency
Sedikides, C., Gaertner, L., Luke, M. A., O’Mara, E. M., & Gebauer, J. E. (2013) · Advances in Experimental Social Psychology · doi:10.1016/b978-0-12-407188-9.00005-3 · Cited by 113
“…is inherent confirmation bias in single-theory testing, a notion shared by many scholars and philosophers of science (Gould, 1981; Lakatos, 1970; Mahoney, 1977; Rosenthal, 1966; Westfall, 1973); as such, testing the importance of just one of the selves is a form of single-theory testing that…”
BackgroundTowards a reduction in publication bias.
Newcombe, R. G. (1987) · BMJ · doi:10.1136/bmj.295.6599.656 · Cited by 113
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.001 · Cited by 108
“…& Higgins, 2012) and post-hoc dissonance (Festinger, 1957) may lead the original authors to dismiss evidence they would have accepted as disconfirmatory had it been available in the early stages of theoretical development (Mahoney, 1977; Nickerson, 1999; Schaller, 2016; Tversky & Kahneman, 1971).”
The Silent Majority: Manuscript Rejection and Its Impact on Scholars
Day, N. E. (2011) · Academy of Management Learning and Education · doi:10.5465/amle.2010.0027 · Cited by 121
Publication bias as a function of study characteristics.
Coburn, K. M., & Vevea, J. L. (2015) · Psychological Methods · doi:10.1037/met0000046 · Cited by 104
O’Keefe, D. J. (2015) · Health Communication · doi:10.1080/10410236.2014.974123 · Cited by 98
“…assessments are influenced not simply by a study’s methods (an appropriate basis for judging study quality) but also by irrelevant considerations such as whether the results were statistically significant or consistent with the reviewer’s own hypotheses (Ernst & Resch, 1994; Mahoney, 1977).”
ResultMiller, C. C., Glick, W. H., & Cardinal, L. B. (2005) · Journal of Organizational Behavior · doi:10.1002/job.325 · Cited by 108
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 116
“Three main factors identified in the literature are: publication bias, including reviewer bias and the file-drawer problem [8,11,12]; questionable research practices (QRPs), including HARKing and p-hacking [13–15,18] (see also [20,21] for researchers’ degrees of freedom); and studies with underpowered designs [17,18,2…”
BackgroundTracking Replicability as a Method of Post-Publication Open Evaluation
Hartshorne, J. K., & Schachner, A. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00008 · Cited by 107
“Peer review, the primary vetting procedure for publication, is often slow, contentious, and uneven (Mahoney, 1977; Cole et al., 1981; Peters and Ceci, 1982; Eysenck and Eysenck, 1992; Newton, 2010).”
BackgroundThe Negational Reference: or the Art of Dissembling
MacRoberts, M. H., & MacRoberts, B. R. (1984) · Social Studies of Science · doi:10.1177/030631284014001006 · Cited by 103
Stanford, P. K. (2015) · Synthese · doi:10.1007/s11229-015-0856-4 · Cited by 125
A Counterintuitive Hypothesis About Employment Interview Validity and Some Supporting Evidence.
Schmidt, F. L., & Zimmerman, R. D. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.3.553 · Cited by 120
“Research suggests that even scientists unfortunately tend to seek out and give credibility to those findings that support their preexisting beliefs while showing a strong tendency to reject findings that are inconsistent with their prior beliefs (Lord, Ross, & Lepper, 1979; Mahoney, 1977).”
BackgroundScafetta, N. (2012) · Journal of Atmospheric and Solar-Terrestrial Physics · doi:10.1016/j.jastp.2012.04.002 · Cited by 102
Ho, R. C.-M., Mak, K.-K., Tao, R., Lu, Y., Day, J. R., & Pan, F. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-74 · Cited by 100
“The current peer review process is based on the fundamental assumption that peer review is objective, rational and free of prejudice [3,4].”
BackgroundEvidence-based or biased? The quality of published reviews of evidence-based practices
Littell, J. H. (2008) · Children and Youth Services Review · doi:10.1016/j.childyouth.2008.04.001 · Cited by 116
“Initially identified by Francis Bacon (1621/1960) and investigated by Watson (1960, 1968) and others, numerous studies show that people (including scientists) are reluctant to consider evidence that is inconsistent with their predictions (e.g., Fugelsang, Stein, Green, & Dunbar, 2004; Mahoney, 1977).”
BackgroundInfluentialBiased Assimilation: Effects of Assumptions and Expectations on the Interpretation of New Evidence
Lord, C. G., & Taylor, C. A. (2009) · Social and Personality Psychology Compass · doi:10.1111/j.1751-9004.2009.00203.x · Cited by 89
Study Quality Assessment in Systematic Reviews of Research on Intervention Effects
Wells, K., & Littell, J. H. (2009) · Research on Social Work Practice · doi:10.1177/1049731508317278 · Cited by 95
“…study reports that contain statistically significant results and those that confirm research hypotheses are significantly more likely to be published than those with null or negative findings, all other relevant factors being equal (Begg, 1994; Mahoney, 1997; Rothstein, Sutton, & Bornstein, 2005).”
Kepes, S., Bennett, A. A., & McDaniel, M. A. (2014) · Academy of Management Learning and Education · doi:10.5465/amle.2013.0193 · Cited by 110
“Article reviews without any knowledge of the results reduces the biases that affect reviewers when evaluating the contribution of manuscripts (Epstein, 1990; Mahoney, 1977).”
BackgroundAssessing precision in the manuscript review process - a little better than a dice roll.
Lindsey, D. (1988) · Scientometrics · doi:10.1007/bf02020243 · Cited by 97
The Seer-Sucker Theory: The Value of Experts in ForecastingPreprint
Armstrong, J. S. (1980) · CogPrints (University of Southampton) · Cited by 97
The Collective Origins of Valued Originality: A Social Identity Approach to Creativity
Haslam, S. A., Adarves‐Yorno, I., Postmes, T., & Jans, L. (2013) · Personality and Social Psychology Review · doi:10.1177/1088868313498001 · Cited by 107
Lilienfeld, S. O. (2002) · American Psychologist · doi:10.1037//0003-066x.57.3.176 · Cited by 94
American psychology in the eighties: Who will buy?
Fishman, D. B., & Neigher, W. D. (1982) · American Psychologist · doi:10.1037/0003-066x.37.5.533 · Cited by 94
Experimental methods and outcome evaluation.
Mahoney, M. J. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.660 · Cited by 102
Self-citationThe benefits of preregistration and Registered Reports
Lakens, D., Mesquida, C., Rasti, S., & Ditroilo, M. (2024) · Evidence-Based Toxicology · doi:10.1080/2833373x.2024.2376046 · Cited by 104
Chong, P. (2013) · Social Studies of Science · doi:10.1177/0306312712475256 · Cited by 90
“…in the process of adjudicating scientific knowledge, including preconceived views of what constitutes ‘good science’ (Daston and Galison, 2007; Mahoney, 1977; Travis and Collins, 1991), debate about what legitimates criteria and methods for evaluation (Dirk, 1999; Guetzkow et al., 2004;…”
BackgroundPeer-review in a world with rational scientists: Toward selection of the average
Thurner, S., & Hanel, R. (2011) · The European Physical Journal B · doi:10.1140/epjb/e2011-20545-7 · Cited by 57
Honeycutt, N., & Freberg, L. A. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616667617 · Cited by 97
“…to discriminate when reviewing scholarly papers or grants is consistent with past research on confirmatory bias indicating that when evaluating research, college faculty are more critical of work that contradicts their personal views (e.g., Mahoney, 1977), but the other findings are novel.”
ResultCredibility of preprints: an interdisciplinary survey of researchers
Soderberg, C. K., Errington, T. M., & Nosek, B. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.201520 · Cited by 98
Sijtsma, K. (2015) · Psychometrika · doi:10.1007/s11336-015-9446-0 · Cited by 96
“…results more than non-significant results, whereas the journals in the field are liable to publication bias, which is the tendency to publish significant results supporting a desirable result and ignoring results that question the desirable result (e.g., Ferguson & Heene, 2012; Mahoney, 1977).”
ResultThe Air We Breathe: A Critical Look at Practices and Alternatives in the Peer-Review Process
Suls, J. M., & Martin, R. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01105.x · Cited by 98
“But the fact is that psychological scientists apparently have shown little interest in launching or joining such research (see Mahoney, 1977, for an exception).”
The development of medicines for children
Rocchi, F., & Tomasi, P. A. (2011) · Pharmacological Research · doi:10.1016/j.phrs.2011.01.016 · Cited by 88
“These studies are often not published in peer-reviewed ournals, as this is not a requirement for registration; nevertheless, he quality of these regulatory studies is very often higher than that f published studies, as the limitations of the peer review system re well known [1,2].”
Representing the advantaged: how politicians reinforce inequality
(2015) · Choice Reviews Online · doi:10.5860/choice.187680 · Cited by 87
van den Akker, O. R., van Assen, M. A. L. M., Bakker, M., Elsherif, M. M., Wong, T. K., & Wicherts, J. M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02277-0 · Cited by 99
“…articles with positive results for publication than those with negative results because they think positive results contribute more to science (Mahoney, 1977), and researchers write up or sub-mit positive results for publication more often than negative results because they think positive…”
BackgroundPeer Review Verfahren auf dem Prüfstand / Peer Review Research – Reviewed
Hirschauer, S. (2004) · Zeitschrift für Soziologie · doi:10.1515/zfsoz-2004-0104 · Cited by 85
“Es könnte sein, dass die schwache verbleibende Reliabilität in den Sozialwissenschaften vor allem auf den confirmatory bias (Mahoney 1977) zurückgeht, mit dem ein Feld seine Irrtümer fortschreibt.”
BackgroundInterpretations and methods: Towards a more effectively self-correcting social psychology
Jussim, L. J., Crawford, J. T., Anglin, S. M., Stevens, S. T., & Duarte, J. L. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.003 · Cited by 89
“Reviewers' theoretical (Epstein, 2004; Greenwald, Pratkanis, Leippe, & Baumgardner, 1986; Mahoney, 1977) and ideological (Abramowitz, Gomes, & Abramowitz, 1975) views can influence their evaluation of research reports, leading them to judge studies that oppose their beliefs more critically than…”
BackgroundFrom Manuscript Evaluation to Article Valuation: The Changing Technologies of Journal Peer Review
Pontille, D., & Torny, D. (2015) · Human Studies · doi:10.1007/s10746-014-9335-z · Cited by 78
“The anonymization of authors spread to other disciplines related to social sciences and the humanities in the mid-1970s, essentially due to the publication of articles examining the issue of reviewer “bias” (Zuckerman and Merton 1971; Cicchetti and Conn 1976; Mahoney 1977).”
Redding, R. E., & Reppucci, N. D. (1999) · Law and Human Behavior · doi:10.1023/a:1022322706533 · Cited by 92
Tell me more! assessing interactions with social robots from speech
Laban, G., George, J.-N., Morrison, V., & Cross, E. S. (2020) · Paladyn, Journal of Behavioral Robotics · doi:10.1515/pjbr-2021-0011 · Cited by 81
The constant causes of never-ending faddishness in the behavioral and social sciences
Starbuck, W. H. (2008) · Scandinavian Journal of Management · doi:10.1016/j.scaman.2008.11.005 · Cited by 81
Forecasting with Econometric Methods: Folklore Versus Fact
Armstrong, J. S. (1978) · The Journal of Business · doi:10.1086/296016 · Cited by 81
Can editors save peer review from peer reviewers?
D’Andrea, R., & O’Dwyer, J. P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186111 · Cited by 80
“As a result, the fairness, reliability, transparency, and sustainability of peer review have been repeatedly questioned [4, 5, 6, 7, 8, 9], with emphasis on a slew of potential biases by all parties involved [10].”
BackgroundDissemination biases in ecology: effect sizes matter more than quality
Barto, E. K., & Rillig, M. C. (2012) · Oikos · doi:10.1111/j.1600-0706.2011.19401.x · Cited by 74
Citizens' Perceptions of Ideological Bias in Research on Public Policy Controversies
MacCoun, R. J., & Paletz, S. (2009) · Political Psychology · doi:10.1111/j.1467-9221.2008.00680.x · Cited by 76
How we can make ecotoxicology more valuable to environmental protection
Hanson, M. L., Wolff, B. A., Green, J. W., Kivi, M., Panter, G. H., Warne, M. S. J., Ågerstrand, M., & Sumpter, J. P. (2016) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2016.07.160 · Cited by 84
“, depends on the reviewers' expertise and availability, as well as their own biases (Mahoney, 1977)) and is coupled with the needs of a journal to fill issues and increase impact factors.”
BackgroundEditorial peer review for improving the quality of reports of biomedical studies
Jefferson, T., Rudin, M., Brodney Folse, S., & Davidoff, F. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000016.pub2 · Cited by 73
Wikipedia and academic peer review: Wikipedia as a recognised medium for scholarly publication?
Black, E. W. (2008) · Online Information Review · doi:10.1108/14684520810865994 · Cited by 97
“OIR 32,1 Techniques designed to protect the identity of the writer and provide a greater degree of quality to the peer review system have also provided mixed results (Hill and Provost, 2003; Mahoney, 1977; Godlee et al., 1998; Gorman, 2007) (NB: Gorman not in list of references).”
BackgroundNull Effects and Publication Bias in Special Education Research
Cook, B. G., & Therrien, W. J. (2017) · Behavioral Disorders · doi:10.1177/0198742917709473 · Cited by 82
“Mahoney (1977) conducted an analogue experiment in which 75 reviewers for the Journal of Applied Behavior Analysis reviewed a version of a hypothetical SCD study; each version had the same introduction and method, but results varied.”
BackgroundMethodResultJournal policies and editors’ opinions on peer review
Hamilton, D. G., Fraser, H., Hoekstra, R., & Fidler, F. (2020) · eLife · doi:10.7554/elife.62529 · Cited by 73
“…reporting low error detection rates by reviewers, inconsistency among reviewers’ recommendations, and biases favouring prestigious institutions and established theories (Nylenna et al., 1994; Schroter et al., 2004; Schroter et al., 2008; Mahoney, 1977; Peters and Ceci, 1982; Kravitz et al., 2010).”
Quality and Peer Review of Research: An Adjudicating Role for Editors
Newton, D. P. (2010) · Accountability in Research · doi:10.1080/08989621003791945 · Cited by 84
“There is a tendency to favour arguments agreeing with the reviewer’s beliefs and to be hostile towards new ideas (Mahoney, 1977; Armstrong, 1980).”
ResultFritz, A., Scherndl, T., & Kühberger, A. (2012) · Theory & Psychology · doi:10.1177/0959354312436870 · Cited by 80
Shadish, W. R., Hedges, L. V., Horner, R. H., & Odom, S. L. (2015) · University of North Texas Digital Library (University of North Texas) · Cited by 87
Solutions to the Credibility Crisis in Management Science
Byington, E. K., & Felps, W. (2016) · Academy of Management Learning and Education · doi:10.5465/amle.2015.0035 · Cited by 77
Metadata
- Authors
- Mahoney, M. J.
- Affiliations
- Pennsylvania State University (US)
- Corresponding author
- Mahoney, Michael J.
- ORCID
- Mahoney, Michael J.
- Venue
- Cognitive Therapy and Research
- Published
- 1977-06
- Volume
- 1
- Issue
- 2
- Pages
- 161-175
- ISSN
- 0147-5916 (ISSN-L)
- 1573-2819
- OpenAlex
- W2164136002
- Semantic Scholar
- d8cf5c798397b6a0be1b41f18f979f0988f1ece7
- Cited by
- 1,013
- References
- 29
- Citations by year
- 2026: 22 · 2025: 18 · 2024: 25 · 2023: 28 · 2022: 52 · 2021: 45 · 2020: 52 · 2019: 40 · 2018: 31 · 2017: 51 · 2016: 38 · 2015: 43 · 2014: 41 · 2013: 51 · 2012: 23
- FWCI
- 5.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Cognitive Therapy and Research (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- confirmation bias, peer review, publication bias, scientific journals, manuscript review, interobserver agreement, negative results, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.