Publication prejudices: An experimental study of confirmatory bias in the peer review system
The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“journal peer reviewers are more likely to recommend publication of reports with statistically positive results.”
From Reporting research results: a moral obligation for all researchers. (2007) · cites this paper as [7]
Experimental — accuracy not yet measuredIn the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
Citations
Cites
Social Theory and Social Structure
Ross, E. J., & Merton, R. K. (1958) · The American Catholic Sociological Review · doi:10.2307/3709394 · Cited by 10,315
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
(1975) · Medical Entomology and Zoology · Cited by 7,245
Social Theory and Social Structure.
Bock, K. E., & Merton, R. K. (1969) · American Sociological Review · doi:10.2307/2091967 · Cited by 6,660
Effects of externally mediated rewards on intrinsic motivation.
Deci, E. L. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030644 · Cited by 4,170
Lehman, H. (1972) · Philosophy of Science · doi:10.1086/288419 · Cited by 2,998
Cognitive Therapy and the Emotional Disorders
Ince, L. P. (1977) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.1977.31.4.633 · Cited by 2,892
Lepper, M. R., Greene, D., & Nisbett, R. E. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0035519 · Cited by 2,559
Lambert, N. M., Ketzenberger, K. E., & Kemp, C. A. (2014) · Encyclopedia of Special Education · doi:10.1002/9781118660584.ese0122 · Cited by 2,220
Social Stratification in Science
Cole, J. R., Cole, S., & deB. Beaver, D. (1974) · American Journal of Physics · doi:10.1119/1.1987897 · Cited by 1,135
Research works printed but not linked: 10
- Adams, J. A.Human memory. New York: McGraw-Hill, 1967.
- American Psychological Association.Standards for educational and psychological tests and manuals. Washington, D.C.: American Psychological Association, 1966.
- Bacon, F.Novum organum. New York: Bobbs-Merrill, 1960. (Originally published, 1621.)
- Dixon, B.What is science for? New York: Harper & Row, 1973.
- Lakatos, I., & Musgrave, A. (Eds.),Criticism and the growth of knowledge. London: Cambridge University Press, 1970.
- Mahoney, M. J., & DeMonbreun, B. G. Confirmatory bias in scientists and non-scientists.Cognitive Therapy and Research, 1977 (in press).
- Mahoney, M. J., & Kimper, T. P. From ethics to logic: A survey of scientists. In M. J. Mahoney,Scientist as subject. Cambridge, Massachusetts: Ballinger, 1976. Pp. 187?193.
- Neisser, V.Cognitive psychology. New York: Appleton-Century-Crofts, 1967.
- Popper, K. R.Objective knowledge: An evaluative approach. London: Oxford University Press, 1972.
- Weimer, W. B.Psychology and the conceptual foundations of science. Hillsdale, New Jersey; Lawrence Erlbaum Associates, 1977.
Cited by
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Lord, C. G., Ross, L., & Lepper, M. R. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.11.2098 · Cited by 4,583
Rabin, M. (1997) · Journal of Economic Literature · Cited by 3,080
Why do humans reason? Arguments for an argumentative theory
Mercier, H., & Sperber, D. (2011) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000968 · Cited by 2,254
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Why Propensity Scores Should Not Be Used for Matching
King, G., & Nielsen, R. (2019) · Political Analysis · doi:10.1017/pan.2019.11 · Cited by 1,706
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079
First Impressions Matter: A Model of Confirmatory Bias
Rabin, M., & Schrag, J. L. (1999) · The Quarterly Journal of Economics · doi:10.1162/003355399555945 · Cited by 1,001
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
“For example, Mahoney (1977) found that reviewers judged the methodological soundness, data presentation, scientific contribution, and publishability of a manuscript to be of higher quality when its data were consistent with the reviewer's theoretical orientation.”
Chapter 10: Addressing reporting biases
Egger, M., & Moher, D. (2011) · Cited by 997
Paap, K. R., Johnson, H. A., & Sawi, O. (2015) · Cortex · doi:10.1016/j.cortex.2015.04.014 · Cited by 810
“Mahoney (1977) showed that experienced reviewers for a psychology journal (who believed they were providing real reviews) were biased in favor of positive results over mixed, negative, or null results.”
BackgroundA disconfirmation bias in the evaluation of arguments.
Edwards, K., & Smith, E. E. (1996) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.71.1.5 · Cited by 848
Scientific Guidelines for Conducting Integrative Research Reviews
Cooper, H. M. (1982) · Review of Educational Research · doi:10.2307/1170314 · Cited by 784
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
“A significant result will increase the possibility of publishing a result (Mahoney, 1977).”
BackgroundThe Good News-Bad News Effect: Asymmetric Processing of Objective Information about Yourself
Eil, D., & Rao, J. M. (2011) · American Economic Journal: Microeconomics · doi:10.1257/mic.3.2.114 · Cited by 657
“A robust finding is that when given the same evidence, both sides of the debate become more convinced of their prior opinions (Lord, Ross and Lepper 1979, Michael J. Mahoney 1977, Scott Plous 1991, Raymond Nickerson 1998).”
BackgroundCognitive Advantage in Bilingualism: An Example of Publication Bias?
de Bruin, A., Treccani, B., & Della Sala, S. (2015) · Psychological Science · doi:10.1177/0956797614557866 · Cited by 599
“Mahoney (1977) asked journal reviewers to referee manuscripts reporting positive, negative, mixed, or null results with identical methodological procedures.”
Pyszczynski, T., & Greenberg, J. (1987) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60417-7 · Cited by 635
Political diversity will improve social psychological science
Duarte, J. L., Crawford, J. T., Stern, C., Haidt, J., Jussim, L., & Tetlock, P. E. (2015) · Behavioral and Brain Sciences · doi:10.1017/s0140525x14000430 · Cited by 563
Under what conditions does theory obstruct research progress?
Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.216 · Cited by 574
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567
Beyond positive psychology? Toward a contextual view of psychological processes and well-being.
McNulty, J. K., & Fincham, F. D. (2012) · American Psychologist · doi:10.1037/a0024572 · Cited by 490
“Indeed, given that unpredicted effects are far less likely to be published than predicted results (Mahoney, 1977), presuming and thus predicting that a particular psychological characteristic will benefit people could lead researchers to accumulate knowledge consistent with that presumption—even if…”
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
A review of the relationships between human population density and biodiversity
Luck, G. W. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00028.x · Cited by 479
“Subjective decisions regarding the quality of studies can introduce bias into the review process since reviewers may be more inclined to identify methodological flaws in papers that report findings contrary to their own world view (Mahoney, 1977; Englund, Sarnelle & Cooper, 1999).”
BackgroundCicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
What is open peer review? A systematic review
Ross-Hellauer, T. (2017) · F1000Research · doi:10.12688/f1000research.11369.2 · Cited by 449
The Nature–Nurture Debates: 25 Years of Challenges in Understanding the Psychology of Gender
Eagly, A. H., & Wood, W. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613484767 · Cited by 408
“As seen in research on biases in judgment and decision making, scientists are clever about setting up tests and conducting analyses to favor their own theories and are highly critical of findings that challenge their ideas (Mahoney, 1977; Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012).”
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461
“For example, various concerned researchers have proposed the elimination of hypothesis testing (Begg & Berlin, 1989; Hubbard & Armstrong, 1997; Nunnally, 1960), a priori peer review and failure to publish studies with inadequate sample sizes (Newcombe, 1987), development of a more positive attitude toward nonsignifica…”
BackgroundSterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
BIASES IN THE INTERPRETATION AND USE OF RESEARCH RESULTS
MacCoun, R. J. (1998) · Annual Review of Psychology · doi:10.1146/annurev.psych.49.1.259 · Cited by 417
Star Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication
Starbuck, W. H. (2005) · Organization Science · doi:10.1287/orsc.1040.0107 · Cited by 434
“One interesting finding is that reviewers criticize the methodology of studies that cast doubt on theories that the reviewers like and they applaud the methodology of studies that support theories that they like (Mahoney 1977, 1979).”
BackgroundInfluentialDecision Making in Health and Medicine: Integrating Evidence and Values
Hunink, M. G. M., Weinstein, M. C., Wittenberg, E., Drummond, M. F., Pliskin, J. S., Wong, J. B., & Glasziou, P. P. (2014) · doi:10.1017/cbo9781139506779 · Cited by 425
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424
“05, are considered by many researchers [27,28] and reviewers [29] to lend support to the hypothesized effects.”
BackgroundBlank, R. M. (2016) · American Economic Review · Cited by 421
Testing and adjusting for publication bias
Møller, A. P., & Jennions, M. D. (2001) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(01)02235-2 · Cited by 418
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 370
“Evidence for both categories of bias exists: Publication bias has been observed in peer review (Atkinson et al., 1982; Mahoney, 1977) and in longitudinal data from a National Science Foundation grant program that found a file-drawering effect for studies with negative results (Franco et al., 2014,…”
BackgroundInfluentialBiased evaluation and persistence in gambling.
Gilovich, T. D. (1983) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.44.6.1110 · Cited by 408
Administrative Science As Socially Constructed Truth
Astley, W. G. (2019) · Postmodern Management Theory · doi:10.4324/9780429431678-1 · Cited by 403
Grey Literature in Meta-Analyses
Conn, V. S., Valentine, J. C., Cooper, H. M., & Rantz, M. J. (2003) · Nursing Research · doi:10.1097/00006199-200307000-00008 · Cited by 400
“Although publication bias is a complex phenomenon, the problem includes both researchers’ habits in submitting articles and editors’ habits when evaluating them (Dickersin & Min, 1993; Dickersin, 1997; Mahoney, 1977; Stern & Simes, 1997).”
BackgroundThe (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356
“These results show that NHST continues to be used in ways that defy the guidelines as proposed by the Task Force on Statistical Inference (Wilkinson and Task Force on Statistical Inference, 1999) and the last two versions of the APA Publication Manual (American Psychological Association, 2001, 2010).”
BackgroundInfluentialMeasuring the effectiveness of scientific gatekeeping
Siler, K., Lee, K., & Bero, L. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1418218112 · Cited by 333
“Although peer review is widely cited as central to academic evaluation (2, 3), numerous scholars have expressed concern about the effectiveness of peer review, particularly regarding the tendency to protect the scientific status quo and suppress innovative findings (4, 5).”
BackgroundChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
“…grants are given, and careers are made on the production of positive results not negative results, tidy evidence and explanation not uncertainty and exceptions, and novel findings not replications or incremental extensions of prior work (Giner- Sorolla, 2012; Mahoney, 1977; Nosek et al., 2012).”
ResultImpediments to accurate clinical judgment and possible ways to minimize their impact.
Arkes, H. R. (1981) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.49.3.323 · Cited by 340
Lilienfeld, S. O., Ritschel, L. A., Lynn, S. J., Cautin, R. L., & Latzman, R. D. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.09.008 · Cited by 341
“Good scientists, including clinical scientists, are probably just as prone to confirmation bias and other errors as are poor scientists (Mahoney, 1977).”
BackgroundEditorial peer review for improving the quality of reports of biomedical studies
Jefferson, T., Rudin, M., Brodney Folse, S., & Davidoff, F. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000016.pub3 · Cited by 312
“Mahoney 1977 {published data only} Mahoney MJ. Publication prejudices: An experimental study of confirmatory bias in the peer review system.”
Anders Ericsson, K., Roring, R. W., & Nandagopal, K. (2007) · High Ability Studies · doi:10.1080/13598130701350593 · Cited by 282
“Furthermore, evidence exists that scientists' evaluations may be biased in several ways; for instance, several studies illustrate how having prestigious reputations (Peters & Ceci, 1982 ), having results matching the reviewer's own beliefs (Mahoney, 1977 ), and even subtler details (self‐citation biases: Mahoney et al.”
BackgroundSoderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
Five pitfalls in decisions about diagnosis and prescribing
Klein, J. G. (2005) · BMJ · doi:10.1136/bmj.330.7494.781 · Cited by 289
A multi-disciplinary perspective on emergent and future innovations in peer review
Tennant, J. P., Dugan, J. M., Graziotin, D., Jacques, D. C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C. K., Crick, T., et al. (2017) · F1000Research · doi:10.12688/f1000research.12037.1 · Cited by 289
Easley, R. W., Madden, C. S., & Dunn, M. G. (2000) · Journal of Business Research · doi:10.1016/s0148-2963(98)00079-4 · Cited by 275
Hypothesis Testing in Ecology: Psychological Aspects and the Importance of Theory Maturation
Loehle, C. (1987) · The Quarterly Review of Biology · doi:10.1086/415619 · Cited by 288
Shadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2014) · Journal of School Psychology · doi:10.1016/j.jsp.2013.11.005 · Cited by 269
“In BSDs, the reason is that reviewers have a lower probability of recommending publication of a manuscript when it reports nonsignificant results (Mahoney, 1977).”
BackgroundMeta-analysis. A quantitative approach to research integration
Thacker, S. B. (1988) · JAMA · doi:10.1001/jama.259.11.1685 · Cited by 237
Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers
Ross-Hellauer, T., Deppe, A., & Schmidt, B. (2017) · PLOS ONE · doi:10.1371/journal.pone.0189311 · Cited by 243
Sud, P., & Thelwall, M. (2014) · Scientometrics · doi:10.1007/s11192-013-1117-2 · Cited by 251
“reviewers do not reject papers because they are written by women’’ (Ceci and Williams 2011), nepotism (Wennerås and Wold 1997), bias against research that supports competing theoretical perspectives (Mahoney 1977), and bias against research competing for limited publication space (Blackburn and Hakel 2006).”
BackgroundHow to review the evidence: systematic identification and review of the scientific literature
Glasziou, P., Irwig, L., Bain, C., & Colditz, G. A. (2000) · Cited by 293
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Peer review for journals: Evidence on quality control, fairness, and innovation
Armstrong, J. S. (1997) · Science and Engineering Ethics · doi:10.1007/s11948-997-0017-3 · Cited by 262
The top-ten in journal impact factor manipulation
Falagas, M. E., & Alexiou, V. G. (2008) · Archivum Immunologiae et Therapiae Experimentalis · doi:10.1007/s00005-008-0024-5 · Cited by 250
“However, in many circumstances it is treated with a strongly biased way by the editors and reviewers [ 7 ].”
BackgroundThe Production of Knowledge: The Challenge of Social Science Research
Starbuck, W. H. (2006) · doi:10.1093/acprof:oso/9780199288533.001.0001 · Cited by 201
Should Unpublished Data Be Included in Meta-analyses?: Current Convictions and Controversies
Cook, D. J. (1993) · JAMA · doi:10.1001/jama.1993.03500210049030 · Cited by 259
Quality of evidence revealing subtle gender biases in science is in the eye of the beholder
Handley, I. M., Brown, E. R., Moss-Racusin, C. A., & Smith, J. L. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510649112 · Cited by 240
“However, there is ample reason to predict that the bias will actually strengthen as people receive greater amounts of information, because they will (unintentionally) process that information based on initial impressions and per their motivation to arrive at a particular conclusion (42, 48, 49).”
BackgroundResultEffect of public subsidies on farm technical efficiency: a meta-analysis of empirical results
Minviel, J. J., & Latruffe, L. (2017) · Applied Economics · doi:10.1080/00036846.2016.1194963 · Cited by 224
“In addition, it has been documented that certain studies remain unpublished because of theoretical or ideological divergences, or conflicts of interest between researchers (Sterling 1959; Mahoney 1977).”
BackgroundOn the Perception of Social Consensus
Krueger, J. (1998) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60384-6 · Cited by 254
Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.406 · Cited by 245
Review of methodology of quantitative reviews using meta‐analysis in ecology
Gates, S. (2002) · Journal of Animal Ecology · doi:10.1046/j.1365-2656.2002.00634.x · Cited by 235
A Modest Proposal: How We might change the Process and Product of Managerial Research
Pfeffer, J. (2007) · Academy of Management Journal · doi:10.5465/amj.2007.28166117 · Cited by 234
“So, for instance, Mahoney (1977) found that referees were more likely to reject a study with evidence that disconfirmed widely held hypotheses and were likely to accept an otherwise-identical paper that supported existing beliefs.”
BackgroundWhat is open peer review? A systematic review
Ross-Hellauer, T. (2017) · F1000Research · doi:10.12688/f1000research.11369.1 · Cited by 232
“Next, “confirmatory bias” is theorized to lead to conservatism, biasing reviewers against innovative methods or results contrary to dominant theoretical perspectives (Chubin & Hackett, 1990; Garcia et al., 2016; Mahoney, 1977).”
BackgroundStructuring decision problems and the ‘bias heuristic’
Berkeley, D., & Humphreys, P. (1982) · Acta Psychologica · doi:10.1016/0001-6918(82)90042-7 · Cited by 228
Coveney, P. V., Dougherty, E. R., & Highfield, R. R. (2016) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2016.0153 · Cited by 229
Demographic Effects in Environmental Aesthetics: A Meta-Analysis
Stamps, A. E. (1999) · Journal of Planning Literature · doi:10.1177/08854129922092630 · Cited by 218
Is nedocromil sodium effective treatment for asthma?
O'Byrne, P., & Cook, D. (1993) · European Respiratory Journal · doi:10.1183/09031936.93.06010005 · Cited by 223
Sociopolitical diversity in psychology: The case for pluralism.
Redding, R. E. (2001) · American Psychologist · doi:10.1037//0003-066x.56.3.205 · Cited by 108
The Preregistration Revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 168
Factors influencing the use of a decision rule in a probabilistic task
Arkes, H. R., Dawes, R. M., & Christensen, C. (1986) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(86)90046-4 · Cited by 216
Problems and Limitations in Conducting Systematic Reviews
Egger, M., Dickersin, K., & Smith, G. D. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch3 · Cited by 202
Emerging trends in peer review—a survey
Walker, R., & Rocha da Silva, P. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00169 · Cited by 190
“Two experimental studies show that reviewers prefer papers reporting positive results to those reporting negative or mixed results, even when the underlying methodologies are identical (Mahoney, 1977; Emerson et al., 2010).”
BackgroundMethodInfluentialLangfeldt, L. (2001) · Social Studies of Science · doi:10.1177/030631201031006002 · Cited by 204
“The main characteristic of peer review – that quality criteria have no standard operationalization, and that judgements depend on the ‘intimate craft knowledge’ (Ravetz 1971, p. 274) of the reviewers – is the main problem for students of peer review; biases are hard to prove for outsiders.5 The aim of the present stud…”
BackgroundHojat, M., Gonnella, J. S., & Caelleigh, A. S. (2003) · Advances in Health Sciences Education · doi:10.1023/a:1022670432373 · Cited by 199
“Mahoney (1977) conducted an empirical study on “confirmatory bias,” defined as a tendency of some reviewers to accept outcomes that agree with commonly accepted theories and to discredit those that do not.”
BackgroundMethodPeer Review for Journals as it Stands Today—Part 1
CAMPANARIO, J. M. (1998) · Science Communication · doi:10.1177/1075547098019003002 · Cited by 195
Redundancy, disaggregation, and the integrity of medical research
Huston, P., & Moher, D. (1996) · The Lancet · doi:10.1016/s0140-6736(96)90153-1 · Cited by 194
The psychology of science: Review and integration of a nascent discipline.
Feist, G. J., & Gorman, M. E. (1998) · Review of General Psychology · doi:10.1037//1089-2680.2.1.3 · Cited by 191
A Task-Based Taxonomy of Cognitive Biases for Information Visualization
Dimara, E., Franconeri, S., Plaisant, C., Bezerianos, A., & Dragicevic, P. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2872577 · Cited by 159
“Another well-known example is the confirmation bias , according to which people seek and favor information that confirms their beliefs [19].”
BackgroundInterrater agreement for journal manuscript reviews.
Whitehurst, G. J. (1984) · American Psychologist · doi:10.1037//0003-066x.39.1.22 · Cited by 189
Miller, A. G., McHoskey, J. W., Bane, C. M., & Dowd, T. G. (1993) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.64.4.561 · Cited by 188
Shall We Vote on Values, But Bet on Beliefs?
HANSON, R. (2013) · Journal of Political Philosophy · doi:10.1111/jopp.12008 · Cited by 178
“In peer review, failures include very low levels of agreement (Chubin & Hackett, 1990), strong biases against obscure institutions (Peters & Ceci, 1982), and biased objections to methods that give disliked results (Mahoney, 1977).”
BackgroundBaldwin, M. (2018) · Isis · doi:10.1086/700070 · Cited by 161
Lynch, J. R., Cunningham, M. R. A., Warme, W. J., Schaad, D. C., Wolf, F. M., & Leopold, S. S. (2007) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200705000-00013 · Cited by 173
Active and Dynamic Self-Regulation of Learning Processes
Iran-Nejad, A. (1990) · Review of Educational Research · doi:10.2307/1170507 · Cited by 175
Maher, C. G., Sherrington, C., Elkins, M., Herbert, R. D., & Moseley, A. M. (2004) · Physical Therapy · doi:10.1093/ptj/84.7.644 · Cited by 181
Turning Lemons into Lemonade: Where Is the Value in Peer Reviews?
Starbuck, W. H. (2003) · Journal of Management Inquiry · doi:10.1177/1056492603258972 · Cited by 175
“Mahoney (1977, 1979) submitted five manuscripts to 75 people who had recently reviewed for the Journal of Applied Behavior Analysis.”
BackgroundResearch on scientific journals: Implications for editors and authors
Armstrong, J. S. (1982) · Journal of Forecasting · doi:10.1002/for.3980010109 · Cited by 174
Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors
Resnik, D. B., & Elmore, S. A. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-015-9625-5 · Cited by 161
The Oxford Handbook of Group Creativity and Innovation
(2019) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190648077.001.0001 · Cited by 150
Lindsey, D. (1989) · Scientometrics · doi:10.1007/bf02017198 · Cited by 164
The dependent patient: A practitioner's guide.
Bornstein, R. F. (2005) · American Psychological Association eBooks · doi:10.1037/11085-000 · Cited by 163
Metadata
- Authors
- Mahoney, M. J.
- Affiliations
- Pennsylvania State University (US)
- Corresponding author
- Mahoney, Michael J.
- ORCID
- Mahoney, Michael J.
- Venue
- Cognitive Therapy and Research
- Published
- 1977-06
- Volume
- 1
- Issue
- 2
- Pages
- 161-175
- ISSN
- 0147-5916 (ISSN-L)
- 1573-2819
- OpenAlex
- W2164136002
- Semantic Scholar
- d8cf5c798397b6a0be1b41f18f979f0988f1ece7
- Cited by
- 1,013
- References
- 29
- Citations by year
- 2026: 22 · 2025: 18 · 2024: 25 · 2023: 28 · 2022: 52 · 2021: 45 · 2020: 52 · 2019: 40 · 2018: 31 · 2017: 51 · 2016: 38 · 2015: 43 · 2014: 41 · 2013: 51 · 2012: 23
- FWCI
- 5.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Cognitive Therapy and Research (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- confirmation bias, peer review, publication bias, scientific journals, manuscript review, interobserver agreement, negative results, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.