Publication prejudices: An experimental study of confirmatory bias in the peer review system
The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“journal peer reviewers are more likely to recommend publication of reports with statistically positive results.”
From Reporting research results: a moral obligation for all researchers. (2007) · cites this paper as [7]
Experimental — accuracy not yet measuredIn the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
Citations
Cites
Social Theory and Social Structure
Ross, E. J., & Merton, R. K. (1958) · The American Catholic Sociological Review · doi:10.2307/3709394 · Cited by 10,315
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
(1975) · Medical Entomology and Zoology · Cited by 7,245
Social Theory and Social Structure.
Bock, K. E., & Merton, R. K. (1969) · American Sociological Review · doi:10.2307/2091967 · Cited by 6,660
Effects of externally mediated rewards on intrinsic motivation.
Deci, E. L. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030644 · Cited by 4,170
Lehman, H. (1972) · Philosophy of Science · doi:10.1086/288419 · Cited by 2,998
Cognitive Therapy and the Emotional Disorders
Ince, L. P. (1977) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.1977.31.4.633 · Cited by 2,892
Lepper, M. R., Greene, D., & Nisbett, R. E. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0035519 · Cited by 2,559
Lambert, N. M., Ketzenberger, K. E., & Kemp, C. A. (2014) · Encyclopedia of Special Education · doi:10.1002/9781118660584.ese0122 · Cited by 2,220
Social Stratification in Science
Cole, J. R., Cole, S., & deB. Beaver, D. (1974) · American Journal of Physics · doi:10.1119/1.1987897 · Cited by 1,135
Research works printed but not linked: 10
- Adams, J. A.Human memory. New York: McGraw-Hill, 1967.
- American Psychological Association.Standards for educational and psychological tests and manuals. Washington, D.C.: American Psychological Association, 1966.
- Bacon, F.Novum organum. New York: Bobbs-Merrill, 1960. (Originally published, 1621.)
- Dixon, B.What is science for? New York: Harper & Row, 1973.
- Lakatos, I., & Musgrave, A. (Eds.),Criticism and the growth of knowledge. London: Cambridge University Press, 1970.
- Mahoney, M. J., & DeMonbreun, B. G. Confirmatory bias in scientists and non-scientists.Cognitive Therapy and Research, 1977 (in press).
- Mahoney, M. J., & Kimper, T. P. From ethics to logic: A survey of scientists. In M. J. Mahoney,Scientist as subject. Cambridge, Massachusetts: Ballinger, 1976. Pp. 187?193.
- Neisser, V.Cognitive psychology. New York: Appleton-Century-Crofts, 1967.
- Popper, K. R.Objective knowledge: An evaluative approach. London: Oxford University Press, 1972.
- Weimer, W. B.Psychology and the conceptual foundations of science. Hillsdale, New Jersey; Lawrence Erlbaum Associates, 1977.
Cited by
Items 301–400 of 1,000
Kliewer, M. A., DeLong, D. M., Freed, K. S., Jenkins, C. B., Paulson, E. K., & Provenzale, J. M. (2004) · American Journal of Roentgenology · doi:10.2214/ajr.183.6.01831545 · Cited by 40
“Confirmatory bias is the tendency to favor manuscripts with views and data that coincide with one’s own beliefs and experiences [12, 13 ].”
BackgroundArmstrong, J. S. (1996) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.26.4.25 · Cited by 43
When will the editors start to edit?
Goodstein, L. D. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011377 · Cited by 41
“The Grievance Studies Affair” Project: Reconstructing and Assessing the Experimental Design
Lagerspetz, M. (2020) · Science Technology & Human Values · doi:10.1177/0162243920923087 · Cited by 59
“Mahoney (1977), and later on, Epstein (1990), approached the impact of confirmatory bias, that is, of granting credibility primarily to findings that seem to confirm the editors’ and reviewers’ previous beliefs.”
BackgroundBias against scientific novelty: A prepublication perspective
Liang, Z., Mao, J., & Li, G. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24725 · Cited by 41
“Regarding content characteristics, it is shown that reviewers judge a manuscript to be of higher quality when its methodology, data, and results are consistent with the reviewers' theoretical orientation (Mahoney, 1977).”
ResultPsychological science’s replicability crisis and what it means for science in the courtroom.
Chin, J. M. (2014) · Psychology Public Policy and Law · doi:10.1037/law0000012 · Cited by 43
Inventive productivity and the statistics of exceedances
Huber, J. C. (1999) · Scientometrics · doi:10.1007/bf02458467 · Cited by 39
Paap, K. R., Mason, L., Zimiga, B. M., Ayala-Silva, Y., & Frost, M. M. (2020) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021819900098 · Cited by 43
Epistemic Diversity and Editor Decisions: A Statistical Matthew Effect
Heesen, R., & Romeijn, J. (2019) · Philosophers Imprint · Cited by 47
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201697 · Cited by 38
Russell, M. C. (2008) · Social Science & Medicine · doi:10.1016/j.socscimed.2008.09.025 · Cited by 41
“Mahoney (1977b) found substantial evidence that reviewers made more positive appraisals of manuscripts whose findings confirmed principles of behavioral therapy and more negative appraisals for identical manuscripts reporting findings that disconfirmed the same principles: an entrenched zeitgeist…”
BackgroundInfluentialOn the reliability of highly magnified micrographs for structural analysis in materials science
Wortmann, M., Layland, A. S., Frese, N., Kahmann, U., Grothe, T., Storck, J. L., Błachowicz, T., Grzybowski, J., Hüsgen, B., & Ehrmann, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-71682-8 · Cited by 34
Data mining: Going beyond traditional statistics
Zhao, C., & Luan, J. (2006) · New Directions for Institutional Research · doi:10.1002/ir.184 · Cited by 38
Redding, R. E. (2013) · Society · doi:10.1007/s12115-013-9686-5 · Cited by 38
“…vary significantly according to whether the paper matches the reviewer’s own theoretical perspective (Abramowitz et al. 1975; Gaffan et al. 1995; Mahoney 1977), scientists tend to overlook methodological flaws in studies they feel are important whereas they tend to detect such flaws in other…”
BackgroundMarketing Science’s Pilgrimage to the Ivory Tower
Simon, H. (1994) · International series in quantitative marketing · doi:10.1007/978-94-011-1402-8_2 · Cited by 49
Strube, M., Gardner, W. P., & Hartmann, D. P. (1985) · Clinical Psychology Review · doi:10.1016/0272-7358(85)90030-3 · Cited by 38
Improving editorial procedures.
Crandall, R. (1990) · American Psychologist · doi:10.1037/0003-066x.45.5.665 · Cited by 38
Theoretical implications of failure to detect prepublished submissions
Eckberg, D. L. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0001133x · Cited by 39
Race and the Race for the White House: On Social Research in the Age of Trump
al‐Gharbi, M. (2018) · The American Sociologist · doi:10.1007/s12108-018-9373-5 · Cited by 41
“– with the resultant works endorsed by so many others who also know better – strongly suggests motivated reasoning and confirmation bias are undermining the quality of our scholarship and the rigor of our critiques (Koehler 1993; Mahoney 1977).”
BackgroundDoes the need for agreement among reviewers inhibit the publication controversial findings?
Armstrong, J. S., & Hubbard, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065699 · Cited by 40
Dickersin, K., Ssemanda, E., Mansell, C., & Rennie, D. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-44 · Cited by 36
Making progress in forecasting
Armstrong, J. S., & Fildes, R. A. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.04.007 · Cited by 37
“In Mahoney’s (1977) experiment, half of the reviewers received a “submitted paper” in which the results agreed with the beliefs commonly held by these reviewers, and half received a paper with results that contradicted the current beliefs.”
Evaluation of publication bias in response interruption and redirection: A meta‐analysis
Dowdy, A., Tincani, M., & Schneider, W. J. (2020) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.724 · Cited by 37
Rejection, rebuttal, revision: Some flexible features of peer review
Rubin, D. B. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011638 · Cited by 38
the Blind Shall See! the Question of Anonyminity in Journal Peer Review
Pontille, D., & Torny, D. (2014) · Scholars' Bank (University of Oregon) · doi:10.7264/n3542kvw · Cited by 29
Scientists’ Argumentative Reasoning
Mercier, H., & Heintz, C. (2013) · Topoi · doi:10.1007/s11245-013-9217-4 · Cited by 36
“Mahoney (1977) also offered such a demonstration by having psychologists review a fake article that either supported the scientists’ theories or not.”
BackgroundKeller, A. C. (2009) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/mup001 · Cited by 36
The two paradigms of software development research
Ralph, P. (2018) · Science of Computer Programming · doi:10.1016/j.scico.2018.01.002 · Cited by 35
Interdisciplinarity can aid the spread of better methods between scientific communities
Smaldino, P. E., & O’Connor, C. (2022) · Collective Intelligence · doi:10.1177/26339137221131816 · Cited by 43
“9 Mahoney (1977) asked reviewers to rate manuscripts with identical procedures, but different conclusions, and found significant bias towards findings supporting the reviewer’s own perspectives.”
BackgroundEthical Practice in the Accounting Publishing Process: Contrasting Opinions of Authors and Editors
Borkowski, S. C., & Welsh, M. J. (2000) · Journal of Business Ethics · doi:10.1023/a:1005939128790 · Cited by 40
Chalmers, I., & Dickersin, K. (2013) · F1000Research · doi:10.12688/f1000research.2-1.v1 · Cited by 37
Successful Authors and Effective Reviewers
Tsui, A. S., & Hollenbeck, J. R. (2008) · Organizational Research Methods · doi:10.1177/1094428108318429 · Cited by 35
Modeling publication bias using weighted distributions in a Bayesian framework
Larose, D. T., & Dey, D. K. (1998) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(97)00039-x · Cited by 40
Cicerone, K. D. (2012) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2011.12.003 · Cited by 38
Confirmatory bias in peer review
García, J. A., Sánchez, R. M. R., & Fdez-Valdivia, J. (2020) · Scientometrics · doi:10.1007/s11192-020-03357-0 · Cited by 41
“In the study by Mahoney (1977), a large number of reviewers were asked to evaluate works which described identical experimental procedures but which reported positive, negative, mixed, or no results.”
BackgroundChanging the culture of peer review for a more inclusive and equitable psychological science.
Aly, M., Colunga, E., Crockett, M. J., Goldrick, M., Gómez, P., Kung, F. Y. H., McKee, P. C., Pérez, M., Stilwell, S. M., & Diekman, A. B. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001461 · Cited by 37
Blind Analysis as a Correction for Confirmatory Bias in Physics and in Psychology
MacCoun, R. J., & Perlmutter, S. (2017) · Wiley eBooks · doi:10.1002/9781119095910.ch15 · Cited by 38
“…on confirmation bias is now quite large, and it has developed from many different disciplinary and theoretical streams (e.g., Bruner & Potter, 1964; Klayman & Ha, 1987; Lord, Ross, & Lepper, 1979; Mahoney, 1977; Nickerson, 1998; Platt, 1964; Rabin & Schrag, 1999; Snyder, 1984; Wason, 1960).”
BackgroundDynamic capital structure with heterogeneous beliefs and market timing
Yang, B. (2013) · Journal of Corporate Finance · doi:10.1016/j.jcorpfin.2013.05.003 · Cited by 43
Schema‐Guided Information Search in Stereotyping of the Elderly1
Carver, C. S., & de la Garza, N. H. (1984) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1984.tb02221.x · Cited by 33
Barriers to Creating KnowledgePreprint
Calhoun, M. A., & Starbuck, W. H. (2003) · SSRN Electronic Journal · Cited by 33
Procedural Injustice in the System of Peer Review and Scientific Misconduct
Clair, J. A. (2014) · Academy of Management Learning and Education · doi:10.5465/amle.2013.0243 · Cited by 35
The Bilingual Advantage Debate
de Bruin, A., & Sala, S. D. (2019) · Wiley eBooks · doi:10.1002/9781119387725.ch35 · Cited by 32
Peer review in an Era of Evaluation: Understanding the Practice of Gatekeeping in Academia
Forsberg, E., Geschwind, L., Levander, S., & Wermke, W. (2022) · Springer eBooks · doi:10.1007/978-3-030-75263-7 · Cited by 43
Ethics and the Accounting Publishing Process: Author, Reviewer, and Editor Issues
Borkowski, S. C., & Welsh, M. J. (1998) · Journal of Business Ethics · doi:10.1023/a:1005773522456 · Cited by 32
Generalization in Decision Research: The Role of Formal Models
Hogarth, R. M. (1986) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1986.4308976 · Cited by 40
The Mission of Marketing Science
Shugan, S. M. (2002) · Marketing Science · doi:10.1287/mksc.21.1.1.162 · Cited by 32
Phillips, J. S. (2011) · Current Medical Research and Opinion · doi:10.1185/03007995.2011.624090 · Cited by 32
Negative Results: Conceptual and Methodological Dimensions in Single-Case Intervention Research
Kratochwill, T. R., Levin, J. R., & Horner, R. H. (2017) · Remedial and Special Education · doi:10.1177/0741932517741721 · Cited by 35
“Traditionally, scientific journals have been unlikely to publish negative results (e.g., Atkinson, Furlong, & Wampold, 1982; Mahoney, 1977), including replication research (Neuliep & Crandall, 1993)―see Cook and Therrien (2017).”
BackgroundCite-seeing and reviewing: A study on citation bias in peer review
Stelmakh, I., Rastogi, C., Liu, R., Chawla, S., Echenique, F., & Shah, N. B. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283980 · Cited by 28
“Testing for other Biases in Peer Review A long line of literature (Mahoney, 1977; Blank, 1991; Lee, 2015; Tomkins et al., 2017; Stelmakh et al., 2020a, 2021; Manzoor and Shah, 2021, and many others) scru-tinizes the peer-review process for various biases.”
BackgroundConfirmation bias emerges from an approximation to Bayesian reasoning
Pilgrim, C., Sanborn, A. N., Malthouse, E., & Hills, T. T. (2024) · Cognition · doi:10.1016/j.cognition.2023.105693 · Cited by 38
Methods to increase research output: some tips looking at the MENA region
Balakrishnan, M. S. (2013) · International Journal of Emerging Markets · doi:10.1108/17468801311330301 · Cited by 35
“Mahoney (1977) found in an experimental study of 75”
BackgroundResultWang, Q., & Sandström, U. (2015) · Research Evaluation · doi:10.1093/reseval/rvv009 · Cited by 35
“A pivotal contribution by Mahoney (1977) found that ‘reviewers were strongly biased against manuscripts which reported results contrary to their theoretical perspective’.”
BackgroundPride and prejudice – What can we learn from peer review?
Sueur, H. L., Dagliati, A., Buchan, I. E., Whetton, A. D., Martin, G. P., Dornan, T., & Geifman, N. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1774527 · Cited by 31
The Quality and Interpretation of Mammographic Screening Trials for Women Ages 40-49
Glasziou, P., & Irwig, L. (1997) · JNCI Monographs · doi:10.1093/jncimono/1997.22.73 · Cited by 31
“Someone Should Do Something”: Replication and an Agenda for Collective Action
Kerr, G. F., Schultz, D. E., & Lings, I. N. (2015) · Journal of Advertising · doi:10.1080/00913367.2015.1077492 · Cited by 30
Truex, D. P., III, Cuellar, M. J., Takeda, H., & Vidgen, R. T. (2011) · European Journal of Information Systems · doi:10.1057/ejis.2011.16 · Cited by 32
Wampler, K. S. (1982) · Journal of Marriage and the Family · doi:10.2307/351460 · Cited by 30
Methodological issues in the meta‐analysis of quasi‐experiments
Bryant, F. B., & Wortman, P. M. (1984) · New Directions for Program Evaluation · doi:10.1002/ev.1375 · Cited by 30
Evaluating Registered Reports: A Naturalistic Comparative Study of Article Impact
Hopun, L. T., Singleton Thorn, F., Nosek, B. A., & Errington, T. M. (2017) · doi:10.31219/osf.io/5y8w7 · Cited by 29
Bias and Groupthink in Science’s Peer-Review System
Resnik, D. B., & Smith, E. M. R. (2020) · Springer eBooks · doi:10.1007/978-3-030-36822-7_9 · Cited by 35
Examining publication bias—a simulation-based evaluation of statistical tests on publication bias
Schneck, A. (2017) · PeerJ · doi:10.7717/peerj.4115 · Cited by 32
Star Wars: The Empirics Strike BackPreprint
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2089580 · Cited by 29
Loss Functions, Axioms, and Peer Review
Noothigattu, R., Shah, N., & Procaccia, A. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12554 · Cited by 28
“One prominent source of subjectivity is the disparity across reviewers in terms of their emphasis on the various criteria used for the overall evaluation of a paper.”
BackgroundProblems, Pitfalls, and Promise in the Peer-Review Process: Commentary on Trafimow & Rice (2009)
Cooper, M. L. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01109.x · Cited by 28
Where does business research go from here? Food-for-thought on academic papers in business research
Geuens, M. (2010) · Journal of Business Research · doi:10.1016/j.jbusres.2010.11.005 · Cited by 33
“Mahoney (1977), for example, wrote two versions of a paper: one that supported and one that contradicted prevailing beliefs and sent the versions to over 80 reviewers.”
BackgroundEve, M. P., Gadie, R., Odeniyi, V., & Parvin, S. (2022) · Digital humanities research · doi:10.14361/9783839455845 · Cited by 33
Bias and effort in peer review
García, J. A., Sánchez, R. M. R., & Fdez-Valdivia, J. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23307 · Cited by 28
“…whereas the costs include the cost of suppression of dissent against “mainstream” theories (Campanario & Martin, 2004; Martin, 1997) or the cost of bias against conclusions that contradict the theoretical perspective of the reviewer (Jelicic & Merckelbach, 2002; Mahoney, 1977; Nickerson, 1998).”
BackgroundEsarey, J. (2017) · PS Political Science & Politics · doi:10.1017/s1049096517001081 · Cited by 29
“…major errors in submitted papers Riis, and Karlsson 1994; Schroter et al. 2004; 2008), allows chance to strongly influence whether a paper will be published (Baxt et al. 1998; Cole, Cole, and Simon 1981; Mayo et al. 2006), and is subject to confirmatory biases by peer reviewers (Mahoney 1977).”
BackgroundResultAudit Reviewers’ Evaluation of Subordinates’ Work Quality
Tan, H., & Shankar, P. G. (2010) · Auditing A Journal of Practice & Theory · doi:10.2308/aud.2010.29.1.251 · Cited by 27
State of the Art: Clothing as a Form of Human Behavior
Hutton, S. S. (1984) · Home Economics Research Journal · doi:10.1177/1077727x8401200309 · Cited by 27
Publication in College & Research Libraries: Accepted, Rejected and Published Papers, 1980-1991
Hernon, P., SMITH, A. A., & Croxen, M. B. (1993) · College & Research Libraries · doi:10.5860/crl_54_04_303 · Cited by 27
Ultra-Short Race-Pace Training (USRPT) In Swimming: Current Perspectives
Nugent, F. J., Comyns, T., Kearney, P. E., & Warrington, G. D. (2019) · Open Access Journal of Sports Medicine · doi:10.2147/oajsm.s180598 · Cited by 33
Use of politeness strategies in signed open peer review
Nobarany, S., & Booth, K. S. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23229 · Cited by 31
“…of anonymity of reviewers on the quality of reviews, the potential effect on other types of biases such as biases in favor of positive results (Mahoney, 1977) or of results that match the reviewers’ stance (Abramowitz, Gomes, & Abramowitz, 1975) or that corroborate their previous work (Ernst…”
ResultThe principal‐agent problem in peer review
García, J. A., Sánchez, R. M. R., & Fdez-Valdivia, J. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23169 · Cited by 27
“In the context of peer review, confirmation bias is understood as reviewer bias against conclusions that contradict the theoretical perspective of the reviewer (Jelicic & Merckelbach, 2002; Mahoney, 1977).”
BackgroundThe perils of peer review in economics and other sciences
F�lster, S. (1995) · Journal of Evolutionary Economics · doi:10.1007/bf01199669 · Cited by 26
Peer-review research: Objections and obligations
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011754 · Cited by 29
Simultaneous Multiple Journal Submissions: The Case Against
Pressman, S. (1994) · American Journal of Economics and Sociology · doi:10.1111/j.1536-7150.1994.tb02601.x · Cited by 26
Bordage, G., & Caelleigh, A. S. (2001) · Academic Medicine · doi:10.1097/00001888-200109000-00013 · Cited by 26
Papaioannou, A. G., Machaira, E., & Theano, V. (2012) · International Journal of Sport and Exercise Psychology · doi:10.1080/1612197x.2013.753726 · Cited by 26
“…beliefs (e.g. Abramowitz, Gomes, & Abramowitz, 1975; Goodstein & Brazis, 1970), preconceptions (Ernst & Resch, 1994), and theoretical perspectives (Mahoney, 1977), while several editors and reviewers maybe tend to reject more easily submitted papers from less well-known institutions (Biggs, 1990).”
BackgroundQuestionable research practices following pre-registrationPreprint
Ikeda, A., Xu, H.-Q., Fuji, N., Zhu, S., & Yamada, Y. (2019) · PsyArXiv (OSF Preprints) · doi:10.24602/sjpr.62.3_281 · Cited by 26
Gausia, K., Thompson, S., Lindeman, M. A., Brown, L., & Perkins, D. (2015) · The Australian journal of rural health · doi:10.1111/ajr.12142 · Cited by 23
Results-blind review: a masked crusader for science
Woznyj, H. M., Grenier, K., Ross, R. L., Banks, G. C., & Rogelberg, S. G. (2018) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2018.1496081 · Cited by 27
“…may not be the case given the evidence of reviewer bias against certain results (Coursol & Wagner, 1986; Emerson et al., 2010; Franco et al., 2014; Mahoney, 1977; Sridharan & Greenland, 2009) and systematic suppression of certain kinds of findings (Fanelli, 2010; Leggett et al., 2013; Masicampo &…”
BackgroundResultInfluentialSome prolegomena to the study of literary difference
Hogan, P. C. (1994) · Poetics · doi:10.1016/0304-422x(94)90007-8 · Cited by 25
Blind versus Nonblind Review: Survey of Selected Medical Journals
Cleary, J. D., & Alexander, B. (1988) · Drug Intelligence & Clinical Pharmacy · doi:10.1177/106002808802200720 · Cited by 25
Examining the predictive validity of NIH peer review scores
Lindner, M. D., & Nakamura, R. K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0126938 · Cited by 24
Impact factor: Imperfect but not yet replaceable
Brody, S. (2012) · Scientometrics · doi:10.1007/s11192-012-0863-x · Cited by 24
“The more vocal opponents of a journal’s publications might seek to crush that journal for non-scientific reasons, exacerbating problems already present in the peer review system (Mahoney 1977).”
On Studying the Powerful (Or Fearing to Do So): A Vital Role for IRBs
Sieber, J. E. (1989) · IRB Ethics and Human Research · doi:10.2307/3564468 · Cited by 24
Reimagining peer review as an expert elicitation process
Marcoci, A., Vercammen, A., Bush, M., Hamilton, D. G., Hanea, A. M., Hemming, V., Wintle, B. C., Burgman, M. A., & Fidler, F. M. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06016-0 · Cited by 26
“Perhaps for these reasons, reviewers have been shown to favour manuscripts on topics familiar to them [42, 43], and to undervalue manuscripts reporting results that contradict their prior beliefs [44] or written by members of traditionally underrepresented groups in academia [45, 46].”
BackgroundMuradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292279 · Cited by 26
“Another potential filter for publishing studies is the direction of study results [16].”
BackgroundCommitment to a developing preference and predecisional distortion of information
Polman, E., & Russo, J. E. (2012) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2012.05.004 · Cited by 31
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Stelmakh, I., Shah, N. B., Singh, A., & Daumé, H. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449149 · Cited by 51
“Next, we note that reviewers’ evaluations are known to be subjective and have a high variance (Kerr et al., 1977; Mahoney, 1977; Ernst and Resch, 1994; Bakanic et al., 1987; Lamont, 2009).”
BackgroundDeterminants of abstract acceptance for the Digestive Diseases Week – a cross sectional study
Timmer, A., Hilsden, R. J., & Sutherland, L. R. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-13 · Cited by 25
The predictive validity of peer review: A neglected issue
Bornstein, R. F. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065717 · Cited by 25
Elson, M., Huff, M., & Utz, S. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919895419 · Cited by 26
“Although the number of intervention studies is slowly increasing (Malički, von Elm, & Marušic, 2014), they are a) still rather underrepresented, and b) with few exceptions (e.g., Epstein, 1990; Mahoney, 1977) restricted to peer review in biomedicine.”
BackgroundHirshman, B. R., Tang, J. A., Jones, L. A., Proudfoot, J. A., Carley, K. M., Marshall, L. K., Carter, B. S., & Chen, C. C. (2016) · Annals of Neurology · doi:10.1002/ana.24569 · Cited by 25
Establishing Interventions via a Theory-Driven Single Case Design Research Cycle
Kilgus, S. P., Riley‐Tillman, T. C., & Kratochwill, T. R. (2016) · School Psychology Review · doi:10.17105/spr45-4.477-498 · Cited by 25
“First, traditionally, journals are reluctant to publish studies wherein moderate to large effects are not demonstrated across the majority of, if not all, participants (Mahoney, 1977; Sham & Smith, 2014).”
BackgroundIn the Heart of a Storm: Leveraging Personal Relevance Through “Inside-Out” Research
Bishop, D. G., Eury, J. L., Gioia, D. A., Treviño, L. K., & Kreiner, G. E. (2019) · Academy of Management Perspectives · doi:10.5465/amp.2018.0089 · Cited by 27
Unpacking Ontological Perspectives in CEM Research: Everything Is Biased
Sherratt, F., & Leicht, R. (2019) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0001734 · Cited by 23
“As Mahoney (1977) found, academics can be very strongly biased against research that reports results or findings that contradict their own theoretical perspectives, and so when a realist meets a rationalist, bias within our own peer-review processes is something of a predictable phenomenon.”
BackgroundOpen Science and Single-Case Design Research
Cook, B. G., Johnson, A. H., Maggin, D. M., Therrien, W. J., Barton, E. E., Lloyd, J. W., Reichow, B., Talbott, E., & Travers, J. C. (2021) · Remedial and Special Education · doi:10.1177/0741932521996452 · Cited by 23
“In contrast, when the effect was large, data overlap low, and data variability low, the average rating was between somewhat and very likely to recommend publication (see also Mahoney, 1977).”
Background
Metadata
- Authors
- Mahoney, M. J.
- Affiliations
- Pennsylvania State University (US)
- Corresponding author
- Mahoney, Michael J.
- ORCID
- Mahoney, Michael J.
- Venue
- Cognitive Therapy and Research
- Published
- 1977-06
- Volume
- 1
- Issue
- 2
- Pages
- 161-175
- ISSN
- 0147-5916 (ISSN-L)
- 1573-2819
- OpenAlex
- W2164136002
- Semantic Scholar
- d8cf5c798397b6a0be1b41f18f979f0988f1ece7
- Cited by
- 1,013
- References
- 29
- Citations by year
- 2026: 22 · 2025: 18 · 2024: 25 · 2023: 28 · 2022: 52 · 2021: 45 · 2020: 52 · 2019: 40 · 2018: 31 · 2017: 51 · 2016: 38 · 2015: 43 · 2014: 41 · 2013: 51 · 2012: 23
- FWCI
- 5.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Cognitive Therapy and Research (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- confirmation bias, peer review, publication bias, scientific journals, manuscript review, interobserver agreement, negative results, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.