Publication prejudices: An experimental study of confirmatory bias in the peer review system
The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“journal peer reviewers are more likely to recommend publication of reports with statistically positive results.”
From Reporting research results: a moral obligation for all researchers. (2007) · cites this paper as [7]
Experimental — accuracy not yet measuredIn the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [10]
Citations
Cites
Social Theory and Social Structure
Ross, E. J., & Merton, R. K. (1958) · The American Catholic Sociological Review · doi:10.2307/3709394 · Cited by 10,315
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
(1975) · Medical Entomology and Zoology · Cited by 7,245
Social Theory and Social Structure.
Bock, K. E., & Merton, R. K. (1969) · American Sociological Review · doi:10.2307/2091967 · Cited by 6,660
Effects of externally mediated rewards on intrinsic motivation.
Deci, E. L. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030644 · Cited by 4,170
Lehman, H. (1972) · Philosophy of Science · doi:10.1086/288419 · Cited by 2,998
Cognitive Therapy and the Emotional Disorders
Ince, L. P. (1977) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.1977.31.4.633 · Cited by 2,892
Lepper, M. R., Greene, D., & Nisbett, R. E. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0035519 · Cited by 2,559
Lambert, N. M., Ketzenberger, K. E., & Kemp, C. A. (2014) · Encyclopedia of Special Education · doi:10.1002/9781118660584.ese0122 · Cited by 2,220
Social Stratification in Science
Cole, J. R., Cole, S., & deB. Beaver, D. (1974) · American Journal of Physics · doi:10.1119/1.1987897 · Cited by 1,135
Research works printed but not linked: 10
- Adams, J. A.Human memory. New York: McGraw-Hill, 1967.
- American Psychological Association.Standards for educational and psychological tests and manuals. Washington, D.C.: American Psychological Association, 1966.
- Bacon, F.Novum organum. New York: Bobbs-Merrill, 1960. (Originally published, 1621.)
- Dixon, B.What is science for? New York: Harper & Row, 1973.
- Lakatos, I., & Musgrave, A. (Eds.),Criticism and the growth of knowledge. London: Cambridge University Press, 1970.
- Mahoney, M. J., & DeMonbreun, B. G. Confirmatory bias in scientists and non-scientists.Cognitive Therapy and Research, 1977 (in press).
- Mahoney, M. J., & Kimper, T. P. From ethics to logic: A survey of scientists. In M. J. Mahoney,Scientist as subject. Cambridge, Massachusetts: Ballinger, 1976. Pp. 187?193.
- Neisser, V.Cognitive psychology. New York: Appleton-Century-Crofts, 1967.
- Popper, K. R.Objective knowledge: An evaluative approach. London: Oxford University Press, 1972.
- Weimer, W. B.Psychology and the conceptual foundations of science. Hillsdale, New Jersey; Lawrence Erlbaum Associates, 1977.
Cited by
Items 901–1,000 of 1,000
A Framework Is Required to Reduce Publication Bias The Academic Surgeon's View
Poon, R. T. P., & Wong, J. (2010) · Springer eBooks · doi:10.1007/978-3-540-71915-1_24
“Good vibrations”: Optimism, social network position and resource-attraction capability
Lopes, M. P. (2008) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 5
A Journey into Hope—Discovering Partial Solutions
Starbuck, W. H. (2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199288533.003.0004
One: Why Data Cleaning is Important: Debunking the Myth of Robustness
Osborne, J. W. (2013) · Sage eBooks · doi:10.4135/9781452269948.n1
Creating Better Social Science
Starbuck, W. H. (2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199288533.003.0001
A Journey into Disillusionment—Discovering Problems
Starbuck, W. H. (2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199288533.003.0002
Starbuck, W. H. (2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199288533.003.0003
Mindell, J. S. (2016) · Journal of Transport & Health · doi:10.1016/j.jth.2016.07.007
Subventions publiques et décisions de production des agriculteurs une analyse microéconomique
Minviel, J. (2015) · theses.fr (ABES) · doi:10.70675/78a13e0az73acz49c0zb3e0z4762a74833ba
“In addition, it has been documented that certain studies remain unpublished because of theoretical or ideological divergences, or conflicts of interest between researchers (Sterling, 1959; Mahoney, 1977).”
BackgroundCHAPTER 5 PEER REVIEW AND EV ALUATION OF R&D IMPACTS
Bozeman, B. L. (1993)
Mualla, N. D. (2017) · Journal of Business Theory and Practice · doi:10.22158/jbtp.v5n1p21
Judgments of Scientific Quality and Their Effects on Published Knowledge and Its Diffusion
Teplitskiy, M. (2016) · Knowledge@UChicago (University of Chicago) · doi:10.6082/3teqk-9xn28
Thinking Clearly about Psychology V1: Matters of Public Interest
Grove, & Cicchetti (1991) · Project Muse (Johns Hopkins University)
Lingham‐Soliar, T. (2015) · Springer eBooks · doi:10.1007/978-3-662-46005-4_7
RETHINKING EDUCATION RESEARCH METHODOLOGIES – DOMINANCE OF WESTERN PERSPECTIVES?
Ong, Y. S. (2016) · European Journal of Education Studies · doi:10.46827/ejes.v0i0.128
Garcia, B. R. V. (2017) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.8.2017.tde-29062017-095050 · Cited by 6
Culturally Responsive Classroom Management in an Urban After-School Program
Duez-Curry, M. (2017) · Cited by 4
Identifying targets of scientific inquiry: obstacles to the advancement of organization science
Newman, J. M., Lillis, M. P., Waite, M. L., & Krefting, L. A. (2009) · International Journal of Organization Theory and Behavior · doi:10.1108/ijotb-12-02-2009-b001
“Reviewers are more likely to reject articles with results differing from their own theoretical perspective (Mahoney, 1977).”
BackgroundExpert-Panel Accreditation Evaluation-Practices: An Autoethnographic Case-Study in Spain
Aróstegui, J. A. R. (2018) · International Journal of Educational Excellence · doi:10.18562/ijee.033
An analysis of rater effects in reviews of scientific manuscripts
Turner, D. P. (2017) · NC Digital Online Collection of Knowledge and Scholarship (The University of North Carolina at Greensboro)
An International Comparison of Educational Research Review Mechanisms
Yang, C.-R. W., & Wang, H. L. (2018) · Frontiers in Education Technology · doi:10.22158/fet.v1n2p180
“…is to avoid personal preference or confirmatory bias—the tendency to emphasize and believe experiences which support one’s views and to ignore or discredit those which do not (Mahoney, 1977)—while also understanding the research situation and new developments in his or her research domain.”
BackgroundPeer Review in Scholarly Journal Publishing
Roberts, J., Overstreet, K., Hendrick, R., & Mahar, J. (2019) · Springer eBooks · doi:10.1007/978-3-319-76040-7_5-1 · Cited by 1
Bazin, Y., & Magne, L. (2020) · Revue internationale de Psychosociologie · doi:10.3917/rips1.064.0123
Theory of the Academic BlockchainPreprint
Wright, M. (2020) · Preprints.org · doi:10.20944/preprints202007.0665.v1
Nguyen, M., Pham, T.-H., Huyền, N. T. T., Ho, M., & Vuong, Q. (2020) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/sk3e6
AS SOMBRAS CEGAS DE NARCISO (um estudo psicossocial sobre o imaginário coletivo)
Arruda, R. T. (2020) · PhilPapers (PhilPapers Foundation) · Cited by 1
Nguyen, M., Huyền, N. T. T., Pham, T.-H., Ho, M., & Vuong, Q. (2020) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/zhe9c
“Thus, this dominance is more likely to make innovative or new ideas under-evaluated or suppressed (Atkeson & Taylor, 2019; Mahoney, 1977).”
BackgroundThe Case for a Technology Solution to the Ethics Crisis in Academic Publishing
Wright, K. (2020) · International Journal of Technoethics · doi:10.4018/ijt.2021010101
Open Up: A Survey on Open and Non-anonymized Peer ReviewingPreprint
Besançon, L., Rönnberg, N., Löwgren, J., Tennant, J., & Cooper, M. D. (2019) · Preprints.org · doi:10.20944/preprints201905.0098.v1
Publication and Registration Bias in Randomized Clinical Trials
Al-Durra, M. (2019) · TSpace (University of Toronto) · Cited by 1
The Near-Miss Effect in Slot Machines: Over Half a Century LaterPreprint
Pisklak, J. M., Yong, J. J. H., & Spetch, M. L. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/md9u7
Protzko, J., Holcombe, A. O., & Simons, D. J. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/pq8m4
Çift Kör Hakemlik Sistemi Ne Kadar Kör
Eyerci, C. (2021) · Journal of Higher Education and Science
Portilla, J. G. (2021) · Contributions to economics · doi:10.1007/978-3-030-78498-0_13
How Blind is Double Blind Review?
Eyerci, C. (2021) · Journal of Higher Education and Science · doi:10.5961/jhes.2021.452
Cognitive Structure and the Example of Racism
(2001) · Duke University Press eBooks · doi:10.1215/9780822380375-005
(2001) · Duke University Press eBooks · doi:10.1215/9780822380375-004
(2001) · Duke University Press eBooks · doi:10.1215/9780822380375-002
(2001) · Duke University Press eBooks · doi:10.1215/9780822380375-003
(2001) · Duke University Press eBooks · doi:10.1215/9780822380375-006
(2001) · Duke University Press eBooks · doi:10.1215/9780822380375-007
Zarrin, S. (2022) · PDXScholar (Portland State University) · doi:10.15760/etd.7796 · Cited by 3
Machuca, D. E. (2022) · Synthese Library/Synthese library · doi:10.1007/978-3-030-91210-9_8
Civil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · Journal of Neurophysiology · doi:10.1152/jn.00338.2013
“27 However, this concept has been questioned repeatedly over the past few decades (2,6,10,12,13), 28”
BackgroundRevealing Reviewers’ Identities as Part of Open Peer Review and Analysis of the Review ReportsPreprint
Bolek, C., Marolov, D., Bolek, M., & Shopovski, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3962733
Civil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · American Journal of Physiology-Renal Physiology · doi:10.1152/ajprenal.00269.2013
“However, this concept has been questioned repeatedly over the past few decades (2, 6, 10, 12, 13), and some have even advocated eliminating prepublication peer review altogether (1, 7).”
BackgroundKolmar, M. (2021) · Classroom companion: economics · doi:10.1007/978-3-030-78167-5_10
Prosecution and Wrongful Convictions
DeShay, R. A., & WORRALL, J. L. (2013) · Springer eBooks · doi:10.1007/978-1-4614-5690-2_36
(2015) · CRC Press eBooks · doi:10.1201/b18131-22
Civil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.00237.2013
Rothstein, H. R. (2014) · Wiley eBooks · doi:10.1002/9781118445112.stat07071 · Cited by 1
Communicating uncertain experimental evidencePreprint
Davis, A., Davis, A. L., & Fischhoff, B. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/7d3g9
Winship, C. (2014) · Sociological Methods & Research · doi:10.1177/0049124114529199
Civil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · American Journal of Physiology-Gastrointestinal and Liver Physiology · doi:10.1152/ajpgi.00154.2013
Civil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00390.2013
“However, this concept has been questioned repeatedly over the past few decades (2, 6, 10, 12, 13), and some have even advocated eliminating prepublication peer review altogether (1, 7).”
BackgroundCivil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · American Journal of Physiology-Endocrinology and Metabolism · doi:10.1152/ajpendo.00261.2013
Morales, J. F., & Dols, J. M. F. (1981) · Studies in Psychology Estudios de Psicología · doi:10.1080/02109395.1981.10821264
Civil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · American Journal of Physiology-Cell Physiology · doi:10.1152/ajpcell.00135.2013
Civil, sensible, and constructive peer review in APS journals
Raff, H., & Brown, D. (2013) · Physiological Genomics · doi:10.1152/physiolgenomics.00080.2013
“However, this concept has been questioned repeatedly over the past few decades (2, 6, 10, 12, 13), and some have even advocated eliminating prepublication peer review altogether (1, 7).”
BackgroundMellenbergh, G. J. (2019) · Springer eBooks · doi:10.1007/978-3-030-12272-0_19
Vedula, S. S., Hróbjartsson, A., & Page, M. J. (2021) · Springer eBooks · doi:10.1007/978-3-319-52677-5_183-1
The pandemic threatens the Registered Reports system as well as human livesPreprint
Sasaki, K., & Yamada, Y. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6wdaz
Bye Bye Peer-Reviewed PublishingPreprint
Abambres, M., Salloom, T., Beganović, N., Dojka, R., Dow, S. R., Verma, T., & Takhar, S. S. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/4ezvf
“” Mahoney (1977) 44 Abambres M, et al. (2019).”
Improving Statistical Inferences: MOOC Enhances Conceptual Understanding Among Online Learners
Herrera-Bennett, A. (2019) · American Educational Research Association Annual Meeting (AERA) · doi:10.3102/1446221
13. Supplementary Material from Online Resources
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.13
Preface: A Vision of Transformed Conservation Practice
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.14
Operationalization Bias: A Suboptimal Research Practice in PsychologyPreprint
Reddy, N. N. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/prqyt
(2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199288533.002.0003
(2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199288533.002.0006
(2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199288533.002.0005
Generation and Acceptance of New Medical Knowledge
Velanovich, V. (2023) · Springer eBooks · doi:10.1007/978-3-031-30371-5_6
Droegemeier, K. K. (2023) · The MIT Press eBooks · doi:10.7551/mitpress/14740.001.0001 · Cited by 3
Open when, why, to whom? Changing challenges, perspectives and practices in a new research culture
Mugnaini, R., Urs, S. R., & Fradkin, C. (2023) · Frontiers research topics · doi:10.3389/978-2-8325-3873-9
Røge, K. M. (2022) · Journal of Pragmatic Constructivism · doi:10.7146/jopracon.v12i1.135312
(2021) · Odile Jacob eBooks · doi:10.3917/oj.merci.2021.01.0411
Yan, L., Cui, H., & Wang, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4877993
Who is Under the Spotlight in Peer Review Networks
Sun, X., Zhang, J., & Di, Z. (2024) · International Conference on Cloud and Network Computing (ICCNC) · doi:10.1109/iccnc63989.2024.00030
“Mahoney et al. [22] examined confirmatory bias in peer evaluations, finding a prevalent tendency to reject papers that challenge existing theories.”
BackgroundFacilitating Constructive Criticism of Established Scientific ParadigmsPreprint
Traunmüller, H. (2024) · Qeios · doi:10.32388/wnrvhr
Facilitating Well-Founded Criticism of Established Scientific ParadigmsPreprint
Traunmüller, H. (2024) · Qeios · doi:10.32388/wnrvhr.2
Gardegaront, M. (2023) · theses.fr (ABES) · doi:10.70675/7052ba14z0ba5z4d24zafd4z2a41ba9d0867
Is public support for the euro influenced by eurosystem speeches?
Hwang, I. D., Lustenberger, T., & Rossi, E. (2026) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2026.102880
Does Scientific Productivity Increase the Publication of Positive Results?
Schiekiera, L., & Niemeyer, H. (2025) · Collabra Psychology · doi:10.1525/collabra.137035
Littell, J. H., Valentine, J. C., Gorman, D. M., & Pigott, T. (2026) · Campbell Systematic Reviews · doi:10.1177/18911803261435892
Ivory Towers and Concrete Flowers: On the Relationship between Political Philosophy and Activism
Engelstad, A. (2025) · Philosophy · doi:10.1017/s0031819124000275
“For instance, research shows that academics are not immune from bias and political partisanship when judging each other’s works in what is supposed to be rigorous review processes (Abramowitz, Gomes, and Abramowitz, 1975; Mahoney, 1977).”
Is Public Support for the Euro Influenced by Eurosystem Speeches?Preprint
Lustenberger, T., Hwang, I. D., & Rossi, E. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5399907
Blind Faith: A Study on the Interrelations among Credibility, Trust, and Non-Exposure to the Media.
Gómez, I. N., Portela, M., Álvarez, M. V., & López, J. M. T. (2025) · International Journal of Marketing Communication and New Media · doi:10.54663/2182-9306.2025.v.13.n.24.189-216
Littell, J. H., Valentine, J. C., Gorman, D. M., & Pigott, T. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/y45cg_v2
Mini-Many Analysts: Using Registered Reports to Learn About Analytical Flexibility
Montoya, A. K., Yu, G. O., & Tawa, J. (2026) · Teaching of Psychology · doi:10.1177/00986283261474045
How Knowledge is Received Across Disciplines
Melissa, L. C., Steven, M. E., & Philip, G. B. (2014) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-62396-552-520251012
Wiedzę stosuj, środowisku służ, biurokraty nie zgniewaj! Recenzowanie jako praca graniczna
Remisiewicz, Ł. (2025) · Kultura i Społeczeństwo · doi:10.35757/kis.2025.69.4.9
Mike, M. S. (1999) · University Microfilms International eBooks
Carneiro, C. F. D., Queiroz, V. G. S., Moulin, T. C., Carvalho, C. A. M., Haas, C. B., Rayêe, D., Henshall, D. E., De‐Souza, E. A., Amorim, F. E., Boos, F. Z., et al. (2020) · Figshare · doi:10.6084/m9.figshare.13310563
Carneiro, C. F. D., Queiroz, V. G. S., Moulin, T. C., Carvalho, C. A. M., Haas, C. B., Rayêe, D., Henshall, D. E., De‐Souza, E. A., Amorim, F. E., Boos, F. Z., et al. (2020) · Figshare · doi:10.6084/m9.figshare.13310584
Metadata
- Authors
- Mahoney, M. J.
- Affiliations
- Pennsylvania State University (US)
- Corresponding author
- Mahoney, Michael J.
- ORCID
- Mahoney, Michael J.
- Venue
- Cognitive Therapy and Research
- Published
- 1977-06
- Volume
- 1
- Issue
- 2
- Pages
- 161-175
- ISSN
- 0147-5916 (ISSN-L)
- 1573-2819
- OpenAlex
- W2164136002
- Semantic Scholar
- d8cf5c798397b6a0be1b41f18f979f0988f1ece7
- Cited by
- 1,013
- References
- 29
- Citations by year
- 2026: 22 · 2025: 18 · 2024: 25 · 2023: 28 · 2022: 52 · 2021: 45 · 2020: 52 · 2019: 40 · 2018: 31 · 2017: 51 · 2016: 38 · 2015: 43 · 2014: 41 · 2013: 51 · 2012: 23
- FWCI
- 5.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Cognitive Therapy and Research (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Keywords
- confirmation bias, peer review, publication bias, scientific journals, manuscript review, interobserver agreement, negative results, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.