Commensuration Bias in Peer Review
Abstract
To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. Commensuration bias also foregrounds a range of structural strategies for realigning peer review practices and institutions with the aims of science.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Commensuration bias is a source of subjectivity in paper review, arising from reviewers having differing opinions about the relative importance of various criteria in determining overall quality of a paper.
“In the context of paper review, one source of subjectivity arises from reviewers having differing opinions about the relative importance of various criteria in determining overall quality of a paper, a phenomenon referred to as “commensuration bias” [39].”
From Peer reviews of peer reviews: A randomized controlled trial and other experiments (2025) · cites this paper as [39]
Another calibration issue occurs when reviewers employ different ways of combining individual criteria (such as novelty, clarity, etc.) into their overall recommendation score-often referred to as 'commensuration bias'.
“Another calibration issue occurs when reviewers employ different ways of combining individual criteria (such as novelty, clarity, etc.) into their overall recommendation score-often referred to as 'commensuration bias' (Lee, 2015).”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Lee, 2015
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
The Essential Tension: Selected Studies in Scientific Tradition and Change
Kuhn, T. S. (1977) · doi:10.7208/chicago/9780226217239.001.0001 · Cited by 3,861 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1× · see citing works
“Publication bias appears to be caused primarily by authors who fail to submit null results in preemptive anticipation of their likely rejection (Dickersin, Min, and Meinert 1992; Ioannidis 1998) and perceived unimportance (Easterbrook et al. 1991).”
BackgroundMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Commensuration as a Social Process
Espeland, W. N., & Stevens, M. L. (1998) · Annual Review of Sociology · doi:10.1146/annurev.soc.24.1.313 · Cited by 2,113 · see citing works
Rising Above The Gathering Storm: Energizing and Employing America for a Brighter Economic Future
Augustine, N. R. (2006) · Cited by 2,066 · see citing works
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · see citing works
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · see citing works
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · see citing works
Misconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232 · cited 1× · see citing works
“2012) and fraudulent (Fang et al. 2012) research practices that boost the likelihood of achieving statistically significant but false effects (Simmons et al.”
BackgroundHow Professors Think: Inside the Curious World of Academic Judgment
Lamont, M. (2009) · doi:10.4159/9780674054158 · Cited by 1,086 · see citing works
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000 · see citing works
Self-citationDickersin, K., Yuan, M.-Q., & Meinert, C. (1992) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.1992.03480030052036 · Cited by 900 · see citing works
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805 · see citing works
The Division of Cognitive Labor
Kitcher, P. (1990) · The Journal of Philosophy · doi:10.2307/2026796 · Cited by 772 · see citing works
Publication bias: the case for an international registry of clinical trials.
Simes, R. J. (1986) · Journal of Clinical Oncology · doi:10.1200/jco.1986.4.10.1529 · Cited by 747 · see citing works
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · see citing works
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.4.281 · Cited by 632 · see citing works
Murray, C., & Acharya, A. K. (1997) · Journal of Health Economics · doi:10.1016/s0167-6296(97)00004-0 · Cited by 606 · see citing works
How professors think: inside the curious world of academic judgment
Lamont, M. 1. (2009) · Choice Reviews Online · doi:10.5860/choice.47-1492 · Cited by 601 · see citing works
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al‐Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 504 · see citing works
Publication Bias in Editorial Decision Making
Olson, C. M. (2002) · JAMA · doi:10.1001/jama.287.21.2825 · Cited by 395 · see citing works
Palmer, A. R. (2000) · Annual Review of Ecology and Systematics · doi:10.1146/annurev.ecolsys.31.1.441 · Cited by 385 · see citing works
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar‐Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 366 · see citing works
New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System
Travis, G., & Collins, H. M. (1991) · Science Technology & Human Values · doi:10.1177/016224399101600303 · Cited by 247 · see citing works
Changing Incentives to Publish
Franzoni, C., Scellato, G., & Stephan, P. E. (2011) · Science · doi:10.1126/science.1197286 · Cited by 235 · see citing works
Langfeldt, L. (2001) · Social Studies of Science · doi:10.1177/030631201031006002 · Cited by 207 · see citing works
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.267.3.374 · Cited by 193 · see citing works
Conservatism and risk-taking in peer review: Emerging ERC practices
Luukkonen, T. (2012) · Research Evaluation · doi:10.1093/reseval/rvs001 · Cited by 185 · see citing works
Editorial bias against replication research.
Neuliep, J. W., & Crandall, R. (1990) · Journal of social behavior and personality · Cited by 139 · see citing works
How Journal Editors View Replication Research
Madden, C. S., Easley, R. W., & Dunn, M. G. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673490 · Cited by 126 · see citing works
Peer Review: A Critical Inquiry
Shatz, D. (2004) · Medical Entomology and Zoology · Cited by 123 · see citing works
Reviewer bias against replication research.
Neuliep, J. W., & Crandall, R. (1993) · Journal of social behavior and personality · Cited by 113 · see citing works
Social Biases and Solutions for Procedural Objectivity
Lee, C. J., & Schunn, C. D. (2011) · Hypatia · doi:10.1111/j.1527-2001.2011.01178.x · Cited by 67 · see citing works
Self-citationA new approach to referees' assessments of manuscripts
Hargens, L. L., & Herting, J. R. (1990) · Social Science Research · doi:10.1016/0049-089x(90)90012-8 · Cited by 67 · see citing works
Measuring the Prevalence of Questionable Research Practices with Incentives for Truth-TellingPreprint
John, L. K., Loewenstein, G. F., & Prelec, D. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.1996631 · Cited by 56 · see citing works
A Kuhnian Critique of Psychometric Research on Peer Review
Lee, C. J. (2012) · Philosophy of Science · doi:10.1086/667841 · Cited by 42 · see citing works
Self-citationExperience with NIH Peer Review: Researchers' Cynicism and Desire for Change
Gillespie, G. W., Chubin, D., & Kurzon, G. M. (1985) · Science Technology & Human Values · doi:10.1177/016224398501000306 · Cited by 40 · see citing works
First Comprehensive Survey of NSF Applicants Focuses on Their Concerns About Proposal Review
McCullough, J. (1989) · Science Technology & Human Values · doi:10.1177/016224398901400107 · Cited by 38 · see citing works
Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and FuturePreprint
Harley, D., & Acord, S. K. (2011) · RePEc: Research Papers in Economics · Cited by 37 · see citing works
Why do funding agencies favor hypothesis testing?
Haufe, C. (2013) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2013.05.002 · Cited by 36 · see citing works
A New and Searching Look at NSF
Carter, L. J. (1979) · Science · doi:10.1126/science.451550 · Cited by 22 · see citing works
Editors’ Requests of Peer Reviewers: A Study and a Proposal
Frank, E. (1996) · Preventive Medicine · doi:10.1006/pmed.1996.0035 · Cited by 21 · see citing works
Zollman, K. (2009) · Episteme · doi:10.3366/e174236000900063x · Cited by 17 · see citing works
(2011) · PsycEXTRA Dataset · doi:10.1037/e580642011-001 · Cited by 17 · see citing works
Lal, B., Wilson, A. G., Jonas, S., Lee, E. C., Richards, A. M., & Peña, V. (2012) · Cited by 8 · see citing works
Lee, C. J. (2013) · Episteme · doi:10.1017/epi.2013.35 · Cited by 5 · see citing works
Self-citationInformation for JAMA Authors—2004
Fontanarosa, P. (2004) · JAMA · doi:10.1001/jama.291.1.107 · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 50 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257 · see citing works
Baldwin, M. (2018) · Isis · doi:10.1086/700070 · Cited by 171 · see citing works
Systemic racial disparities in funding rates at the National Science Foundation
Chen, C. Y., Kahanamoku, S. S., Tripati, A., Alegado, R. ʻ., Morris, V. R., Andrade, K., & Hosbey, J. (2022) · eLife · doi:10.7554/elife.83071 · Cited by 156 · see citing works
Langfeldt, L., Nedeva, M., Sörlin, S., & Thomas, D. A. (2019) · Minerva · doi:10.1007/s11024-019-09385-2 · Cited by 154 · see citing works
NIH peer review: Criterion scores completely account for racial disparities in overall impact scores
Erosheva, E. A., Grant, S., Chen, M.-C., Lindner, M. D., Nakamura, R. K., & Lee, C. J. (2020) · Science Advances · doi:10.1126/sciadv.aaz4868 · Cited by 125 · see citing works
Self-citationWhat do we know about grant peer review in the health sciences?
Guthrie, S., Ghiga, I., & Wooding, S. (2018) · F1000Research · doi:10.12688/f1000research.11917.2 · Cited by 102 · see citing works
Roumbanis, L. (2019) · Science Technology & Human Values · doi:10.1177/0162243918822744 · Cited by 79 · see citing works
The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications
Gallo, S. A., Sullivan, J., & Glisson, S. R. (2016) · PLoS ONE · doi:10.1371/journal.pone.0165147 · Cited by 76 · see citing works
What do we know about grant peer review in the health sciences?
Guthrie, S., Ghiga, I., & Wooding, S. (2017) · F1000Research · doi:10.12688/f1000research.11917.1 · Cited by 69 · see citing works
van Bommel, K., Rasche, A., & Spicer, A. (2022) · Organization & Environment · doi:10.1177/10860266221086617 · Cited by 51 · see citing works
Uncertainty and Risk-Taking in Science: Meaning, Measurement and Management
Franzoni, C., & Stephan, P. E. (2021) · Research Policy · doi:10.1016/j.respol.2022.104706 · Cited by 46 · see citing works
Centralized Funding and Epistemic Exploration
Avin, S. (2017) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axx059 · Cited by 37 · see citing works
Risk evaluation in peer review of grant applications
Gallo, S. A., THOMPSON, L. A., Schmaling, K. B., & Glisson, S. R. (2018) · Environment Systems & Decisions · doi:10.1007/s10669-018-9677-6 · Cited by 32 · see citing works
Peer review in funding-by-lottery: A systematic overview and expansion
Shaw, J. (2022) · Research Evaluation · doi:10.1093/reseval/rvac022 · Cited by 30 · see citing works
What should be rewarded? Gender and evaluation criteria for tenure and promotion
Castro, L. C., & Menéndez, L. S. (2021) · Journal of Informetrics · doi:10.1016/j.joi.2021.101196 · Cited by 29 · see citing works
Loss Functions, Axioms, and Peer Review
Noothigattu, R., Shah, N., & Procaccia, A. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12554 · Cited by 28 · see citing works
Uncovering Latent Biases in Text: Method and Application to Peer Review
Manzoor, E., & Shah, N. B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i6.16608 · Cited by 28 · see citing works
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · see citing works
“A number of venues have used algorithms [NSP21] to address the issue of subjectivity or commensuration bias [Lee15].”
MethodCite-seeing and reviewing: A study on citation bias in peer review
Stelmakh, I., Rastogi, C., Liu, R., Chawla, S., Echenique, F., & Shah, N. B. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283980 · Cited by 26 · see citing works
Feyerabend’s well-ordered science: how an anarchist distributes funds
Shaw, J. (2018) · Synthese · doi:10.1007/s11229-018-02026-3 · Cited by 23 · see citing works
Epistemic Inclusion as the Key to Benefiting from Cognitive Diversity in Science
Sikimić, V. (2023) · Social Epistemology · doi:10.1080/02691728.2023.2258831 · Cited by 22 · see citing works
Reimagining peer review as an expert elicitation process
Marcoci, A., Vercammen, A., Bush, M., Hamilton, D. G., Hanea, A. M., Hemming, V., Wintle, B. C., Burgman, M. A., & Fidler, F. M. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06016-0 · Cited by 22 · see citing works
Feyerabend and manufactured disagreement: reflections on expertise, consensus, and science policy
Shaw, J. (2020) · Synthese · doi:10.1007/s11229-020-02538-x · Cited by 21 · see citing works
Franzoni, C., & Stephan, P. E. (2021) · National Bureau of Economic Research · doi:10.3386/w28562 · Cited by 21 · see citing works
Peer reviews of peer reviews: A randomized controlled trial and other experiments
Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., & Shah, N. B. (2025) · PLOS ONE · doi:10.1371/journal.pone.0320444 · Cited by 18 · see citing works
“In the context of paper review, reviewers may have differing opinions about the relative importance of various criteria in determining overall quality of a paper, a phenomenon often referred to as “commensuration bias” [Lee15].”
A large scale randomized controlled trial on herding in peer-review discussions
Stelmakh, I., Rastogi, C., Shah, N. B., Singh, A., & Daum'e, H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0287443 · Cited by 26 · see citing works
Do peers share the same criteria for assessing grant applications?
Hug, S. E., & Ochsner, M. (2021) · Research Evaluation · doi:10.1093/reseval/rvab034 · Cited by 15 · see citing works
Hug, S. E. (2024) · Scientometrics · doi:10.1007/s11192-023-04915-y · Cited by 13 · see citing works
Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators’ Expertise
Dhull, K., Jecmen, S., Kothari, P. K., & Shah, N. B. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21987 · Cited by 12 · see citing works
External Tests of Peer Review Validity Via Impact Measures
Gallo, S. A., & Glisson, S. R. (2018) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2018.00022 · Cited by 12 · see citing works
Debiasing Evaluations That Are Biased by Evaluations
Wang, J., Stelmakh, I., Wei, Y., & Shah, N. B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17214 · Cited by 11 · see citing works
Does the inclusion of non-academic reviewers make any difference for grant impact panels?
Luo, J., Ma, L., & Shankar, K. (2021) · Science and Public Policy · doi:10.1093/scipol/scab046 · Cited by 10 · see citing works
Two conceptions of the sources of conservatism in scientific research
Bedessem, B. (2019) · Synthese · doi:10.1007/s11229-019-02479-0 · Cited by 10 · see citing works
A plea for minimally biased naturalistic philosophy
Polonioli, A. (2017) · Synthese · doi:10.1007/s11229-017-1628-0 · Cited by 10 · see citing works
Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study
Feliciani, T., Luo, J., & Shankar, K. (2022) · Quantitative Science Studies · doi:10.1162/qss_a_00207 · Cited by 9 · see citing works
How Citations Relate to Research Quality
Aksnes, D. W., & Langfeldt, L. (2024) · SpringerBriefs in political science · doi:10.1007/978-3-031-69580-3_6 · Cited by 9 · see citing works
Competencies of University Teachers and Changes for Working in the Knowledge Society
Močinić, S., Lazarić, L., & Gortan-Carlin, I. P. (2021) · Interdisciplinary Description of Complex Systems · doi:10.7906/indecs.20.4.9 · Cited by 9 · see citing works
Margherita, A., Elia, G., & Petti, C. (2022) · Sustainability · doi:10.3390/su14053034 · Cited by 9 · see citing works
Variability and negligence: grant peer review panels evaluating impact ex ante
Oxley, K., & Gulbrandsen, M. (2024) · Science and Public Policy · doi:10.1093/scipol/scae081 · Cited by 8 · see citing works
The Reference Class Problem for Credit Valuation in Science
Lee, C. J. (2020) · Philosophy of Science · doi:10.1086/710615 · Cited by 8 · see citing works
Self-citationShah, N. B., & Lipton, Z. C. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3383129 · Cited by 8 · see citing works
The Power of Peer Review on Transdisciplinary Discovery
Mäkinen, E. I. (2019) · Science Technology & Human Values · doi:10.1177/0162243918822741 · Cited by 8 · see citing works
On the Privacy-Utility Tradeoff in Peer-Review Data AnalysisPreprint
Ding, W., Shah, N. B., & Wang, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16385 · Cited by 8 · see citing works
Lane, J. N. (2022) · Research Policy · doi:10.1016/j.respol.2022.104707 · Cited by 7 · see citing works
Atjonen, P. (2018) · Learned Publishing · doi:10.1002/leap.1204 · Cited by 7 · see citing works
Langfeldt, L., & Scordato, L. (2016) · BIBSYS Brage (BIBSYS (Norway)) · doi:10.13140/rg.2.1.4568.2165 · Cited by 7 · see citing works
Academic capitalism and market thinking in higher education
Wieczorek, O., & Muench, R. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-818630-5.02086-8 · Cited by 7 · see citing works
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
The negativity crisis of AI ethics
Königs, P. (2025) · Synthese · doi:10.1007/s11229-025-05378-9 · Cited by 6 · see citing works
Shaw, J. (2023) · Philosophy of Science · doi:10.1017/psa.2023.35 · Cited by 6 · see citing works
Do Peer Reviews Predict Impact? Evidence from the American Sociological Review, 1978 to 1982
Teplitskiy, M., & Bakanic, V. (2016) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023116640278 · Cited by 6 · see citing works
Shah, N. B. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441660 · Cited by 6 · see citing works
Subconscious value influences on science
Elliott, K. C., Resnik, D. B., & Lipworth, W. L. (2025) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2025.102079 · Cited by 5 · see citing works
Randomisation as a tool for organisational decision-making: a debatable or debilitating proposition?
Dushnitsky, G., Eggers, J. P., Franzoni, C., & Teodoridis, F. (2023) · Industry and Innovation · doi:10.1080/13662716.2023.2281983 · Cited by 4 · see citing works
Testing for reviewer anchoring in peer review: A randomized controlled trial
Liu, R., Jecmen, S., Conitzer, V., Fang, F., & Shah, N. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301111 · Cited by 4 · see citing works
Holst, S., & Hägg, S. (2017) · F1000Research · doi:10.12688/f1000research.13030.2 · Cited by 4 · see citing works
Role, function, and expectations of research funding committees: Perspectives from committee members
Blatch-Jones, A., Boxall, C., & Meadmore, K. L. (2025) · F1000Research · doi:10.12688/f1000research.154665.3 · Cited by 3 · see citing works
The Many Dilemmas of Grant Peer Review
Langfeldt, L. (2024) · SpringerBriefs in political science · doi:10.1007/978-3-031-69580-3_5 · Cited by 3 · see citing works
Zhou, L., Zhang, R., Dai, X., Hershcovich, D., & Li, H. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114988 · Cited by 3 · see citing works
The Necessity of Commensuration Bias in Grant Peer Review
Heesen, R. (2022) · Ergo an Open Access Journal of Philosophy · doi:10.3998/ergo.2240 · Cited by 3 · see citing works
N. Lane, J., Friis, S., Cai, T., Menietti, M., Weber, G., & Guinan, E. (2024) · doi:10.2139/ssrn.4769282 · Cited by 3 · see citing works
How Values and Uncertainty Shape Scientific Advance in Peer Review
Smith, D. S., Kennard, N. N., Du, T., & McFarland, D. A. (2025) · American Sociological Review · doi:10.1177/00031224251362254 · Cited by 2 · see citing works
Problems and Prospects with the Scientonomic Workflow
Shaw, J., & Barseghyan, H. (2019) · Scientonomy Journal for the Science of Science · doi:10.33137/js.v3i0.33509 · Cited by 2 · see citing works
Implicit criteria weighting and evaluators' backgrounds in grant peer review
Franzoni, C., Broström, A., Stanaj, A., & Stephan, P. E. (2026) · Research Policy · doi:10.1016/j.respol.2026.105535 · Cited by 2 · see citing works
Calibration with Privacy in Peer Review
Ding, W., Kamath, G., Wang, W., & Shah, N. B. (2022) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit50566.2022.9834716 · Cited by 2 · see citing works
Saylor, K. W., & Joffe, S. (2023) · Journal of Medical Ethics · doi:10.1136/jme-2022-108883 · Cited by 2 · see citing works
The Participation and Motivations of Grant Peer Reviewers: A Comprehensive SurveyPreprint
Gallo, S. A., Thompson, L. A., Schmaling, K. B., & Glisson, S. R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/479816 · Cited by 2 · see citing works
Shaw, J., Pinto, M. F., & Wilholt, T. (2026) · Perspectives on Science · doi:10.1162/posc.a.584 · Cited by 2 · see citing works
Franssen, T., Borlaug, S. B., & Langfeldt, L. (2024) · Research Evaluation · doi:10.1093/reseval/rvae054 · Cited by 1 · see citing works
Bias and truth in science evaluation: a simulation model of grant review panel discussions.
Bethencourt, A. M., Luo, J., & Feliciani, T. (2021) · Cited by 1 · see citing works
Shah, N. B. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00075 · Cited by 1 · see citing works
Ranking Scientific Papers Using Preference Learning.Preprint
Dycke, N., Simpson, E., Kuznetsov, I., & Gurevych, I. (2021) · arXiv (Cornell University) · Cited by 1 · see citing works
Dunleavy, D. J. (2022) · Accountability in Research · doi:10.1080/08989621.2022.2135992 · Cited by 1 · see citing works
Scoring and Funding Breakthrough Ideas: Evidence from a Global Pharmaceutical CompanyPreprint
Krieger, J., Nanda, R., Hunt, I., Reynolds, A., & Tarsa, P. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4649716 · Cited by 1 · see citing works
Centralised Funding and the Division of Cognitive Labour
Avin, S. (2016) · PhilSci-Archive (University of Pittsburgh)
SROI in the Context of Policy and Governance Developments
Then, V., Schöber, C., Rauscher, O., & Kehl, K. (2017) · Springer eBooks · doi:10.1007/978-3-319-71401-1_3
A Pluralism Worth Having: Feyerabend's Well-Ordered Science
Shaw, J. (2018) · Scholarship@Western (Western University)
KDD 2021 Tutorial on Systemic Challenges and Solutions on Bias and Unfairness in Peer Review
Shah, N. B. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3470826
The Expectations and the Actual Framework of University Academic Ranking
Adaileh, A. M. (2021) · Academy of Information and Management Sciences journal
Цитируемость, показатели цитируемости и качество исследований: обзор базовых понятий и теорий
Aksnes, D. W., ЛАНГФЕЛТ, Л., & ВУТЕРС, П. (2021) · Международный форум по информации · doi:10.36535/0203-6460-2021-03-1
Holmes, A., & Rubin, H. (2022) · Accountability in Research · doi:10.1080/08989621.2022.2129016
Further thoughts on Bolek’s analysis of peer review reports
Yeo-Teh, N. S. L., & Tang, B. L. (2022) · Accountability in Research · doi:10.1080/08989621.2022.2143267
Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and Protection
Goldberg, A., Fanti, G. C., & Shah, N. B. (2023) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3579335
Du Bois on the Centralized Organization of Science
Bright, L. K. (2023) · Springer eBooks · doi:10.1007/978-3-031-13405-0_3
Demystifying the Academic Research Enterprise
Droegemeier, K. K. (2023) · The MIT Press eBooks · doi:10.7551/mitpress/14740.001.0001
No Agreement Without Loss: Learning and Social Choice in Peer Review
Barceló, P., Duarte, M., Rojas, C., & Steifer, T. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230270
Role, function, and expectations of research funding committees: Perspectives from committee members
Blatch-Jones, A., Boxall, C., & Meadmore, K. L. (2025) · F1000Research · doi:10.12688/f1000research.154665.2
Larue, L. (2025) · Journal of Applied Philosophy · doi:10.1111/japp.70057
أزمة التشاؤمية في حقل أخلاقيات الذكاء الاصطناعي
Königs, P. (2026) · تبين للدراسات الفكرية و الثقافية · doi:10.31430/vifv3059
Butters, A., Marshall, M. B., Pinfield, S., Stafford, T., Bondarenko, A., Neubauer, B., Nuske, R., Schwidlinski, P., & Denecke, H. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00246-7
Shaw, J., & Šešelja, D. (2026) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2026.102195
Feng, R., Xie, Y., 左元, Yang, W., Wang, N., & Wu, J. (2026) · Tertiary Education and Management · doi:10.1007/s11233-026-09185-w
When AI Helps Too Well: Transformative Inquiry and the Risk of Second-Order Drift in Ethics Research
Titz, I. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01209-y
- Authors
- Lee, C. J.
- Corresponding author
- Lee, Carole J.
- Venue
- Philosophy of Science
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-12
- Volume
- 82
- Issue
- 5
- Pages
- 1272-1283
- ISSN
- 0031-8248 (ISSN-L)
- 1539-767X
- OpenAlex
- W2273766569
- Cited by
- 90
- References
- 50
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 7 · 2023: 11 · 2022: 15 · 2021: 14 · 2020: 5 · 2019: 6 · 2018: 9 · 2017: 4 · 2016: 4
- FWCI
- 36.34
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Philosophy of Science (publishedVersion)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- peer review, peer review criteria, evaluation metrics, publication practices, research funding, science policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
