A Kuhnian Critique of Psychometric Research on Peer Review
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Psychometrically oriented researchers construe low interrater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain unclear the extent to which low interrater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Essential Tension: Selected Studies in Scientific Tradition and Change
Kuhn, T. S. (1977) · doi:10.7208/chicago/9780226217239.001.0001 · Cited by 3,861
How Professors Think: Inside the Curious World of Academic Judgment
Lamont, M. (2009) · doi:10.4159/9780674054158 · Cited by 1,086
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Self-citationPublication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898 · cited 1×
“In medicine, the interest in novelty is expressed as the preference for results of randomized, controlled trials that favor a new therapy as opposed to the standard therapy (Hopewell et al. 2009).”
BackgroundChance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623 · cited 1×
“High correlations between mean reviewer recommendations and final decisions by editors and grant panels suggest that reviewer recommendations are taken very seriously (Cole et al. 1981; Bakanic, McPhail, and Simon 1987; Marsh and Ball 1989; Hargens and Herting 2006).”
BackgroundCicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Godlee, F., Gale, C. R., & Martyn, C. N. (1998) · JAMA · doi:10.1001/jama.280.3.237 · Cited by 333
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008) · American Psychologist · doi:10.1037/0003-066x.63.3.160 · Cited by 299 · cited 2×
“Jayasinghe et al. found that the decision to fund a grant submission “was based substantially on chance” since few grant proposals were far from the cutoff value for funding when 95% confidence intervals were constructed for each proposal (Marsh et al. 2008, 162).”
BackgroundInfluentialDoes Masking Author Identity Improve Peer Review Quality?: A Randomized Controlled Trial
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & and the PEER Investigators (1998) · JAMA · doi:10.1001/jama.280.3.240 · Cited by 278
Fiske, D. W., & Fogg, L. F. (1990) · American Psychologist · doi:10.1037/0003-066x.45.5.591 · Cited by 142
Cited by
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Self-citationBoudreau, K. J., Guinan, E. C., Lakhani, K. R., & Riedl, C. (2016) · Management Science · doi:10.1287/mnsc.2015.2285 · Cited by 481
NIH peer review: Criterion scores completely account for racial disparities in overall impact scores
Erosheva, E. A., Grant, S., Chen, M.-C., Lindner, M. D., Nakamura, R. K., & Lee, C. J. (2020) · Science Advances · doi:10.1126/sciadv.aaz4868 · Cited by 125
“Note that individual differences between reviewers—reflected by the reviewer random intercept in our models—can be thought of as being due to individual differences in areas of expertise, scientific interests, and value systems (33, 34).”
BackgroundSelf-citationThe sociology of scientific validity: How professional networks shape judgement in peer review
Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., Körding, K., & Evans, J. (2018) · Research Policy · doi:10.1016/j.respol.2018.06.014 · Cited by 113
Commensuration Bias in Peer Review
Lee, C. J. (2015) · Philosophy of Science · doi:10.1086/683652 · Cited by 90
Self-citationErosheva, E. A., Martinková, P., & Lee, C. J. (2021) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12681 · Cited by 39
“Although editors and program officers may actively seek out diverse forms of expertise and opinion when evaluating submissions, which could lead to lower reliability rates but epistemically improved outcomes (Hargens & Herting, 1990; Lee, 2012), Marsh et al.”
BackgroundSelf-citationCausal Effect of Group Diversity on Redundancy and Coverage in Peer-Reviewing
Goyal, N., Stelmakh, I., Shah, N., & Daumé, H. (2024) · arXiv · doi:10.48550/arxiv.2411.11437
Metadata
- Authors
- Lee, C. J.
- Venue
- Philosophy of Science
- Publisher
- Cambridge University Press (CUP)
- Published
- 2012-12
- Volume
- 79
- Issue
- 5
- Pages
- 859-870
- ISSN
- 0031-8248 (ISSN-L)
- 1539-767X
- OpenAlex
- W2173425202
- Cited by
- 41
- References
- 11
- FWCI
- 14.66
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.