Panel discussion does not improve reliability of peer review for medical research grant proposals.
Abstract
Objective Peer review is the gold standard for evaluating scientific quality. Compared with studies on inter-reviewer variability, research on panel evaluation is scarce. To appraise the reliability of panel evaluations in grant review, we compared scores by two expert panels reviewing the same grant proposals. Our main interest was to evaluate whether panel discussion improves reliability. Methods Thirty reviewers were randomly allocated to one of the two panels. Sixty-five grant proposals in the fields of clinical medicine and epidemiology were reviewed by both panels. All reviewers received 5-12 proposals. Each proposal was evaluated by two reviewers, using a six-point scale. The reliability of reviewer and panel scores was evaluated using Cohen's kappa with linear weighting. In addition, reliability was also evaluated for the panel mean scores (mean of reviewer scores was used as panel score). Results The proportion of large differences (at least two points) was 40% for reviewers in panel A, 36% for reviewers in panel B, 26% for the panel discussion scores, and 14% when the means of the two reviewer scores were used. The kappa for panel score after discussion was 0.23 (95% confidence interval: 0.08, 0.39). By using the mean of the reviewer scores, the panel coefficient was similarly 0.23 (0.00, 0.46). Conclusion The reliability between panel scores was higher than between reviewer scores. The similar interpanel reliability, when using the final panel score or the mean val…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Low inter-reviewer agreement is not primarily a consequence of reviewer incompetence but reflects the inherently multidimensional and judgement-based nature of grant assessment.
“More recent evidence suggests that low inter-reviewer agreement is not primarily a consequence of reviewer incompetence but rather reflects the inherently multidimensional and judgement-based nature of grant assessment.”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Fogelholm et al., 2012
Cites
Chance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623 · cited 4× · see citing works
“These differences mean that the fate of a particular manuscript or proposal will partly depend on the reviewers selected [7,8].”
BackgroundResultInfluential
Flagged first, then by citations
Cited by
Sorted by citations
Low agreement among reviewers evaluating the same NIH grant applications
Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714379115 · Cited by 210 · see citing works
Beygelzimer, A., Dauphin, Y. N., Liang, P., & Vaughan, J. W. (2023) · arXiv · doi:10.48550/arxiv.2306.03262 · Cited by 2 · see citing works
Sandström, U., & Thelwall, M. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0048 · Cited by 1 · see citing works
- Authors
- Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A., & Väänänen, K.
- Affiliations
- Research Council of Finland (FI)
- University of Tampere (FI)
- University of Eastern Finland (FI)
- Corresponding author
- Fogelholm, Mikael
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2012-01
- Volume
- 65
- Issue
- 1
- Pages
- 47-52
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1982340630
- Cited by
- 83
- References
- 1
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 7 · 2023: 6 · 2022: 6 · 2021: 6 · 2020: 8 · 2019: 7 · 2018: 6 · 2017: 9 · 2016: 5 · 2015: 10 · 2014: 1 · 2013: 1 · 2012: 2
- FWCI
- 1.27
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Journal of Clinical Epidemiology
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- peer review, grant proposal review, panel discussion, health research funding, clinical medicine
- MeSH terms
- Algorithms, Communication, Financing, Organized, Finland, Humans, Research Support as Topic, Reproducibility of Results, Peer Review, Research, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
