Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant's future performance, the results show a statistically significant association between selection decisions and the applicants' scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regression Analysis of Count Data
Cameron, A. C., & Trivedi, P. K. (2013) · doi:10.1017/cbo9781139013567 · Cited by 6,640
The Increasing Dominance of Teams in Production of Knowledge
Wuchty, S., Jones, B. F., & Uzzi, B. (2007) · Science · doi:10.1126/science.1136099 · Cited by 3,615
Econometric Models for Count Data with an Application to the Patents-R & D Relationship
Hausman, J., Hall, B. H., & Griliches, Z. (1984) · Econometrica · doi:10.2307/1911191 · Cited by 3,393
What do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · cited 1×
“Citation counts has been a controversial measure of both quality and scientific progress”
BackgroundSelf-citationChance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623 · cited 1×
“Cole and his colleagues [8] concluded in their highly influential study on grant peer review at the National Science Foundation (NSF, Arlington, VA, USA) that ‘‘the fate of a particular application is roughly half determined by the characteristics of the proposal and the principal investigator, and about half by appar…”
BackgroundMarsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008) · American Psychologist · doi:10.1037/0003-066x.63.3.160 · Cited by 299
Competition and Careers in Biosciences
Freeman, R., Weinstein, E., Marincola, E., Rosenbaum, J., & Solomon, F. (2001) · Science · doi:10.1126/science.1067477 · Cited by 105 · cited 1×
“. If the rejection rate is low, there is less risk of under-estimation, i.e. type II error. In contrast, if the approval rate is low, only few approvals are at the risk of being over-estimated, i.e. type I error. Due to scarce financial resources on one side and a large number of applicants on the other side, the pres…”
BackgroundStatistical analysis of the national institutes of health peer review system
Johnson, V. E. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0804538105 · Cited by 54
Research on research evaluation
Hemlin, S. (1996) · Social Epistemology · doi:10.1080/02691729608578815 · Cited by 48
Cited by
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567
Self-citationMarsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009) · Review of Educational Research · doi:10.3102/0034654309334143 · Cited by 174
Interdisciplinarity in the environmental sciences: barriers and frontiers
HICKS, C. C., FITZSIMMONS, C., & POLUNIN, N. V. C. (2010) · Environmental Conservation · doi:10.1017/s0376892910000822 · Cited by 109
Bias in Research Grant Evaluation Has Dire Consequences for Small Universities
Murray, D. L., Morris, D., Lavoie, C., Leavitt, P. R., MacIsaac, H., Masson, M. E. J., & Villard, M.-A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155876 · Cited by 87
“These procedures usually succeed in identifying flawed or conceptually weak proposals [5, 9].”
Background
Metadata
- Authors
- Bornmann, L., Wallon, G., & Ledin, A.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2008-10-22
- Volume
- 3
- Issue
- 10
- Pages
- e3480
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2016553928
- Cited by
- 111
- References
- 9
- FWCI
- 6.74
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes. https://app.cochat.ai/papers/10.1371/journal.pone.0003480
- The paper
- Bornmann, L., Wallon, G., & Ledin, A. (2008). Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes. PLoS ONE. https://doi.org/10.1371/journal.pone.0003480
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.