The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Bias in Research Grant Evaluation Has Dire Consequences for Small Universities
Open accessAbstract
Federal funding for basic scientific research is the cornerstone of societal progress, economy, health and well-being. There is a direct relationship between financial investment in science and a nation's scientific discoveries, making it a priority for governments to distribute public funding appropriately in support of the best science. However, research grant proposal success rate and funding level can be skewed toward certain groups of applicants, and such skew may be driven by systemic bias arising during grant proposal evaluation and scoring. Policies to best redress this problem are not well established. Here, we show that funding success and grant amounts for applications to Canada's Natural Sciences and Engineering Research Council (NSERC) Discovery Grant program (2011-2014) are consistently lower for applicants from small institutions. This pattern persists across applicant experience levels, is consistent among three criteria used to score grant proposals, and therefore is interpreted as representing systemic bias targeting applicants from small institutions. When current funding success rates are projected forward, forecasts reveal that future science funding at small schools in Canada will decline precipitously in the next decade, if skews are left uncorrected. We show that a recently-adopted pilot program to bolster success by lowering standards for select applicants from small institutions will not erase funding skew, nor will several other post-evaluation corrective measures. Rather, to support objective and robust review of grant applications, it is necessary for research councils to address evaluation skew directly, by adopting procedures such as blind review of research proposals and bibliometric assessment of performance. Such measures will be important in restoring confidence in the objectivity and fairness of science funding decisions. Likewise, small institutions can improve their research success by more strongly supporting productive researchers and developing competitive graduate programming opportunities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The globalisation of academic research has underscored the need for open-access publications and international collaborations, allowing researchers to share their findings widely and learn from one another across borders (18).”
From Research Publication Challenges Faced by Dentists in India: A Narrative Review of Multilevel Barriers (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Customised training modules for faculty can improve confidence and competence in research activities (18).”
From Research Publication Challenges Faced by Dentists in India: A Narrative Review of Multilevel Barriers (2026) · cites this paper as [18]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013) · Wiley Series in Probability and Statistics · doi:10.1002/9781118548387 · Cited by 13,861
The scientific impact of nations
King, D. A. (2004) · Nature · doi:10.1038/430311a · Cited by 1,245
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Rescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817 · cited 2×
“Further, given the contemporary job market and the glut of PhDs [30, 31], it is unlikely that small schools systematically hire weaker researchers to fill tenure-track positions, or that elite ECRs accept offers only from large institutions.”
BackgroundUniversity Research Evaluation and Funding: An International Comparison
Geuna, A., & Martin, B. R. (2003) · Minerva · doi:10.1023/b:mine.0000005155.70870.bd · Cited by 644
The Scientific Wealth of Nations
May, R. M. (1997) · Science · doi:10.1126/science.275.5301.793 · Cited by 599
Double-blind review favours increased representation of female authors
BUDDEN, A., TREGENZA, T., AARSSEN, L., KORICHEVA, J., LEIMU, R., & LORTIE, C. (2008) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2007.07.008 · Cited by 552 · cited 1×
“Blind review is becoming increasingly common as a means of ensuring objectivity for manuscripts under consideration for publication [44, 45], although a potential drawback of blind review of research grant proposals is the loss of context between applicants’ experience and their ability to successfully conduct the pro…”
BackgroundQuantitative conservation biology
Morris, W. F., & Doak, D. F. (2002) · Cited by 418
A Conceptual Framework for Ordered Logistic Regression Models
Fullerton, A. S. (2009) · Sociological Methods & Research · doi:10.1177/0049124109346162 · Cited by 397 · cited 1×
“We used ordinary logistic regression [21] to determine patterns of funding success for established researchers and ECRs based on a series of a priorimodels; differences in funding bins and funding levels and scores on each of the three evaluation criteria were examined using ordered logistic regression [22], which all…”
MethodFrom apprentice to colleague: The metamorphosis of Early Career Researchers
Laudel, G., & Gläser, J. (2007) · Higher Education · doi:10.1007/s10734-007-9063-7 · Cited by 375
Organizational and institutional influences on creativity in scientific research
Heinze, T., Philip Shapira, A., Rogers, J. D., & Senker, J. M. (2009) · Research Policy · doi:10.1016/j.respol.2009.01.014 · Cited by 369 · cited 1×
“Researchers at small schools may also have fewer local collaborators and research-related resources than their counterparts at larger schools [29].”
BackgroundMarsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008) · American Psychologist · doi:10.1037/0003-066x.63.3.160 · Cited by 299
Faculty Mentoring Programs: Reenvisioning Rather Than Reinventing the Wheel
Zellers, D. F., Howard, V., & Barcic, M. A. (2008) · Review of Educational Research · doi:10.3102/0034654308320966 · Cited by 271 · cited 1×
“Some institutions have adopted peer-mentoring policies for new researchers as an added level of support which can assist with grant proposal or manuscript editing, research direction, and helping balance conflicting time demands [46, 47].”
BackgroundNew Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System
Travis, G., & Collins, H. M. (1991) · Science Technology & Human Values · doi:10.1177/016224399101600303 · Cited by 247
Publication metrics and success on the academic job market
van Dijk, D., Manor, O., & Carey, L. B. (2014) · Current Biology · doi:10.1016/j.cub.2014.04.039 · Cited by 238
Pathogens, Nutritional Deficiency, and Climate Influences on a Declining Moose Population
Murray, D. L., Cox, E. W., Ballard, W. B., Whitlaw, H. A., Lenarz, M. S., Custer, T. W., Barnett, T. L., & Fuller, T. K. (2006) · Wildlife Monographs · doi:10.2193/0084-0173(2006)166[1:pndaci]2.0.co;2 · Cited by 236 · cited 1×
“More importantly, we conducted a sensitivity analysis by varying individual parameters across a range of values and documenting the relative contribution of applicant status to changes in funding rates according to institution size [23, 24].”
MethodSelf-citationBig names or big ideas: Do peer-review panels select the best science proposals?
Li, D., & Agha, L. (2015) · Science · doi:10.1126/science.aaa0185 · Cited by 227
Big Science vs. Little Science: How Scientific Impact Scales with Funding
Fortin, J.-M., & Currie, D. J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0065263 · Cited by 190 · cited 1×
“Their ability to conduct research and training will be further eroded unless national research councils reverse contemporary trends that concentrate grant funding in the hands of select researchers and elite universities [32, 33].”
BackgroundMarsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009) · Review of Educational Research · doi:10.3102/0034654309334143 · Cited by 174 · cited 1×
“Indeed, finding causes underlying skewed evaluation scores and funding decisions is especially challenging [13], leading to recent questions about whether evaluation skew is even as prevalent or as unjustified as was previously thought [14, 15].”
BackgroundCampbell, D. A., Picard-Aitken, M., Côté, G., Caruso, J., Valentim, R., Edmonds, S. D., Williams, G. T., Macaluso, B., Robitaille, J.-P., Bastien, N., et al. (2010) · American Journal of Evaluation · doi:10.1177/1098214009354774 · Cited by 160 · cited 5×
“NSERC Discovery Grant program NSERC is Canada’s primary university-based scientific research funder, and the Discovery Grant program is the cornerstone of basic and applied post-secondary research in the natural and engineering-related sciences.”
MethodInfluentialPeer review of grant applications: what do we know?
Wessely, S. C. (1998) · The Lancet · doi:10.1016/s0140-6736(97)11129-1 · Cited by 146 · cited 1×
“However, decisions concerning ranking and funding level of more competitive proposals can be markedly subjective and opaque, often leading to questions about the outcome of decisions and the integrity of the adjudication process [7, 8].”
BackgroundThreats to objectivity in peer review: the case of gender
Kaatz, A., Gutierrez, B., & Carnes, M. L. (2014) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2014.06.005 · Cited by 135 · cited 1×
“Indeed, finding causes underlying skewed evaluation scores and funding decisions is especially challenging [13], leading to recent questions about whether evaluation skew is even as prevalent or as unjustified as was previously thought [14, 15].”
BackgroundDetermining scientific impact using a collaboration index
Stallings, J. D., Vance, E. A., Yang, J., Vannier, M. W., Liang, J., Pang, L., Dai, L. Y., Ye, I. B., & Wang, G. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1220184110 · Cited by 127
The Scientific Competitiveness of Nations
Cimini, G., Gabrielli, A., & Labini, F. S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0113470 · Cited by 120
Mentoring at the University of Pennsylvania: Results of a Faculty Survey
Wasserstein, A. G., Quistberg, D. A., & Shea, J. A. (2007) · Journal of General Internal Medicine · doi:10.1007/s11606-006-0051-x · Cited by 119 · cited 1×
“Some institutions have adopted peer-mentoring policies for new researchers as an added level of support which can assist with grant proposal or manuscript editing, research direction, and helping balance conflicting time demands [46, 47].”
BackgroundBornmann, L., Wallon, G., & Ledin, A. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003480 · Cited by 113 · cited 1×
“These procedures usually succeed in identifying flawed or conceptually weak proposals [5, 9].”
BackgroundAnatomy of funded research in science
Ma, A., Mondragón, R. J., & Latora, V. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1513651112 · Cited by 103 · cited 1×
“Their ability to conduct research and training will be further eroded unless national research councils reverse contemporary trends that concentrate grant funding in the hands of select researchers and elite universities [32, 33].”
BackgroundPeer review for improving the quality of grant applications
Demicheli, V., & Pietrantonj, C. D. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000003.pub2 · Cited by 99 · cited 1×
“Indeed, finding causes underlying skewed evaluation scores and funding decisions is especially challenging [13], leading to recent questions about whether evaluation skew is even as prevalent or as unjustified as was previously thought [14, 15].”
BackgroundUniversity lecturer publication output: qualifications, time and confidence count
Hemmings, B., & Kay, R. (2010) · Journal of Higher Education Policy and Management · doi:10.1080/13600800903575520 · Cited by 53 · cited 1×
“Indeed, establishment of successful research programs is closely linked to the availability of time to conduct research [27, 28], which may be more limited at smaller institutions.”
BackgroundGrants Peer Review in Theory and Practice
Chubin, D. E. (1994) · Evaluation Review · doi:10.1177/0193841x9401800103 · Cited by 51
Neufeld, J., Huber, N., & Wegner, A. (2013) · Research Evaluation · doi:10.1093/reseval/rvt014 · Cited by 44
The big consequences of small biases: A simulation of peer review
Day, T. E. (2015) · Research Policy · doi:10.1016/j.respol.2015.01.006 · Cited by 42 · cited 1×
“Regardless, unjustified or unwanted grant proposal evaluation skew should be a major concern because it disadvantages individual researchers, their institutions and students, it calls into question integrity of the funding process itself, and it compromises not only fair distribution of resources but also productivity…”
BackgroundFerreira, C., Bastille‐Rousseau, G., Bennett, A. M., Ellington, E. H., Terwissen, C. V., Austin, C., Borlestean, A., Boudreau, M. R., Chan, K., Forsythe, A., et al. (2015) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12185 · Cited by 36
Self-citationvon Bartheld, C. S., Houmanfar, R. A., & Candido, A. (2015) · PeerJ · doi:10.7717/peerj.1262 · Cited by 35
The predictive power of NSF reviewers and panels
Scheiner, S. M., & Bouchie, L. M. (2013) · Frontiers in Ecology and the Environment · doi:10.1890/13.wb.017 · Cited by 25
Lee, M., Om, K., & Koh, J. (2000) · Scientometrics · doi:10.1023/a:1005636503358 · Cited by 14 · cited 1×
“Such an approach will separate an applicant’s institutional affiliation from the proposed work, and thereby should limit spurious correlation between scores for researcher excellence and proposal merit, even in a small research community where reviewers may be familiar with individual applicants [42, 43].”
BackgroundPeer review for improving the quality of grant applications
Demicheli, V., & Pietrantonj, C. D. (2003) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000003 · Cited by 12
University Entrepreneurship and Competition: The Case of the Small Universities.
Bélanger, C. H. (1989) · Canadian Journal of Higher Education · Cited by 8
Research at Small Canadian Universities
Owen, M. (1992) · Canadian Journal of Higher Education · doi:10.47678/cjhe.v22i2.183130 · Cited by 6
Life Science Graduates Face Daunting Labor Market
Carr, J. (2013) · BioScience · doi:10.1525/bio.2013.63.12.3 · Cited by 5
ADVANCES IN INFORMATION SCIENCE Bias in Peer Review
Lee, C. J., Sugimoto, C. R., Guo, Z., & Cronin, B. (2013) · Cited by 4
Research at Two Small Canadian Universities: The Views of Faculty
Furrow, D. R., & Taylor, C. (1996) · Canadian Journal of Higher Education · doi:10.47678/cjhe.v26i1.183232 · Cited by 1
Research works printed but not linked: 2
- L Bornmann, 2004, Reliability, fairness, and predictive validity of committee peer review, BIF Futura, 19, 7
- Bornmann, 2011, Improving peer review in scholarly journals, Eur Sci Edit, 37, 41
Cited by
Sorted by citations
The science of science: From the perspective of complex systems
Zeng, A., Shen, Z., Zhou, J., Wu, J., Fan, Y., Wang, Y., & Stanley, E. (2017) · Physics Reports · doi:10.1016/j.physrep.2017.10.001 · Cited by 420
“conducted a case study of the applications to Canada’s Natural Sciences and Engineering Research Council (NSERC) Discovery Grant program and showed that both the success rates and grant amounts are lower for applicants from small institutes [370].”
MethodImplicit bias in healthcare: clinical practice, research and decision making
Gopal, D. P., Chetty, U., O'Donnell, P., Gajria, C., & Blackadder-Weinstein, J. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2020-0233 · Cited by 298
Feminist scholars working around the neoliberal university
Acker, S., & Wagner, A. (2017) · Gender and Education · doi:10.1080/09540253.2017.1296117 · Cited by 106
“…and managerialism are thought to deepen the disadvantages of women and ethnic minority academics in pursuing research, as well as those in small universities or in countries on the periphery (Angervall, Beach, and Gustafsson 2015; Curtis 2016; Leberman, Eames, and Barnett 2015; Murray et al. 2016).”
BackgroundDeSci based on Web3 and DAO: a comprehensive overview and reference model
Ding, W., Hou, J., Li, J., Guo, C., Qin, J., Kozma, R., & Wang, F.-Y. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3204745 · Cited by 98
“First, research funding is biased and inefficient [10], [11], [12].”
BackgroundEntradas, M., Bauer, M. W., O’Muircheartaigh, C. A., Marcinkowski, F., Okamura, A., Pellegrini, G., Besley, J. C., Massarani, L., Russo, P., Dudo, A. D., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235191 · Cited by 133
“But the fact that the medical sciences may be more susceptible to politicisation and driven by external funding for research [48] [49], may also explain part of this phenomenon.”
Background
Metadata
- Authors
- Murray, D. L., Morris, D., Lavoie, C., Leavitt, P. R., MacIsaac, H., Masson, M. E. J., & Villard, M.-A.
- Affiliations
- Trent University (CA)
- Lakehead University (CA)
- Université Laval (CA)
- University of Regina (CA)
- Great Lakes Institute for Environmental Research (CA)
- University of Victoria (CA)
- Corresponding author
- Murray, Dennis L.
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-06-03
- Volume
- 11
- Issue
- 6
- Pages
- e0155876
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4892638
- OpenAlex
- W2412531419
- Cited by
- 87
- References
- 43
- Citations by year
- 2026: 5 · 2025: 16 · 2024: 12 · 2023: 12 · 2022: 12 · 2021: 10 · 2020: 5 · 2019: 4 · 2018: 4 · 2017: 7
- FWCI
- 9.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UVic’s Research and Learning Repository (University of Victoria) (submittedVersion)
- oURspace (University of Regina) (submittedVersion, cc-by)
- Scholarship at UWindsor (University of Windsor) (submittedVersion, cc-by-nc-nd)
- Funders
- 0 (0)
- Topics
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Political Science and International Relations › Science and Science Education
- Keywords
- research grant evaluation, systemic bias, small universities, research funding disparities, blind peer review, bibliometric assessment, research funding policy, graduate program development
- MeSH terms
- Humans, Research, Research Personnel, Research Support as Topic, Science, Universities, Program Development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
