Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches
The DOI matches its Crossref record; 88 of the 90 references it cites were found in a registry or an index.
Abstract
Peer review is valued in higher education, but also widely criticized in terms of potential biases, particularly gender. We evaluate gender differences in peer reviews of grant applications, extending Bornmann, Mutz, and Daniel’s meta-analyses that reported small gender differences in favor of men ( d = .04), but a substantial heterogeneity in effect sizes that compromised the robustness of their results. We contrast these findings with the most comprehensive single primary study (Marsh, Jayasinghe, and Bond) that found no gender differences for grant proposals. We juxtapose traditional (fixed- and random-effects) and multilevel models, demonstrating important advantages to the multilevel approach. Consistent with Marsh et al.’s primary study, there were no gender differences for the 40 (of 66) effect sizes from Bornmann et al. that were based on grant proposals. This lack of a gender effect for grant proposals was very robust, generalizing over country, discipline, and publication year
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In peer review research, previously published meta-analyses investigated only gender differences in the selection process of grant proposals [12], [13].”
From A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants (2010) · cites this paper as [13]
In the citing paper
“though simulation evidence suggests that this may not have a large impact on the estimation of intervention effects.99”
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 13,906
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
(2000) · Cited by 9,198
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720
(1968) · PubMed · Cited by 4,803
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,784
Statistical Analysis With Missing Data
Ghosh, S. (1988) · Technometrics · doi:10.1080/00401706.1988.10488446 · Cited by 4,576
The Sociology of Science: Theoretical and Empirical Investigations
Kaplan, N., Merton, R. K., & Storer, N. W. (1974) · Technology and Culture · doi:10.2307/3102264 · Cited by 4,005
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
The gender similarities hypothesis.
Hyde, J. S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.6.581 · Cited by 3,084
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
Structural Equation Modeling: A Second Course
(2007) · Personnel Psychology · doi:10.1111/j.1744-6570.2007.00081_13.x · Cited by 2,825
Hox, J., Moerbeek, M., & van de Schoot, R. (2002) · doi:10.4324/9781410604118 · Cited by 2,419
THE SOCIOLOGY OF SCIENCE:: Theoretical and Empirical Investigations
Merton, R. Κ., & Storer, N. W. (2017) · Cited by 1,462
Nepotism and sexism in peer-review
Wennerås, C., & Wold, A. (1997) · Nature · doi:10.1038/387341a0 · Cited by 1,345
Goulias, K. (2002) · New Directions in Civil Engineering · doi:10.1201/9781420042283.ch9 · Cited by 1,270
Journal of the Royal Statistical Society: Series C (Applied Statistics)
(2023) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/(issn)1467-9876 · Cited by 1,178
Hox, J. J. (1995) · Utrecht University Repository (Utrecht University) · Cited by 1,168
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 927
Zuckerman, H., & Merton, R. K. (1971) · Minerva · doi:10.1007/bf01553188 · Cited by 789
Users' Guides to the Medical Literature: VI. How to Use an Overview
Oxman, A. D. (1994) · JAMA · doi:10.1001/jama.1994.03520170077040 · Cited by 775
Integrating Research: A Guide for Literature Reviews
Kolbe, R. H., & Cooper, H. M. (1991) · Journal of Marketing Research · doi:10.2307/3172876 · Cited by 594
Multilevel Statistical Models (3rd ed.)
Lethonen, R. (2005) · Quarterly Publications of the American Statistical Association · Cited by 544
How meta-analysis increases statistical power.
Cohn, L. D., & Becker, B. J. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.243 · Cited by 498
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Users' Guides to the Medical Literature: I. How to Get Started
Oxman, A. D. (1993) · JAMA · doi:10.1001/jama.1993.03510170083036 · Cited by 469
O'Mara, A. J., Marsh, H. W., Craven, R. G., & Debus, R. L. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4103_4 · Cited by 466
The Outer circle: women in the scientific community
(1992) · Choice Reviews Online · doi:10.5860/choice.29-3860 · Cited by 391
Multilevel Analysis of Educational Data
(1989) · Elsevier eBooks · doi:10.1016/c2009-0-21681-3 · Cited by 359
Peer review in health sciences
Taraprasad, D. (2002) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 350
Gender differences in grant peer review: A meta-analysis
BORNMANN, L., MUTZ, R., & DANIEL, H. (2007) · Journal of Informetrics · doi:10.1016/j.joi.2007.03.001 · Cited by 337
Self-citationRaudenbush, S. W. (1984) · Journal of Educational Psychology · doi:10.1037/0022-0663.76.1.85 · Cited by 302 · cited 1×
“Although a multilevel approach to meta-analysis was first introduced as early as 1984 (Raudenbush, 1984; also see Goldstein, Yang, Omar, Turner & Thompson, 2000), there have been very few applications of this approach in actual practice, and almost none that have used a three-level approach rather than a two-level app…”
MethodMarsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008) · American Psychologist · doi:10.1037/0003-066x.63.3.160 · Cited by 299
Self-citationFair Science: Women in the Scientific Community
Ekstrom, R. B. (1981) · The Journal of Higher Education · doi:10.1080/00221546.1981.11780117 · Cited by 291
Entrance into the Academic Career
Long, J. S., Allison, P. D., & McGinnis, R. (1979) · American Sociological Review · doi:10.2307/2094529 · Cited by 230
A multivariate mixed linear model for meta-analysis.
Kalaian, H. A., & Raudenbush, S. W. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.3.227 · Cited by 229
Marsh, H. W., & Hau, K.-T. (2007) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2006.10.008 · Cited by 225
Multilevel Meta-analysis: a comparison with traditional meta-analytical procedures
Van Den Noortgate, W., & Onghena, P. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164403251027 · Cited by 218
Stratification in American Science*
Zuckerman, H. (1970) · Sociological Inquiry · doi:10.1111/j.1475-682x.1970.tb01010.x · Cited by 209
Marsh, H. W., Martin, A. J., & Cheng, J. H. S. (2008) · Journal of Educational Psychology · doi:10.1037/0022-0663.100.1.78 · Cited by 180
Persistent nepotism in peer-review
Sandström, U., & Hällsten, M. (2008) · Scientometrics · doi:10.1007/s11192-008-0211-3 · Cited by 178
The Psychology of Written Communication
Winne, P. H., & Hartley, J. (1981) · Canadian Journal of Education / Revue canadienne de l'éducation · doi:10.2307/1494717 · Cited by 170
Bateman, I. J., & Jones, A. P. (2007) · The Economics of Non-Market Goods and Resources · doi:10.1007/1-4020-5405-x_8 · Cited by 169
EREZ, A., BLOOM, M. C., & WELLS, M. T. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01801.x · Cited by 161
Egger, M., Ebrahim, S., & Smith, G. D. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.1 · Cited by 160
Improving manuscript evaluation procedures.
Bowen, D. D., Perloff, R., & Jacoby, J. (1972) · American Psychologist · doi:10.1037/h0033849 · Cited by 159
Bateman, I. J., & Jones, A. P. (2003) · Land Economics · doi:10.2307/3146869 · Cited by 158
Gender bias in the refereeing process?
Tregenza, T. (2002) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(02)02545-4 · Cited by 156
Bornmann, L., & Daniel, H. (2005) · Scientometrics · doi:10.1007/s11192-005-0214-2 · Cited by 146
Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2003) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/1467-985x.00278 · Cited by 139
Self-citationMeta-Analysis Using Multilevel Models with an Application to the Study of Class Size Effects
Goldstein, H., Yang, M., Omar, R., Turner, R., & Thompson, S. (2000) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/1467-9876.00200 · Cited by 135
Is there gender bias in JAMA's peer review process?
Gilbert, J. R. (1994) · JAMA · doi:10.1001/jama.272.2.139 · Cited by 134
Open exchange and epistemic progress.
Mahoney, M. J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.1.29 · Cited by 131
Is There Gender Bias in JAMA's Peer Review Process?
Gilbert, J. R. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020065018 · Cited by 128
Marsh, H. W., & Ball, S. (1989) · The Journal of Experimental Education · doi:10.1080/00220973.1989.10806503 · Cited by 120
Dependent Effect Sizes in Meta-Analysis: Incorporating the Degree of Interdependence.
Cheung, S. F., & Chan, D. K.-S. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.5.780 · Cited by 117
Bornmann, L., Wallon, G., & Ledin, A. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003480 · Cited by 111
Marsh, H. W., Wen, Z., Hau, K.-T., Little, T. D., Bovaird, J. A., & Widaman, K. F. (2007) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705510701303921 · Cited by 111
Fair Science: Women in the Scientific Community.
Rosenfeld, R. A., & Cole, J. R. (1982) · Social Forces · doi:10.2307/2578104 · Cited by 102
The Academic Marketplace Revisited: A Study of Faculty Mobility Using the Cartter Ratings
Crane, D. (1970) · American Journal of Sociology · doi:10.1086/224848 · Cited by 101
Peer review for improving the quality of grant applications
Demicheli, V., & Di Pietrantonj, C. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000003.pub2 · Cited by 99
Evidence on peer review—scientific quality control or smokescreen?
Goldbeck-Wood, S. (1999) · BMJ · doi:10.1136/bmj.318.7175.44 · Cited by 97
FRANCK, L. S. (1990) · Nursing Research · doi:10.1097/00006199-199011000-00015 · Cited by 94
Interjudgmental reliability of reviews for the Journal of Educational Psychology.
Marsh, H. W., & Ball, S. (1981) · Journal of Educational Psychology · doi:10.1037/0022-0663.73.6.872 · Cited by 89
Multilevel Models for Meta-Analysis
(2003) · Multilevel Modeling · doi:10.4324/9781410606747-9 · Cited by 88
(2009) · Introduction to Meta‐Analysis · doi:10.1002/9780470743386.ch12 · Cited by 85
Bornmann, L., & Daniel, H. (2008) · Angewandte Chemie International Edition · doi:10.1002/anie.200800513 · Cited by 70
Renin-angiotensin system dependence of renal hemodynamics in mice.
Traynor, T. R., & Schnermann, J. B. (1999) · PubMed · Cited by 66
The Gendered Nature of Assessment Procedures in Scientific Research Funding: The Dutch Case
Brouns, M. L. M. (2000) · Higher Education in Europe · doi:10.1080/713669261 · Cited by 63
Peer review in the funding of research in higher education: The Australian experience
Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2001) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737023004343 · Cited by 58
Self-citationGrants Peer Review in Theory and Practice
Chubin, D. E. (1994) · Evaluation Review · doi:10.1177/0193841x9401800103 · Cited by 51
Dependent Correlations in Meta-Analysis: The Case of Heterogeneous Dependence
Cheung, S. F., & Chan, D. K.-S. (2008) · Educational and Psychological Measurement · doi:10.1177/0013164408315263 · Cited by 45
Bornmann, L., & Daniel, H. (2006) · Journal of Informetrics · doi:10.1016/j.joi.2006.09.005 · Cited by 44
Wennerås, C., & Wold, A. E. (2000) · Nature · doi:10.1038/35047182 · Cited by 38
A new reader trial approach to peer review in funding research grants: An Australian experiment
Jayasinghe, U. W., Marsh, H. W., & Bond, N. W. (2006) · Scientometrics · doi:10.1007/s11192-006-0171-4 · Cited by 38
Marsh, H. W., Bond, N. W., & Jayasinghe, U. W. (2007) · Australian Psychologist · doi:10.1080/00050060600823275 · Cited by 35
Self-citationPeer Review of Grant Applications: A Systematic Review
WOOD, F. M., & Wessely, S. C. (2003) · RUNE (Research UNE) · Cited by 33
Reflections on the peer review process
Marsh, H. W., & Ball, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065912 · Cited by 18
Bentley, J. T., & Adamson, R. (2003) · Cited by 15
Manwell, C., & Baker, C. M. A. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011493 · Cited by 15
Bornmann, L. (2007) · Nature · doi:10.1038/nj7127-566a · Cited by 13
Peer review in the assessment and funding of research by the Australian Research Council
Jayasinghe, U. W. (2003) · Cited by 12
"STRANGERS IN A STRANGE LAND": A LITERATURE REVIEW OF WOMEN IN SCIENCE
Sheridan, B. (1998) · Cited by 8
Rowe, J. (2008) · Cited by 4
A simple method for converting an odds ratio to effect size for use in meta-analysis
Chinn, S. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001130)19:22<3127::aid-sim784>3.0.co;2-m
Research works printed but not linked: 8
- Cicchetti DV, 1979, Paper presented at the 1979 Proceedings of the Social Statistics Section
- Crane D, 1967, American Sociologist, 32, 195
- Hox JJ, 2003, Multilevel modelling: Methodological advances, issues, and applications, 90
- Mahoney MJ, 1978, Human Behaviour, 7, 38
- Mahoney MJ, 1987, Journal of Social Behaviour and Personality, 2, 165
- Marsh HW, 2008, Peer reviews of grant applications: Effects of applicant and assessor gender
- Rabe-Hesketh S, 2004, GLLAMM manual
- StataCorp, 2005, Stata statistical software: Release 9 [computer software]
Cited by
Cheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342
“The second data set was based on Bornmann et al. (2007) and Marsh et al. (2009).”
BackgroundMethodGender differences in scholastic achievement: A meta-analysis.
Voyer, D., & Voyer, S. D. (2014) · Psychological Bulletin · doi:10.1037/a0036620 · Cited by 1,337
Gender Similarities and Differences
Hyde, J. S. (2014) · Annual Review of Psychology · doi:10.1146/annurev-psych-010213-115057 · Cited by 1,262
Understanding current causes of women's underrepresentation in science
Ceci, S. J., & Williams, W. M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014871108 · Cited by 1,114
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130
Metadata
- Authors
- Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A.
- Affiliations
- University of Freiburg (DE)
- University of Oxford
- Venue
- Review of Educational Research
- Published
- 2009-09
- Volume
- 79
- Issue
- 3
- Pages
- 1290-1326
- ISSN
- 0034-6543 (ISSN-L)
- 1935-1046
- OpenAlex
- W2133561842
- Semantic Scholar
- bdfdb0de6e512a0c203f85789db1811c206c3338
- Cited by
- 174
- References
- 90
- Citations by year
- 2026: 3 · 2025: 15 · 2024: 12 · 2023: 9 · 2022: 13 · 2021: 7 · 2020: 13 · 2019: 8 · 2018: 7 · 2017: 10 · 2016: 10 · 2015: 14 · 2014: 13 · 2013: 10 · 2012: 12
- FWCI
- 12.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of Educational Research (publishedVersion)
- Research Bank (Australian Catholic University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Economic and Social Research Council (ES/F041292/1)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- gender bias, peer review, grant applications, research funding, multilevel models, fixed and random effects models, effect size heterogeneity, research evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.