The DOI matches its Crossref record; 88 of the 90 references it cites were found in a registry or an index.
Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches
Abstract
Peer review is valued in higher education, but also widely criticized in terms of potential biases, particularly gender. We evaluate gender differences in peer reviews of grant applications, extending Bornmann, Mutz, and Daniel’s meta-analyses that reported small gender differences in favor of men ( d = .04), but a substantial heterogeneity in effect sizes that compromised the robustness of their results. We contrast these findings with the most comprehensive single primary study (Marsh, Jayasinghe, and Bond) that found no gender differences for grant proposals. We juxtapose traditional (fixed- and random-effects) and multilevel models, demonstrating important advantages to the multilevel approach. Consistent with Marsh et al.’s primary study, there were no gender differences for the 40 (of 66) effect sizes from Bornmann et al. that were based on grant proposals. This lack of a gender effect for grant proposals was very robust, generalizing over country, discipline, and publication year
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In peer review research, previously published meta-analyses investigated only gender differences in the selection process of grant proposals [12], [13].”
From A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants (2010) · cites this paper as [13]
In the citing paper
“though simulation evidence suggests that this may not have a large impact on the estimation of intervention effects.99”
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 13,786
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
(2000) · Cited by 9,198
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720
(1968) · PubMed · Cited by 4,803
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,784
Statistical Analysis With Missing Data
Ghosh, S. (1988) · Technometrics · doi:10.1080/00401706.1988.10488446 · Cited by 4,576
The Sociology of Science: Theoretical and Empirical Investigations
Kaplan, N., Merton, R. K., & Storer, N. W. (1974) · Technology and Culture · doi:10.2307/3102264 · Cited by 4,005
Research works printed but not linked: 8
- Cicchetti DV, 1979, Paper presented at the 1979 Proceedings of the Social Statistics Section
- Crane D, 1967, American Sociologist, 32, 195
- Hox JJ, 2003, Multilevel modelling: Methodological advances, issues, and applications, 90
- Mahoney MJ, 1978, Human Behaviour, 7, 38
- Mahoney MJ, 1987, Journal of Social Behaviour and Personality, 2, 165
- Marsh HW, 2008, Peer reviews of grant applications: Effects of applicant and assessor gender
- Rabe-Hesketh S, 2004, GLLAMM manual
- StataCorp, 2005, Stata statistical software: Release 9 [computer software]
Cited by
Cheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342
“The second data set was based on Bornmann et al. (2007) and Marsh et al. (2009).”
BackgroundMethodGender differences in scholastic achievement: A meta-analysis.
Voyer, D., & Voyer, S. D. (2014) · Psychological Bulletin · doi:10.1037/a0036620 · Cited by 1,337
Gender Similarities and Differences
Hyde, J. S. (2014) · Annual Review of Psychology · doi:10.1146/annurev-psych-010213-115057 · Cited by 1,262
Understanding current causes of women's underrepresentation in science
Ceci, S. J., & Williams, W. M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014871108 · Cited by 1,114
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130
Metadata
- Authors
- Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A.
- Affiliations
- University of Freiburg (DE)
- University of Oxford
- Venue
- Review of Educational Research
- Published
- 2009-09
- Volume
- 79
- Issue
- 3
- Pages
- 1290-1326
- ISSN
- 0034-6543 (ISSN-L)
- 1935-1046
- OpenAlex
- W2133561842
- Semantic Scholar
- bdfdb0de6e512a0c203f85789db1811c206c3338
- Cited by
- 174
- References
- 90
- Citations by year
- 2026: 3 · 2025: 15 · 2024: 12 · 2023: 9 · 2022: 13 · 2021: 7 · 2020: 13 · 2019: 8 · 2018: 7 · 2017: 10 · 2016: 10 · 2015: 14 · 2014: 13 · 2013: 10 · 2012: 12
- FWCI
- 12.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of Educational Research (publishedVersion)
- Research Bank (Australian Catholic University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Economic and Social Research Council (ES/F041292/1)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- gender bias, peer review, grant applications, research funding, multilevel models, fixed and random effects models, effect size heterogeneity, research evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches. https://app.cochat.ai/papers/10.3102/0034654309334143
- The paper
- Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009). Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches. Review of Educational Research. https://doi.org/10.3102/0034654309334143
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.