CoChat
Sign Up
Cited by 1scientometrics and bibliometrics researchPsychology Research and BibliometricsResearch, Science, and Academia

Reliability and validity of a high-profile peer review study: Probing Wennerås and Wold’s data in Nature

Open access
Sandström, Ulf, Riis, Ulla·2025·Quantitative Science Studies
DOI 10.1162/qss.a.395
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract This article revisits the widely cited study by Wennerås and Wold (1997), which reported evidence of gender bias and nepotism in the peer review process for Swedish medical research fellowships. Their study has had substantial scholarly and policy influence, often cited as a foundational demonstration of systemic bias in science evaluation to the disadvantage of women. By accessing and reanalyzing the original data set and reconstructing the analytical model used by Wennerås and Wold, we reproduce their findings while identifying key methodological inconsistencies and overlooked complexities. Applications to program types were excluded from their analysis without justification. With the use of normalization techniques, we reassess gender disparities in reviewer scores and demonstrate that much of the reported gender bias can be attributed to disciplinary and programmatic variation rather than applicant sex per se. Moreover, we introduce a network-based analysis of applicant success rates. We conclude that while Wennerås and Wold’s original study remains a milestone in exposing bias in science, aspects of its empirical foundation merit reconsideration. Our findings contribute to ongoing debates about the reproducibility of influential social science studies and the persistent challenges in measuring fairness in research evaluation. The broader political context within Sweden is also outlined.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/qss.a.395

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. Albers, C. J. (2015). Dutch research funding, gender bias, and Simpson's paradox. Proceedings of the National Academy of Sciences of the United States of America, (December 15), 112 (50):112, E6828-E6829.
  2. Bailyn, L. (2003). Academic Careers and Gender Equity: Lessons Learned from MIT. Gender, Work & Organization, 10(2), 137-153. [DOI: 10.1111/1468-0432.00008] · DOI
  3. Bellotti, E., Czerniawska, D., Everett, M.G. & Guadalupi, L. (2022). Gender inequalities in research funding: Unequal network configurations, or unequal network returns?, Social Networks, 70: 138-151. [DOI:10.1016/j.socnet.2021.12.007]. · DOI
  4. Bornmann, L. & Daniel H.-D. (2005a). Criteria used by a peer review committee for selection of research fellows: A Boolean probit analysis. International Journal of Selection and Assessment, 13(4), 296-303.
  5. Bornmann, L. & Daniel H.-D. (2005b). Selection of research fellowship recipients by committee review. Analysis of reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics, 63(2), 297-320.
  6. Ceci, S. J. & Williams, W. M. (2011). Understanding current causes of women's underrepresentation in science. Proceedings of the National Academy of Sciences of the United States of America, 108(8), 3157-3162.
  7. Cole, J. R. (1979). Fair science: Women in the scientific community. New York: Free Press.
  8. Cruz-Castro, L., Ginther, D. & Sanz-Menéndez, L. (2022). Gender and Underrepresented Minority Differences in Research Funding. National Bureau of Economic Research Working Paper Series, No. 30107. [DOI:10.3386/w30107]. · DOI
  9. Cruz-Castro, L., & Sanz-Menéndez, L. (2020). Grant allocation disparities from a gender perspective: Literature review. Synthesis report. [DOI:10.20350/digitalCSIC/10548]. · DOI
  10. Sandström, U., and Riis, U. (2025). Reliability and Validity of a High-Profile Peer Review Study: Probing the Wennerås & Wold's Data in Nature. Quantitative Science Studies. Advance Publication. https://doi.org/10.1162/QSS.a.395 Copyright: © 2025 Ulf Sandström & Ulla Riis. Published under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. · DOI
  11. Cruz-Castro, L., & Sanz-Menéndez, L. (2023). Gender bias in funding evaluation: A randomized experiment. Quantitative Science Studies, 4(3), 594-621. [DOI:10.1162/qss_a_00263]. Elsevier (2020). The Researcher Journey Through a Gender Lens. [https://www.elsevier.com/__data/assets/pdf_file/0074/1075373/ElsevierGenderReport2020_FINAL.pdf] · DOI
  12. final. Brussels. European Commission (2009). PRAGES guidelines: Guidelines for Gender Equality Programmes in Science. European Commission (2012). Directorate-General for Research and Innovation, Structural change in research institutions: Enhancing excellence, gender equality and efficiency in research and innovation, Publications Office, 2012, [DOI:10.2777/37288] · DOI
  13. Fox, M. F. (1991). Gender, environmental milieu, and productivity in science. In H. Zuckerman, J. R. Cole, & J. T. Bruer (Eds.), The outer circle: Women in the scientific community (pp. 188-204). New York: W. W. Norton.
  14. Grant, J., Burden, S. & Breen, G. (1997). No evidence of sexism in peer review. Nature 390, 6659, 438-438.
  15. Hyland, K. (1996). Writing without conviction? Hedging in science research articles. Applied Linguistics 17 (4): 433- 454.
  16. Hyland, K. (1998). Boosting, hedging and the negotiation of academic knowledge. Text, 18(3): 349-382.
  17. Knobloch-Westerwick, S., Glynn, C. J., & Huge, M. (2013). The Matilda effect in science communication. Science Communication, 35(5), 603-625. [DOI:10.1177/1075547012472684] · DOI · Paper page
  18. Larregue, J., & Nielsen, M. W. (2024). Knowledge Hierarchies and Gender Disparities in Social Science Funding. Sociology, 58(1), 45-65. [DOI:10.1177/00380385231163071]. · DOI
  19. McAllister, P. R., Narin, F. & Corrigan, J. G. (1983). Programmatic evaluation and comparison based on standardized citation scores. IEEE Transactions on Engineering Management, vol. EM-30, 205-211.
  20. Moed, M. & van Raan, A. F. J. (1988). Indicators of research performance: Applications in university research policy. In: van Raan (Ed.) Handbook of Quantitative Studies of Science and Technology. Amsterdam: North-Holland, 177-206.
  21. Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L. & Handelsman, J. (2012). Science faculty's subtle gender biases favor male students. Proceedings of the National Academy of Sciences of the United States of America, 109(41), 16474- 16479. [DOI:10.1073/pnas.1211286109] · DOI · Paper page
  22. NASEM 2006) National Academies of Sciences, Engineering, and Medicine. (2006). Biological, social, and organizational components of success for women in academic science and engineering: Workshop report. National Academies Press. (NASEM 2007) National Academies of Sciences, National Academy of Engineering & Institute of Medicine. (2007). Beyond bias and barriers: Fulfilling the potential of women in academic science and engineering. National Academies Press. New Scientist: Editorial: 'Naming the game'. (1997, May 24).
  23. Sandström, U., and Riis, U. (2025). Reliability and Validity of a High-Profile Peer Review Study: Probing the Wennerås & Wold's Data in Nature. Quantitative Science Studies. Advance Publication. https://doi.org/10.1162/QSS.a.395 Copyright: © 2025 Ulf Sandström & Ulla Riis. Published under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. (NRC 2000). National Research Council. (2001). The advancement of women in science and engineering. National Academies Press. · DOI
  24. Ranga, M., Gupta, N., & Etzkowitz, H. (2012). Gender effects in research funding: A review of the scientific discussion on the gender-specific aspects of the evaluation of funding applications and the award of funding. Deutsche Forschungsgemeinschaft (DFG), Bonn.
  25. Reskin, B. F. (2003). Rethinking Employment Discrimination and Its Remedies. In D. B. Grusky (Ed.), Social Stratification: Class, Race, and Gender in Sociological Perspective. Westview Press.
  26. Rossiter, M. W. (1993). The Matthew/Matilda effect in science. Social Studies of Science 23 (2): 325-341.
  27. Sandström, U. & Hällsten, M. (2007). Bibliometrics and gender: Methods make a difference. Paper for the pre- conference at ISSI conference in Madrid 2007. (Unpublished)
  28. Schubert, A., Glänzel, W. & Braun, T. (1988). Against absolute methods: Relative scientometric indicators and relational charts as evaluation tools. In: van Raan (Ed.) Handbook of Quantitative Studies of Science and Technology. Amsterdam: North-Holland, 137-176.
  29. Shen, H. (2013). Inequality quantified: Mind the gender gap. Nature, 495(7439), 22-24. [DOI:/10.1038/495022a] · DOI
  30. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society Series B-Statistical Methodology, 13 (2), 238-241.
  31. Traag, V. A. & Waltman, L. (2024). Causal foundations of bias, disparity and fairness. arXiv:2207.13665v3 [cs.DL] 19 Jul 2024. · arXiv
  32. Van den Besselaar, P. & Sandström, U. (2019). Measuring researcher independence using bibliometric data: A proposal for a new performance indicator. PLoS ONE 14(3): e0202712. [DOI:10.1371/journal.pone.0202712]. · DOI
  33. Waisbren, S. E., Bowles, H. R., Hasan, T. & Zou, K. H. (2008). Gender differences in research grant applications and funding outcomes for medical school faculty. Journal of Women's Health 17(2):207-214.
  34. Wennerås, C. & Wold, A. (1996). Peer review favors males. Department of Clinical Immunology, University of Gothenburg. Mimeograph. Report to the Ministry of Education.
  35. Wennerås, C. & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387, 341-343.
  36. Wessely, S. (1998). Peer review of grant applications: What do we know? Lancet, 352 (9124), 301-305.
  37. Witteman, H. O., Hendricks, M., Straus, S., and Tannenbaum, C. (2019). Are gender gaps due to evaluations of the applicant or the science? A natural experiment at a national funding agency. The Lancet, 393(10171), 531-540. [DOI: 10.1016/S0140-6736(18)32611-4]. · DOI

Cited by

Sorted by citations

Authors
Sandström, U., & Riis, U.
Affiliations
  • KTH Royal Institute of Technology (SE)
  • Uppsala University (SE)
Corresponding author
Sandström, Ulf
ORCID
Venue
Quantitative Science Studies
Publisher
MIT Press
Published
2025
Volume
6
Pages
1315-1335
ISSN
2641-3337 (ISSN-L)
DOI
10.1162/qss.a.395
OpenAlex
W4415517725
Cited by
0
References
2
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Quantitative Science Studies (publishedVersion, cc-by)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
Funders
EU H2020 (GA-824574)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
Keywords
peer review, gender bias, nepotism, research evaluation, medical research fellowships, reproducibility, reanalysis, disciplinary variation, program type, women in science

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review