CoChat
Sign Up
PreprintCited by 1scientometrics and bibliometrics researchPublishing and Scholarly CommunicationPsychology Research and Bibliometrics

Causal Effect of Group Diversity on Redundancy and Coverage in Peer-Reviewing

Open access
Goyal, Navita, Stelmakh, Ivan, Shah, Nihar, Daumé, Hal·2024·arXiv
DOI 10.48550/arxiv.2411.11437arXiv 2411.11437
Open this paper in CoChat — free accountOpen PDF

Abstract

A large host of scientific journals and conferences solicit peer reviews from multiple reviewers for the same submission, aiming to gather a broader range of perspectives and mitigate individual biases. In this work, we reflect on the role of diversity in the slate of reviewers assigned to evaluate a submitted paper as a factor in diversifying perspectives and improving the utility of the peer-review process. We propose two measures for assessing review utility: review coverage -- reviews should cover most contents of the paper -- and review redundancy -- reviews should add information not already present in other reviews. We hypothesize that reviews from diverse reviewers will exhibit high coverage and low redundancy. We conduct a causal study of different measures of reviewer diversity on review coverage and redundancy using observational data from a peer-reviewed conference with approximately 5,000 submitted papers. Our study reveals disparate effects of different diversity measures on review coverage and redundancy. Our study finds that assigning a group of reviewers that are topically diverse, have different seniority levels, or have distinct publication networks leads to broader coverage of the paper or review criteria, but we find no evidence of an increase in coverage for reviewer slates with reviewers from diverse organizations or geographical locations. Reviewers from different organizations, seniority levels, topics, or publications networks (all except geographical diversity) lead to a decrease in redundancy in reviews. Furthermore, publication network-based diversity alone also helps bring in varying perspectives (that is, low redundancy), even within specific review criteria. Our study adopts a group decision-making perspective for reviewer assignments in peer review and suggests dimensions of diversity that can help guide the reviewer assignment process.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.11437

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multi-reviewer peer review draws much of its value from the diversity of perspectives that different reviewers bring to a manuscript.

    “The motivation is that multi-reviewer peer review draws much of its value from the diversity of perspectives that different reviewers bring to a manuscript [Page, 2008]: assigning a slate of reviewers with diverse expertise both improves coverage of the manuscript and reduces redundancy across reviews [Goyal et al., 2024].”

    From On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists (2026) · cites this paper as Goyal et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Argument Mining for Understanding Peer Reviews

    Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 3× · see citing works

    “…being commented upon, namely, Summary, Motivation/Impact, Originality, Soundness/Correctness, Substance, Replicability, Meaningful Comparison, and Clarity (Yuan et al., 2022) and (2) the type of argument being presented, namely, Evaluation, Fact, Request, Reference, and Quote (Hua et al., 2019).”

Flagged first, then by citations

Research works printed but not linked: 48

  1. Rebecca Abma-Schouten, Joey Gijbels, Wendy Reijmerink, and Ingeborg Meijer. 2023. Evaluation of research proposals by peer review panels: broader panels for broader assessments? Science and Public Policy, 50(4):619-632. · DOI
  2. Karen A Bantel and Susan E Jackson. 1989. Top management and innovations in banking: Does the composition of the top team make a difference? Strategic management journal, 10(S1):107-124. · DOI
  3. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289-300.
  4. Federico Bianchi and Flaminio Squazzoni. 2015. Is three better than one? Simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review. In Proceedings of the 2015 Winter Simulation Conference, WSC '15, page 4081-4089. IEEE Press. · DOI
  5. David Blei, Andrew Ng, and Michael Jordan. 2003. Latent dirichlet allocation. The Journal of Machine Learning Research, 3(1):993-1022. · DOI
  6. Harvey Brooks. 1978. The problem of research priorities. Daedalus, 107(2):171-190.
  7. Laurent Charlin and Richard Zemel. 2013. The toronto paper matching system: An automated paper-reviewer assignment system. In Proceedings of the International Conference on Machine Learning Workshop on Peer Reviewing and Publishing Models.
  8. Daryl E Chubin and Edward J Hackett. 1990. Peerless science: Peer review and US science policy. State University of New York Press.
  9. Ronald Aylmer Fisher. 1936. Design of experiments. British Medical Journal, 1(3923):554.
  10. Mikael Fogelholm, Saara Leppinen, Anssi Auvinen, Jani Raitanen, Anu Nuutinen, and Kalervo Väänänen. 2012. Panel discussion does not improve reliability of peer review for medical research grant proposals. Journal of clinical epidemiology, 65(1):47-52.
  11. Francisco Grimaldo and Mario Paolucci. 2013. A simulation of disagreement for control of rational cheating in peer review. Advances in Complex Systems, 16(07):1350004. · DOI
  12. Lowell Hargens and Jerald Herting. 1990. Neglected considerations in the analysis of agreement among journal referees. Scientometrics, 19(1-2):91-106. · DOI
  13. L Richard Hoffman, Ernest Harburg, and Norman RF Maier. 1962. Differences and disagreement as factors in creative group problem solving. The Journal of Abnormal and Social Psychology, 64(3):206. · DOI
  14. Bas Hofstra, Vivek V. Kulkarni, Sebastian Munoz-Najar Galvez, Bryan He, Dan Jurafsky, and Daniel A. McFarland. 2020. The diversity-innovation paradox in science. Proceedings of the National Academy of Sciences, 117(17):9284-9291. · DOI
  15. Lu Hong and Scott E. Page. 2004. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences, 101(46):16385-16389. · DOI · Paper page
  16. Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar Shah, Vincent Conitzer, and Fei Fang. 2020. Mitigating manipulation in peer review via randomized reviewer assignments. In Advances in Neural Information Processing Systems, volume 33, pages 12533-12545.
  17. Tom Jefferson, Philip Alderson, Elizabeth Wager, and Frank Davidoff. 2002a. Effects of editorial peer review: a systematic review. JAMA, 287(21):2784-2786. · DOI · Paper page
  18. Tom Jefferson, Elizabeth Wager, and Frank Davidoff. 2002b. Measuring the Quality of Editorial Peer Review. JAMA, 287(21):2786-2790. · DOI · Paper page
  19. Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, et al. 2024. What can natural language processing do for peer review? arXiv preprint arXiv:2405.06563. · arXiv
  20. Grit Laudel. 2006. Conclave in the Tower of Babel: How peers review interdisciplinary research proposals. Research Evaluation, 15(1):57-68. · DOI
  21. Carole J. Lee. 2012. A kuhnian critique of psychometric research on peer review. Philosophy of Science, 79(5):859-870. · DOI
  22. Carole J Lee, Cassidy R Sugimoto, Guo Zhang, and Blaise Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2-17. · DOI · Paper page
  23. Sheen S Levine, Evan P Apfelbaum, Mark Bernard, Valerie L Bartelt, Edward J Zajac, and David Stark. 2014. Ethnic diversity deflates price bubbles. Proceedings of the National Academy of Sciences, 111(52):18524-18529. · DOI
  24. Kevin Leyton-Brown, Mausam, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, and Dinesh Raghu. 2024. Matching papers and reviewers at large conferences. Artificial Intelligence, 331:104119. · DOI · Paper page
  25. Dakota Murray, Kyle Siler, Vincent Larivière, Wei Mun Chan, Andrew M. Collings, Jennifer Raymond, and Cassidy R. Sugimoto. 2018. Gender and international diversity improves equity in peer review. bioRxiv. · DOI
  26. Michael Obrecht, Karl Tibelius, and Guy D'Aloisio. 2007. Examining the value added by committee discussion in the review of applications for research awards. Research Evaluation, 16(2):79-91. · DOI
  27. Meike Olbrecht and Lutz Bornmann. 2010. Panel peer review of grant applications: what do we know from research in social psychology on judgment and decision-making in groups? Research Evaluation, 19(4):293-304. · DOI
  28. Scott Page. 2008. The difference: How the power of diversity creates better groups, firms, schools, and societies-new edition. Princeton University Press.
  29. Karl Pearson. 1895. Note on Regression and Inheritance in the Case of Two Parents. Proceedings of the Royal Society of London Series I, 58:240-242.
  30. Elizabeth Pier, Joshua Raclaw, Anna Kaatz, Markus Brauer, Molly Carnes, Mitchell Nathan, and Cecilia Ford. 2017. Your comments are meaner than your score: score calibration talk influences intra-and inter-panel variability during scientific grant peer review. Research Evaluation, 26(1):1-14. · DOI
  31. Alan L. Porter and Frederick A. Rossini. 1985. Peer review of interdisciplinary research proposals. Science, Technology, & Human Values, 10(3):33-38. · DOI
  32. Charvi Rastogi, Xiangchen Song, Zhijing Jin, Ivan Stelmakh, Hal Daumé III, Kun Zhang, and Nihar B Shah. 2024. A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. arXiv preprint arXiv:2403.01015. · arXiv
  33. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
  34. Alison Reynolds and David Lewis. 2017. Teams solve problems faster when they're more cognitively diverse. Harvard Business Review, 30:1-8.
  35. Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM '15, page 399-408, New York, NY, USA. Association for Computing Machinery. · DOI
  36. Paul R Rosenbaum and Donald B Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55. · DOI · Paper page
  37. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. · arXiv
  38. Martin Saveski, Steven Jecmen, Nihar Shah, and Johan Ugander. 2023. Counterfactual evaluation of peer-review assignment policies. In Advances in Neural Information Processing Systems, volume 36, pages 58765-58786.
  39. Nihar B. Shah. 2022. Challenges, experiments, and computational solutions in peer review. Commun. ACM, 65(6):76-87. · DOI · Paper page
  40. Samuel R Sommers. 2006. On racial diversity and group decision making: identifying multiple effects of racial composition on jury deliberations. Journal of personality and social psychology, 90(4):597. · DOI · Paper page
  41. Ivan Stelmakh, Charvi Rastogi, Ryan Liu, Shuchi Chawla, Federico Echenique, and Nihar B Shah. 2023a. Cite-seeing and reviewing: A study on citation bias in peer review. PLoS ONE, 18(7):e0283980. · DOI
  42. Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daumé III. 2023b. A large scale randomized controlled trial on herding in peer-review discussions. PLoS ONE, 18(7):e0287443. · DOI
  43. Misha Teplitskiy, Hardeep Ranu, Gary S Gray, Michael Menietti, Eva Guinan, and Karim R Lakhani. 2019. Do experts listen to other experts? field experimental evidence from scientific peer review.
  44. Andrew Tomkins, Min Zhang, and William D. Heavlin. 2017. Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48):12708-12713. · DOI · Paper page
  45. Mark Ware. 2008. Peer review: benefits, perceptions and alternatives. Citeseer.
  46. Wenting Xiong and Diane Litman. 2011. Automatically predicting peer-review helpfulness. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 502-507, Portland, Oregon, USA. Association for Computational Linguistics.
  47. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171-212. · DOI · Paper page
  48. James M. Zumel Dumlao and Misha Teplitskiy. 2023. The effect of reviewer geographical diversity on evaluations is reduced by anonymizing submissions. SocArXiv. · DOI

Cited by

Sorted by citations

Authors
Goyal, N., Stelmakh, I., Shah, N., & Daumé, H.
Affiliations
  • University of Maryland, College Park (US)
  • New Economic School (RU)
  • Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2411.11437
arXiv
2411.11437
OpenAlex
W4404570840
Cited by
0
References
1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1942124)
  • Office of Naval Research (N000142212181)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
peer review, reviewer diversity, reviewer assignment, group decision making, publication networks, geographic diversity, conference peer review, bias mitigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review