CoChat
Sign Up
Cited by 24Publishing and Scholarly CommunicationAcademic Writing and PublishingExpert finding and Q&A systems

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Open access
Stelmakh, Ivan, Shah, Nihar B., Singh, Aarti, Daumé III, Hal·2021·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v35i6.16610
Open this paper in CoChat — free account

Abstract

Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others. A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process by increasing the burden on the pool of qualified reviewers which is growing at a much slower rate. In this work, we consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. Specifically, we design a procedure for (i) recruiting reviewers from the population not typically covered by major conferences and (ii) guiding them through the reviewing pipeline. In conjunction with the ICML 2020 --- a large, top-tier machine learning conference --- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers. Our experiment reveals that a combination of the recruiting and guiding mechanisms allows for a principled enhancement of the reviewer pool and results in reviews of superior quality compared to the conventional pool of reviews as evaluated by senior members of the program committee (meta-reviewers).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v35i6.16610

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • At ICML 2020, newly recruited and trained reviewers were evaluated as higher quality than reviewers in the standard reviewer pool according to a number of metrics, including meta-reviewers' evaluations of reviews.

    “Similarly, a study at the ICML 2020 conference [13] designed a special process to recruit new paper reviewers and asked meta-reviewers to evaluate the review quality from this group and from the standard group of reviewers. They found that their newly recruited and trained reviewers were evaluated as higher quality than reviewers in the standard reviewer pool according to a number of metrics which”

    From Peer reviews of peer reviews: A randomized controlled trial and other experiments (2025) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 35

  1. ACLCommittee (2018). A report on the review process of ACL 2018. https://acl2018.org/2018/05/19/ how-decisions-made/ [Accessed: 9/7/2020].
  2. Beygelzimer, A., Fox, E., dÁlche Buc, F., and Larochelle, H. (2019). What we learned from NeurIPS 2019 data. https: //medium.com/@NeurIPSConf/what-we-learned-from-neurips-2019-data-111ab996462c [Accessed: 9/7/2020].
  3. Chandler, D. and Kapelner, A. (2013). Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior & Organization , 90:123 - 133.
  4. Feldmann, A. (2005). Experiences from the sigcomm 2005 european shadow pc experiment. ACM SIGCOMM Computer Communication Review , 35(3):97-102.
  5. Fiez, T., Shah, N., and Ratliff, L. (2020). A SUPER* algorithm to optimize paper bidding in peer review. In Conference on Uncertainty in Artificial Intelligence .
  6. Fisher, R. A. (1935). The design of experiments. Oliver & Boyd, Oxford, England.
  7. Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., and Miyao, Y. (2019). Does my rebuttal matter? Insights from a major NLP conference. CoRR , abs/1903.11367. · arXiv
  8. Garisto, D. (2019). Diversifying peer review by adding junior scientists. https://www.natureindex.com/news-blog/ diversifying-peer-review-by-adding-junior-scientists [Accessed: 9/7/2020].
  9. Jecmen, S., Zhang, H., Liu, R., Shah, N. B., Conitzer, V., and Fang, F. (2020). Mitigating manipulation in peer review via randomized reviewer assignments. In NeurIPS .
  10. Kerzendorf, W. E., Patat, F., Bordelon, D., van de Ven, G., and Pritchard, T. A. (2020). Distributed peer review enhanced with natural language processing and machine learning.
  11. Khosla, A., Hoiem, D., and Belongie, S. (2013). Analysis of reviews for CVPR 2012.
  12. Köhler, T., González-Morales, M. G., Banks, G. C., O'Boyle, E. H., Allen, J. A., Sinha, R., Woo, S. E., and Gulick, L. M. (2020). Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: Introducing a competency framework for peer review. Industrial and Organizational Psychology , 13(1):1-27.
  13. Kotturi, Y., Kahng, A., Procaccia, A. D., and Kulkarni, C. (2020). Hirepeer: Impartial peer-assessed hiring at scale in expert crowdsourcing markets. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence , AAAI'20. AAAI Press.
  14. Kurokawa, D., Lev, O., Morgenstern, J., and Procaccia, A. D. (2015). Impartial peer review. In Proceedings of the 24th International Conference on Artificial Intelligence , IJCAI'15, page 582-588. AAAI Press.
  15. Levy, P. and Sarne, D. (2018). Understanding over participation in simple contests. In Proceedings of the ThirtySecond AAAI Conference on Artificial Intelligence , AAAI'18. AAAI Press.
  16. Lian, J. W., Mattei, N., Noble, R., and Walsh, T. (2018). The conference paper assignment problem: Using order weighted averages to assign indivisible goods. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , AAAI'18. AAAI Press.
  17. Lin, H.-T., Balcan, M. F., Hadsell, R., and Ranzato, M. (2020). What we learned from NeurIPS 2020 reviewing process. https://medium.com/@NeurIPSConf/ what-we-learned-from-neurips-2020-reviewing-process-e24549eea38f [Accessed: 11/23/2020].
  18. McCambridge, J., De Bruin, M., and Witton, J. (2012). The effects of demand characteristics on research participant behaviours in non-laboratory settings: a systematic review. PloS one , 7(6):e39116.
  19. McCook, A. (2006). Is peer review broken? submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. what's wrong with peer review? The scientist , 20(2):26-35.
  20. Mogul, J. C. (2013). Towards more constructive reviewing of SIGCOMM papers. SIGCOMM Comput. Commun. Rev. , 43(3):90-94.
  21. Noothigattu, R., Shah, N. B., and Procaccia, A. D. (2020). Loss functions, axioms, and peer review. In ICML workshop on Incentives in Machine Learning .
  22. Papagiannaki, K. (2007). Author feedback experiment at pam 2007. SIGCOMM Comput. Commun. Rev. , 37(3):73-78.
  23. Patat, F., Kerzendorf, W., Bordelon, D., Van de Ven, G., and Pritchard, T. (2019). The Distributed Peer Review Experiment. The Messenger , 177:3-13.
  24. Picciotto, M. (2018). New reviewer mentoring program. Journal of Neuroscience , 38(3):511-511.
  25. Sculley, D., Snoek, J., and Wiltschko, A. B. (2019). Avoiding a tragedy of the commons in the peer review process. CoRR , abs/1901.06246. · arXiv
  26. Shah, N. B. (2019b). Principled methods to improve peer review.
  27. Shah, N. B., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018). Design and analysis of the NIPS 2016 review process. The Journal of Machine Learning Research , 19(1):1913-1946.
  28. Stelmakh, I., Shah, N., and Singh, A. (2019). On testing for biases in peer review. In NeurIPS .
  29. Stelmakh, I., Shah, N. B., and Singh, A. (2018). PeerReview4All: Fair and accurate reviewer assignment in peer review. arXiv preprint arXiv:1806.06237 . · arXiv
  30. Stelmakh, I., Shah, N. B., Singh, A., and Daumé III, H. (2020). Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. Preprint http://www.cs.cmu.edu/afs/cs.cmu.edu/user/istelmak/www/ icml/prior.pdf .
  31. Tomiyama, A. J. (2007). Getting involved in the peer review process. https://www.apa.org/science/about/psa/ 2007/06/student-council [Accessed: 9/7/2020].
  32. Vardi, M. Y. (2010). Hypercriticality. Communications of the ACM , 53(7):5-5.
  33. Weber, E. J., Katz, P. P., Waeckerle, J. F., and Callaham, M. L. (2002). Author perception of peer review: impact of review quality and acceptance on satisfaction. JAMA , 287(21):2790-2793.
  34. Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association , 22(158):209-212.
  35. Xu, Y., Zhao, H., Shi, X., and Shah, N. (2019). On strategyproof conference review. In IJCAI .

Cited by

Sorted by citations

Authors
Stelmakh, I., Shah, N. B., Singh, A., & Daumé III, H.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2021-05-18
Volume
35
Issue
6
Pages
4785-4793
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v35i6.16610
OpenAlex
W3108144518
Cited by
24
References
0
Citations by year
2026: 4 · 2025: 3 · 2024: 4 · 2023: 3 · 2022: 2 · 2021: 7 · 2020: 1
FWCI
40.28
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1763734, 1942124, CAREER)
  • Carnegie Mellon University
  • Strong
  • Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México
Topics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
Keywords
conference peer review, reviewer recruitment, reviewer training, review quality, program committee

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review