CoChat
Sign Up
PreprintCited by 3Recommender Systems and TechniquesGame Theory and Voting SystemsInformation Retrieval and Search Behavior

Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective

Open access
Dycke, Nils, Simpson, Edwin, Kuznetsov, Ilia, Gurevych, Iryna·2021·arXiv
DOI 10.48550/arxiv.2109.01190arXiv 2109.01190
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2109.01190

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The agreement level is in line with agreement levels reported for peer review ratings and typical for subjective annotation tasks.

    “This is in line with agreement levels reported for peer review ratings (Bornmann, 2011;Dycke et al., 2021) and typical for subjective annotation tasks.”

    From STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond (2025) · cites this paper as Dycke et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Argument Mining for Understanding Peer Reviews

    Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 1× · see citing works

    “In the AMPERE dataset [Hua et al. , 2019], 400 computer science reviews are annotated with sentence-level discourse labels, including e.g. request or non-argumentative .”
    Method

Flagged first, then by citations

Research works printed but not linked: 28

  1. Aksnes et al., 2019] Dag W. Aksnes, Liv Langfeldt, and Paul Wouters. Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. SAGE Open, 9:215824401982957, January 2019.
  2. Jacob P. Baskin and Shri- ram Krishnamurthi. Preference aggregation in group rec- ommender systems for committee decision-making. In Proceedings of the 3rd ACM RecSys '09, page 337, 2009.
  3. Aliaksandr Birukou, Joseph Rushton Wakeling, Claudio Bartolini, Fabio Casati, Maurizio Marchese, Katsiaryna Mirylenka, Nardine Osman, Az- zurra Ragone, Carles Sierra, and Aalam Wassef. Alter- natives to Peer Review: Novel Approaches for Research Evaluation. Frontiers in Computational Neuroscience, 5, 2011.
  4. Lutz Bornmann. Scientific peer review. Annual Review of Information Science and Technology, 45(1):197-245, 2011.
  5. Chu and Ghahramani, 2005] Wei Chu and Zoubin Ghahra- mani. Preference learning with Gaussian processes. In Proceedings of the 22nd ICML, pages 137-144, 2005.
  6. Kenneth Church. Reviewing the reviewers. Computational Linguistics, 31(4):575-578, 2005.
  7. Wade D. Cook, Boaz Golany, Michal Penn, and Tal Raviv. Creating a consensus ranking of pro- posals from reviewers' partial ordinal rankings. Comput- ers & Operations Research, 34(4):954-965, April 2007.
  8. Douglas E. Critchlow, Michael A. Fligner, and Joseph S. Verducci. Probability models on rankings. Journal of mathematical psychology, 35(3):294- 318, 1991.
  9. Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Under- standing. In Proceedings of NAACL 2019: Human Lan- guage Technologies, Vol. 1, pages 4171-4186, June 2019.
  10. Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. Does My Rebuttal Matter? Insights from a Major NLP Conference. In Pro- ceedings of NAACL 2019: Human Language Technolo- gies, Volume 1, pages 1274-1290, June 2019.
  11. Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, and Pushpak Bhattacharyya. DeepSentiPeer: Har- nessing sentiment in review texts to recommend peer re- view decisions. In Proceedings of the 57th Annual Meet- ing of the ACL, pages 1120-1130, July 2019.
  12. Jefferson et al., 2002] Tom Jefferson, Elizabeth Wager, and Frank Davidoff. Measuring the quality of editorial peer review. Jama, 287(21):2786-2790, 2002.
  13. Kang et al., 2018] Dongyeop Kang, Waleed Ammar, Bha- vana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A Dataset of Peer Re- views (PeerRead): Collection, Insights and NLP Applica- tions. In Proceedings of NAACL 2018: Human Language Technologies, Volume 1, pages 1647-1661, June 2018.
  14. K. Krippendorff. Content Analysis: An Introduction To Its Methodology. Sage Publications, 1980.
  15. Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. Bias in peer review. Journal of the ASIS&T, 64(1):2-17, 2013.
  16. Carole J Lee. Commensuration Bias in Peer Re- view. Philosophy of Science, 82(5):1272-1283, 2015.
  17. Li et al., 2019] Siqing Li, Wayne Xin Zhao, Eddy Jing Yin, and Ji-Rong Wen. A Neural Citation Count Prediction Model based on Peer Review Text. In Proceedings of the 2019 EMNLP-IJCNLP, pages 4913-4923, China, 2019.
  18. Maillette de Buy Wenniger et al., 2020] Gideon Mail- lette de Buy Wenniger, Thomas van Dongen, Eleri Aedmaa, Herbert Teun Kruitbosch, Edwin A. Valentijn, and Lambert Schomaker. Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction. In Proceedings of the First Workshop on Scholarly Document Processing, pages 158-167, November 2020.
  19. McGraw and Wong, 1996]
  20. Kenneth O McGraw and Seok P Wong. Forming inferences about some intraclass correla- tion coefficients. Psychological methods, 1(1):30, 1996.
  21. Saif M. Mohammad. Examining Cita- tions of Natural Language Processing Literature. In Pro- ceedings of the 58th ACL, pages 5199-5209, 2020.
  22. Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. On peer review in computer science: analysis of its effec- tiveness and suggestions for improvement. Scientometrics, 97(2):317-356, November 2013.
  23. Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006.
  24. Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 EMNLP-IJCNLP, pages 3982-3992. Association for Computational Linguistics, November 2019.
  25. Simpson and Gurevych, 2018] Edwin Simpson and Iryna Gurevych. Finding convincing arguments using scalable bayesian preference learning. Transactions of the ACL, 6:357-371, 2018.
  26. Lukas Stappen, Georgios Rizos, Mad- ina Hasan, Thomas Hain, and Björn W. Schuller. Uncertainty-Aware Machine Support for Paper Review- ing on the Interspeech 2019 Submission Corpus. In In- terspeech 2020, pages 1808-1812, October 2020.
  27. Walker and Rocha da Silva, 2015] Richard Walker and Pas- cal Rocha da Silva. Emerging trends in peer review-a survey. Frontiers in Neuroscience, 9, May 2015.
  28. Wang and Shah, 2019] Jingyan Wang and Nihar B. Shah. Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Mis- calibrations in Ratings. In Proceedings of the 18th AA- MAS, volume 18, 2019.

Cited by

Sorted by citations

Authors
Dycke, N., Simpson, E., Kuznetsov, I., & Gurevych, I.
Affiliations
  • Technische Universität Darmstadt (DE)
  • University of Bristol (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2109.01190
arXiv
2109.01190
OpenAlex
W4286989422
Cited by
3
References
1
Citations by year
2026: 1 · 2023: 1 · 2022: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
scholarly peer review, paper ranking, preference learning, review scores, review text, assessment fairness, evaluation efficiency, evaluation effectiveness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review