CoChat
Sign Up
PreprintCited by 4

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

Jecmen, Steven, Zhang, Hanrui, Liu, Ryan, Shah, Nihar B., Conitzer, Vincent, Fang, Fei·2020·arXiv
DOI 10.48550/arxiv.2006.16437arXiv 2006.16437
Open this paper in CoChat — free account

Abstract

We consider three important challenges in conference peer review: (i) reviewers maliciously attempting to get assigned to certain papers to provide positive reviews, possibly as part of quid-pro-quo arrangements with the authors; (ii) "torpedo reviewing," where reviewers deliberately attempt to get assigned to certain papers that they dislike in order to reject them; (iii) reviewer de-anonymization on release of the similarities and the reviewer-assignment code. On the conceptual front, we identify connections between these three problems and present a framework that brings all these challenges under a common umbrella. We then present a (randomized) algorithm for reviewer assignment that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair. We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case. Finally, we experimentally evaluate our algorithms on datasets from past conferences, where we observe that they can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity. Our algorithms still achieve this similarity while also preventing reviewers with close associations from being assigned to the same paper.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.16437

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Other AI/ML conferences have addressed incentives for strategic behavior by adding randomness to reviewer-paper matching scores to make the formation of collusion rings more difficult.

    “Other AI/ML conferences have further addressed incentives for strategic behavior by adding randomness to reviewer-paper matching scores to make the formation of collusion rings more difficult (Jecmen et al., 2020).”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Jecmen et al., 2020

    The Introduction describes randomized reviewer assignments designed to reduce manipulation, including collusion-related risks. But Related Literature says the paper takes similarity-score computation as given, and the Introduction describes tests on past conference datasets—not conferences adopting randomness in matching scores. No page numbers are available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Adrian Mulligan, Louise Hall, and Ellen Raphael. Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the Association for Information Science and Technology, 64(1):132-161, 2013.
  2. David Nicholas, Anthony Watkinson, Hamid R Jamali, Eti Herman, Carol Tenopir, Rachel Volentine, Suzie Allard, and Kenneth Levine. Peer review: Still king in the digital age. Learned Publishing, 28(1):15-21, 2015.
  3. Mark Ware. Peer review: Benefits, perceptions and alternatives. Publishing Research Consortium, 4:1-20, 2008.
  4. T. N. Vijaykumar. Potential organized fraud in ACM/IEEE computer architecture conferences. https: //medium.com/@tnvijayk/potential-organized-fraud-in-acm-ieee-computer-architecture- conferences-ccd61169370d, 2020 (accessed August 17, 2020).
  5. Cat Ferguson, Adam Marcus, and Ivan Oransky. Publishing: The peer-review scam. Nature News, 515(7528):480, 2014.
  6. Jian Gao and Tao Zhou. Retractions: Stamp out fake peer review. Nature, 546(7656):33-33, 2017.
  7. Edward F Barroga. Safeguarding the integrity of science communication by restraining 'rational cheating' in peer review. Journal of Korean Medical Science, 29(11):1450-1452, 2014.
  8. Mario Paolucci and Francisco Grimaldo. Mechanism change in a simulation of peer review: From junk support to elitism. Scientometrics, 99(3):663-688, 2014.
  9. Jef Akst. I hate your paper. Many say the peer review system is broken. Here's how some journals are trying to fix it. The Scientist, 24(8):36, 2010.
  10. Arvind Narayanan and Vitaly Shmatikov. How to break anonymity of the Netflix Prize dataset. arXiv preprint arXiv:cs/0610105, 2006. · arXiv
  11. Laurent Charlin, Richard S Zemel, and Craig Boutilier. A framework for optimizing paper matching. arXiv preprint arXiv:1202.3706, 2012. · arXiv
  12. Cheng Long, Raymond Chi-Wing Wong, Yu Peng, and Liangliang Ye. On good and fair paper-reviewer assignment. In 2013 IEEE 13th International Conference on Data Mining, pages 1145-1150. IEEE, 2013.
  13. Judy Goldsmith and Robert H Sloan. The AI conference paper assignment problem. In Proc. AAAI Workshop on Preference Handling for Artificial Intelligence, Vancouver, pages 53-57, 2007.
  14. Wenbin Tang, Jie Tang, and Chenhao Tan. Expertise matching via constraint-based optimization. In 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, volume 1, pages 34-41. IEEE, 2010.
  15. Peter A Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. ACM SIGKDD Explorations Newsletter, 11(2):63-67, 2010.
  16. Jing Wu Lian, Nicholas Mattei, Renee Noble, and Toby Walsh. The conference paper assignment problem: Using order weighted averages to assign indivisible goods. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  17. Ari Kobren, Barna Saha, and Andrew McCallum. Paper matching with local fairness constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.
  18. N. Garg, T. Kavitha, A. Kumar, K. Mehlhorn, and J. Mestre. Assigning papers to referees. Algorithmica, 58(1):119-136, Sep 2010.
  19. Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. arXiv preprint arXiv:1806.06237, 2018. · arXiv
  20. Hong Ge, Max Welling, and Zoubin Ghahramani. A Bayesian model for calibrating conference review scores. http://mlg.eng.cam.ac.uk/hong/unpublished/nips-review-model.pdf, 2013 (accessed June 29, 2020).
  21. Ritesh Noothigattu, Nihar B Shah, and Ariel D Procaccia. Loss functions, axioms, and peer review. arXiv preprint arXiv:1808.09057, 2018. · arXiv
  22. T Fiez, N Shah, and L Ratliff. A SUPER* algorithm to optimize paper bidding in peer review. In Conference on Uncertainty in Artificial Intelligence (UAI), 2020.
  23. Magnus Roos, Jörg Rothe, and Björn Scheuermann. How to calibrate the scores of biased reviewers by quadratic programming. In AAAI Conference on Artificial Intelligence, 2011.
  24. Ivan Stelmakh, Nihar Shah, and Aarti Singh. On testing for biases in peer review. In NeurIPS, 2019.
  25. N. Lawrence and C. Cortes. The NIPS experiment. http://inverseprobability.com/2014/12/16/ the-nips-experiment, 2014 (accessed June 29, 2020).
  26. Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the NIPS 2016 review process. The Journal of Machine Learning Research, 19(1):1913-1946, 2018.
  27. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard H. Hovy, and Roy Schwartz. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. arXiv preprint arXiv:1804.09635, 2018. · arXiv
  28. Emaad Manzoor and Nihar B Shah. Uncovering latent biases in text: Method and application to peer review. In INFORMS Workshop on Data Science, 2020.
  29. Ivan Stelmakh, Nihar Shah, Aarti Singh, and Hal Daumé III. Prior and prejudice: The bias against resubmissions in conference peer review. arXiv, 2020.
  30. Ivan Stelmakh, Nihar Shah, and Aarti Singh. Catch me if I can: Detecting strategic behaviour in peer assessment. arXiv, 2020.
  31. Yichong Xu, Han Zhao, Xiaofei Shi, Jeremy Zhang, and Nihar B. Shah. On strategyproof conference peer review. arXiv preprint arXiv:1806.06266, 2018. · arXiv
  32. David Kurokawa, Omer Lev, Jamie Morgenstern, and Ariel D Procaccia. Impartial peer review. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
  33. Anson Kahng, Yasmine Kotturi, Chinmay Kulkarni, David Kurokawa, and Ariel D Procaccia. Ranking wily people who rank each other. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  34. Ron Holzman and Hervé Moulin. Impartial nominations for a prize. Econometrica, 81(1):173-196, 2013.
  35. Wenxin Ding, Nihar B., and Weina Wang. On the privacy-utility tradeoff in peer-review data analysis. arXiv, 2020.
  36. Leman Akoglu, Hanghang Tong, and Danai Koutra. Graph based anomaly detection and description: A survey. Data mining and knowledge discovery, 29(3):626-688, 2015.
  37. Bryan Hooi, Hyun Ah Song, Alex Beutel, Neil Shah, Kijung Shin, and Christos Faloutsos. FRAUDAR: Bounding graph fraud in the face of camouflage. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 895-904, 2016.
  38. Srijan Kumar, Bryan Hooi, Disha Makhija, Mohit Kumar, Christos Faloutsos, and VS Subrahmanian. REV2: Fraudulent user prediction in rating platforms. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 333-341, 2018.
  39. Guan Wang, Sihong Xie, Bing Liu, and S Yu Philip. Review graph based online store review spammer detection. In 2011 IEEE 11th International Conference on Data Mining, pages 1242-1247. IEEE, 2011.
  40. Leman Akoglu, Rishi Chandy, and Christos Faloutsos. Opinion fraud detection in online reviews by network effects. ICWSM, 13(2-11):29, 2013.
  41. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock. Finding deceptive opinion spam by any stretch of the imagination. arXiv preprint arXiv:1107.4557, 2011. · arXiv
  42. Mingdan Si and Qingshan Li. Shilling attacks against collaborative recommender systems: A review. Artificial Intelligence Review, 53(1):291-319, 2020.
  43. Aanund Hylland and Richard Zeckhauser. The efficient allocation of individuals to positions. Journal of Political economy, 87(2):293-314, 1979.
  44. Anna Bogomolnaia and Hervé Moulin. A new solution to the random assignment problem. Journal of Economic theory, 100(2):295-328, 2001.
  45. Dmytro Korzhyk, Vincent Conitzer, and Ronald Parr. Complexity of computing optimal Stackelberg strategies in security resource allocation games. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
  46. John Von Neumann. A certain zero-sum two-person game equivalent to the optimal assignment problem. Contributions to the Theory of Games, 2(0):5-12, 1953.
  47. Shunhua Jiang, Zhao Song, Omri Weinstein, and Hengjie Zhang. Faster dynamic matrix inverse for faster LPs. arXiv preprint arXiv:2004.07470, 2020. · arXiv
  48. Richard M Karp. Reducibility among combinatorial problems. In Complexity of computer computations, pages 85-103. Springer, 1972.

Cited by

Sorted by citations

Authors
Jecmen, S., Zhang, H., Liu, R., Shah, N. B., Conitzer, V., & Fang, F.
Venue
arXiv
Publisher
arXiv
Published
2020
DOI
10.48550/arxiv.2006.16437
arXiv
2006.16437
References
1
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review