CoChat
Sign Up
Cited by 20

Matching papers and reviewers at large conferences

Leyton-Brown, Kevin, Mausam, Nandwani, Yatin, Zarkoob, Hedayat, Cameron, Chris, Newman, Neil, Raghu, Dinesh·2024·Artificial Intelligence
DOI 10.1016/j.artint.2024.104119
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.artint.2024.104119

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AAAI-2021 applied a system of this kind and found at least one unreported conflict for 78.8% of submissions-over 96,000 unreported conflicts in total.

    “AAAI-2021 applied a system of this kind and found at least one unreported conflict for 78.8% of submissions-over 96,000 unreported conflicts in total (Leyton-Brown et al., 2024, Section 5.1.4).”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Leyton-Brown et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. Graham Neubig, John Wieting, Arya McCarthy, Amanda Stent, Natalie Schluter, and Trevor Cohn. ACL reviewer matching code. https://github.com/acl-org/reviewer-paper-matching, 2020.
  2. Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B Shah, Vincent Conitzer, and Fei Fang. Mitigating manipu- lation in peer review via randomized reviewer assignments. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 12533-12545, 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 93fb39474c51b8a82a68413e2a5ae17a-Paper.pdf.
  3. Peter A Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. ACM SIGKDD Explorations Newsletter, 11(2):63-67, 2010.
  4. Naveen Garg, Telikepalli Kavitha, Amit Kumar, Kurt Mehlhorn, and Julián Mestre. Assigning papers to referees. Algo- rithmica, 58(1):119-136, 2010. doi: 10.1007/s00453-009-9386-0. URL https://doi.org/10.1007/s00453-009-9386-0. · DOI
  5. Wenbin Tang, Jie Tang, and Chenhao Tan. Expertise matching via constraint-based optimization. In IEEE/WIC/ACM International Conference on Web Intelligence, pages 34-41, 2010. doi: 10.1109/WI-IAT.2010.133. URL https: //doi.org/10.1109/WI-IAT.2010.133. · DOI
  6. Jing Wu Lian, Nicholas Mattei, Renee Noble, and Toby Walsh. The conference paper assignment problem: Using order weighted averages to assign indivisible goods. In AAAI Conference on Artificial Intelligence, pages 1138-1145, 2018. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17396.
  7. Camillo J Taylor. On the optimal assignment of conference papers to reviewers. 2008.
  8. Ari Kobren, Barna Saha, and Andrew McCallum. Paper matching with local fairness constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages 1247-1257, 2019. doi: 10.1145/ 3292500.3330899. URL https://doi.org/10.1145/3292500.3330899. · DOI · Paper page
  9. Laurent Charlin, Richard Zemel, and Craig Boutilier. A framework for optimizing paper matching. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, pages 86-95, 2011.
  10. Michael L Littman. Collusion rings threaten the integrity of computer science research. Com- munications of the ACM, 64(6):43-44, 2021. URL https://cacm.acm.org/magazines/2021/6/ 252840-collusion-rings-threaten-the-integrity-of-computer-science-research/fulltext.
  11. Neil Lawrence and Corinna Cortes. The NIPS experiment. http://inverseprobability.com/2014/12/16/ the-nips-experiment/, 2014.
  12. Christian Bessiere. [IJCAI-PRICAI 2020] discussions about summary reject. http://www.lirmm.fr/~bessiere/Site/ Media/SR-clarification, 2020.
  13. Cheng Long, Raymond Chi-Wing Wong, Yu Peng, and Liangliang Ye. On good and fair paper-reviewer assignment. In IEEE International Conference on Data Mining, pages 1145-1150, 2013.
  14. Ritesh Noothigattu, Nihar B Shah, and Ariel Procaccia. Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research, 70:1481-1515, 2021.
  15. Ruihan Wu, Chuan Guo, Felix Wu, Rahul Kidambi, Laurens Van Der Maaten, and Kilian Weinberger. Making paper reviewing robust to bid manipulation attacks. In International Conference on Machine Learning, pages 11240-11250.
  16. PMLR, 2021.
  17. David M. Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 500-509, 2007. doi: 10.1145/1281192. 1281247. URL https://doi.org/10.1145/1281192.1281247. · DOI
  18. Omer Anjum, Hongyu Gong, Suma Bhat, Wen-Mei Hwu, and Jinjun Xiong. PaRe: A paper-reviewer matching approach using a common topic space. In Empirical Methods in Natural Language Processing, pages 518-528, 2019. doi: 10.18653/v1/D19-1049. URL https://doi.org/10.18653/v1/D19-1049. · DOI
  19. John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. Simple and effective paraphrastic similarity from parallel translations. In Conference of the Association for Computational Linguistics, ACL, pages 4602-4608, 2019. doi: 10.18653/v1/p19-1453. URL https://doi.org/10.18653/v1/p19-1453. · DOI
  20. Don Conry, Yehuda Koren, and Naren Ramakrishnan. Recommender systems for the conference paper assignment problem. In ACM conference on Recommender systems, pages 357-360, 2009.
  21. David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
  22. Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and deto- kenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-2012. URL https://aclanthology.org/D18-2012. · DOI
  23. Ivan Stelmakh, Nihar B Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. In Algorithmic Learning Theory, pages 828-856. PMLR, 2019.
  24. Yuan Yuan. Poorly-explained NeurIPS 2020 desk-rejects peeve ML researchers. https://syncedreview.com/2020/07/16/ poorly-explained-neurips-2020-desk-rejects-peeve-ml-researchers/, 2020.
  25. Leon Bottou and Michael Littman. ICML 2009's call for paper. https://icml.cc/Conferences/2009/cfp.html, 2009.
  26. Vikram S. Adve and James C. Hoe. ASPLOS XV: Proceedings of the fifteenth international conference on architectural support for programming languages and operating systems. https://dl.acm.org/doi/proceedings/10.1145/1736020, 2010. · DOI
  27. Cristina V. Lopes. The oopsla two-phase review process. SIGPLAN Not., 49(4S):27-32, jul 2014. ISSN 0362-1340. doi: 10.1145/2641638.2641648. URL https://doi.org/10.1145/2641638.2641648. · DOI
  28. Christos Kozyrakis and Emery Berger. Using extended abstracts to improve peer review. https://www.sigarch.org/ using-extended-abstracts-to-improve-peer-review/, 2021.
  29. Hans Boehm, Jack Davidson, Kathleen Fisher, Cormac Flanagan, Jeffrey S. Foster, Jeremy Gibbons, Mary Hall, Graham Hutton, David Padua, Frank Tip, Jan Vitek, Philip Wadler, Keshav Pingali, Michael O'Boyle, Steve Blackburn, and Emery Berger. Practices of PLDI. https://www.sigplan.org/sites/default/files/PracticesofPLDI.pdf, 2021.
  30. Steven Jecmen, Hanrui Zhang, Ryan Liu, Fei Fang, Vincent Conitzer, and Nihar B Shah. Near-optimal reviewer splitting in two-phase paper reviewing and conference experiment design, 2021. URL https://arxiv.org/abs/2108.06371. · arXiv
  31. Neil Lawrence. Paper allocation for NIPS, Jun 2014. URL https://inverseprobability.com/2014/06/28/ paper-allocation-for-nips.
  32. OpenResearch. AAAI acceptance rate. https://www.openresearch.org/wiki/AAAI, 2021.
  33. Denise Rey and Markus Neuhäuser. Wilcoxon-Signed-Rank Test, pages 1658-1659. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 978-3-642-04898-2. doi: 10.1007/978-3-642-04898-2_616. URL https://doi.org/10.1007/ 978-3-642-04898-2_616. · DOI

Cited by

Sorted by citations

Authors
Leyton-Brown, K., Mausam, Nandwani, Y., Zarkoob, H., Cameron, C., Newman, N., & Raghu, D.
Venue
Artificial Intelligence
Publisher
Elsevier BV
Published
2024-06
Volume
331
Pages
104119
ISSN
0004-3702
DOI
10.1016/j.artint.2024.104119
Cited by
20
References
0
Language
en
Licence
Funders
  • CIFAR
  • NSERC

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review