CoChat
Sign Up
Cited by 2

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Thorn Jakobsen, Terne, Rogers, Anna·2022·Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
DOI 10.18653/v1/2022.naacl-main.354
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.naacl-main.354

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overview of all respondents and overlap of their roles for their last experience at NLP venues.
Figure 1 Overview of all respondents and overlap of their roles for their last experience at NLP venues.CC-BY · From the PDF
Figure 2 Career status of the respondents vs their experience receiving peer review. Numerical data is available in table 1 in the appendix.
Figure 2 Career status of the respondents vs their experience receiving peer review. Numerical data is available in table 1 in the appendix.CC-BY · From the PDF
Figure 3 Types of research performed by respondents (multiple options could be selected).
Figure 3 Types of research performed by respondents (multiple options could be selected).CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 6 The diverging bars shows the experienced utility of different kinds of information about reviewers that ACs may have been presented with to assist in manual checks of paper-reviewer matches. If the respondent had never been presented with the specific kind of information they chose 'Never provided'.
Figure 6 The diverging bars shows the experienced utility of different kinds of information about reviewers that ACs may have been presented with to assist in manual checks of paper-reviewer matches. If the respondent had never been presented with the specific kind of information they chose 'Never provided'.CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF
(a) Reported satisfaction with most recent experience. AC / Action Editor question: (...) how satisfied were you with the support provided to you to improve the paper-reviewer matching? Reviewer question: (...) How satisfied were you with the paper-reviewer matching? Author question: (...) How satisfied were you with the amount of constructive criticism in the reviews you received?
(a) Reported satisfaction with most recent experience. AC / Action Editor question: (...) how satisfied were you with the support provided to you to improve the paper-reviewer matching? Reviewer question: (...) How satisfied were you with the paper-reviewer matching? Author question: (...) How satisfied were you with the amount of constructive criticism in the reviews you received?CC-BY · From the PDF
(b) Question: would you say your most recent experience with paper-reviewer matching/paper assignment(s)/set of reviews described above, was better or worse than on average?
(b) Question: would you say your most recent experience with paper-reviewer matching/paper assignment(s)/set of reviews described above, was better or worse than on average?CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 30

  1. Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. 2021. The NeurIPS 2021 Consistency Experiment.
  2. Claire Cardie, Iryna Gurevych, and Yusuke Miyao. ACL 2018: A Report on the Review Process of ACL 2018.
  3. Johan S. G. Chu and James A. Evans. 2021. Slowed canonical progress in large fields of science. Pro- ceedings of the National Academy of Sciences, 118(41). · DOI
  4. Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL.
  5. Andrew Collins, Dominika Tkaczyk, Akiko Aizawa, and Joeran Beel. 2018. A Study of Position Bias in Digital Library Recommender Systems. arXiv:1802.06565 [cs. · arXiv
  6. Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An experimental comparison of click position-bias models. In Proceedings of the 2008 International Conference on Web Search and Data Mining, WSDM '08, pages 87-94, New York, NY, USA. Association for Computing Machinery. · DOI
  7. Jane Dai, Jeremy Cone, and Jeff Moher. 2020. Percep- tual salience influences food choices independently of health and taste preferences. Cognitive research: principles and implications, 5(1):2-13.
  8. Hal Daumé III. 2015. Some NAACL 2013 statistics on author response, review quality, etc.
  9. Richard Gordon and Bryan J. Poulin. 2009. Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Re- searcher a Baseline Grant. Accountability in Re- search, 16(1):13-40. · DOI
  10. David Hartvigsen, Jerry C. Wei, and Richard Czuch- lewski. 1999. The Conference Paper-Reviewer As- signment Problem*. Decision Sciences, 30(3):865- 876. · DOI
  11. Laura Hyman, Julie Lamb, and Martin Bulmer. 2006. The use of pre-existing survey questions: Implica- tions for data quality. In Proceedings of the Eu- ropean Conference on Quality in Survey Statistics, pages 1-8. Cardiff Wales, UK.
  12. Xinlian Li and Toyohide Watanabe. 2013. Automatic Paper-to-reviewer Assignment, based on the Match- ing Degree of the Reviewers. Procedia Computer Science, 22:633-642. · DOI
  13. Michael L. Littman. 2021. Collusion rings threaten the integrity of computer science research. Communica- tions of the ACM, 64(6):43-44. · DOI · Paper page
  14. Mingnan Liu and Laura Wronski. 2018. Trap ques- tions in online surveys: Results from three web sur- vey experiments. International Journal of Market Research, 60(1):32-49. · DOI
  15. Cheng Long, Raymond Chi-Wing Wong, Yu Peng, and Liangliang Ye. 2013. On Good and Fair Paper- Reviewer Assignment. In 2013 IEEE 13th Inter- national Conference on Data Mining, pages 1145- 1150. · DOI
  16. Graham Neubig. 2020. ACL Rolling Review Proposal.
  17. Graham Neubig, John Wieting, Arya McCarthy, Amanda Stent, Natalie Schluter, and Trevor Cohn. 2021. ACL Reviewer Matching Code. Association for Computational Linguistics.
  18. Eric Price. 2014. The NIPS experiment.
  19. Priscilla Rasmussen. 2021. Acl membership report 2012-2021 summer statistics. ACL Admin Wiki.
  20. Anna Rogers and Isabelle Augenstein. 2020. What Can We Do to Improve Peer Review in NLP? In Findings of EMNLP, pages 1256-1262, Online. Association for Computational Linguistics.
  21. Nihar B Shah. 2022. An overview of challenges, exper- iments, and computational solutions in peer review (extended version). Communications of the ACM.
  22. Anders Søgaard. 2021. Explainable Natural Language Processing. Synthesis Lectures on Human Language Technologies, 14(3):1-123. · DOI
  23. Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. 2019. PeerReview4All: Fair and Accurate Reviewer As- signment in Peer Review. In Proceedings of the 30th International Conference on Algorithmic Learning Theory, pages 828-856.
  24. Seymour Sudman and Norman M. Bradburn. 1973. Ef- fects of time and memory factors on response in sur- veys. Journal of the American Statistical Associa- tion, 68(344):805-815.
  25. A Tversky and D Kahneman. 1981. The framing of decisions and the psychology of choice. Science (American Association for the Advancement of Sci- ence), 211(4481):453-458.
  26. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131.
  27. Fan Wang, Ning Shi, and Ben Chen. 2010. A compre- hensive survey of the reviewer assignment problem. International Journal of Information Technology & Decision Making, 09(04):645-668. · DOI
  28. John Wieting, Kevin Gimpel, Graham Neubig, and Tay- lor Berg-Kirkpatrick. 2019. Simple and effective paraphrastic similarity from parallel translations. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4602- 4608, Florence, Italy. Association for Computational Linguistics. · DOI
  29. Fei Xia, Wenjie Li, and Roberto Navigli. 2021. ACL- IJCNLP 2021 Program Chair Report.
  30. H. Öztas Ayhan and Semih Isiksal. 2005. Memory re- call errors in retrospective surveys: A reverse record check study. Quality and Quantity, 38(5):475-493. Copyright -Kluwer Academic Publishers 2004; Last updated -2021-09-09.

Cited by

Sorted by citations

Metadata

Authors
Thorn Jakobsen, T., & Rogers, A.
Venue
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Publisher
Association for Computational Linguistics
Published
2022
Pages
4810-4823
DOI
10.18653/v1/2022.naacl-main.354
Cited by
1
References
5

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review