Emerging trends: Reviewing the reviewers (again)
Abstract
The ACL-2019 Business meeting ended with a discussion of reviewing. Conferences are experiencing a success catastrophe. They are becoming bigger and bigger, which is not only a sign of success but also a challenge (for reviewing and more). Various proposals for reducing submissions were discussed at the Business meeting. IMHO, the problem is not so much too many submissions, but rather, random reviewing. We cannot afford to do reviewing as badly as we do (because that leads to even more submissions). Negative feedback loops are effective. The reviewing process will improve over time if reviewers teach authors how to write better submissions, and authors teach reviewers how to write more constructive reviews. If you have received a not-ok (unhelpful/offensive) review, please help program committees improve by sharing your not-ok reviews on social media.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even with stellar scientific content, a poorly composed paper is more likely to be rejected and later resubmitted, incurring additional work for both authors and future reviewers.
“Even with stellar scientific content, a poorly composed paper is more likely to be rejected and later resubmitted (Church, 2020), incurring additional work for both authors and future reviewers.”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Church, 2020
In §2.1.1 (page unavailable), Church describes a paper with better content receiving a lower score than one with better English; §2.1.2 says nonnative English is sometimes used as an excuse for rejection. Section 2.4 discusses repeated submissions, but attributes them to unpredictable reviewing, not poor composition. The paper therefore supports the risk of rejection despite strong content, but not the claimed composition-to-resubmission sequence. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605 · see citing works
Fox, C. W., Albert, A. Y. K., & Vines, T. H. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0027-x · Cited by 109 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- Chauvin A., Ravaud P., Baron G., Barnes C. and Boutron I. (2015). The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors. BMC Medicine 13(1), 158.
- Church K. (2005). Reviewing the reviewers. Computational Linguistics 31(4), 575-578.
- Church K. (2017). Emerging trends: Inflation. Natural Language Engineering 23(5), 807-812, Cambridge University Press.
- Fox M.F. (1994). Scientific misconduct and editorial and peer review processes. The Journal of Higher Education 65(3), 298-309.
- Gropp R., Glisson S., Gallo S. and Thompson L. (2017). Peer review: A system under stress. BioScience 67(5), 407-410.
- Klovanis M., Porcher R., Ravaud P. and Trinquart L. (2016) The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise. PloS One, e0166387.
- List B. (2017). Crowd-based peer review can be good and fast. Nature News 546(7656), 9.
- Marcus M., Santorini B. and Marcinkiewicz M.A. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19(2), 313-330.
- Moher D., Galipeau J., Alam S. et al. (2017). Core competencies for scientific editors of biomedical journals: Consensus statement. BMC Medicine 15(1), 1741-7015.
- Smith R. (2010). Classical peer review: An empty gun. Breast Cancer Research 12(4), S13.
- Stelmakh I., Shah N. and Singh A. (2019). On testing for biases in peer review. In Advances in Neural Information Processing Systems (NeurIPS), vol. 32, pp. 5287-5297.
- Tennant J., Dugan J. et al. (2017). A multi-disciplinary perspective on emergent and future innovations in peer review. F1000Research, 6, Faculty of 1000 Ltd.
- Van de Walle, F. (1986). Infant Mortality and the European Demographic Transition. Princeton, New Jersey: Princeton University Press.
- Webber B. (2007). Breaking news: Changing attitudes and practices. Computational Linguistics 33(4), 607-611, MIT Press, Cambridge, MA.
- Yarowsky D. and Radu F. (1999). Taking the load off the conference chairs-towards a digital paper-routing assistant. In EMNLP. Cite this article: Church KW (2020). Emerging trends: Reviewing the reviewers (again). Natural Language Engineering 26, 245-257. https://doi.org/10.1017/S1351324920000030
Cited by
Sorted by citations
Church, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28 · see citing works
Self-citationWhat Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
- Authors
- Church, K. W.
- Venue
- Natural Language Engineering
- Publisher
- Cambridge University Press (CUP)
- Published
- 2020-03
- Volume
- 26
- Issue
- 2
- Pages
- 245-257
- ISSN
- 1351-3249
- 1469-8110
- Cited by
- 6
- References
- 3
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
