Does My Rebuttal Matter? Insights from a Major NLP Conference
Open accessAbstract
Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, Yusuke Miyao. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing peer-review datasets are used in the vast majority to predict acceptance and scores or citations.
“They compare existing peer-review datasets and their applications; in the vast majority, the existing items are used to predict acceptance and scores ( [Gao et al., 2019, Kang et al., 2018]) or citations ( [Plank and van Dalen, 2019]).”
From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Gao et al., 2019, Kang et al., 2018]
The available passages from our copy are bibliographic records, not the paper’s discussion of peer-review datasets (no section or page is available). Searches for dataset applications and prediction of acceptance or scores returned no relevant passage. The citation to Plank and van Dalen, rather than Gao et al., is attached to the separate citations claim. (LLM, conf=0.70)
Experimental — accuracy not yet measuredRebuttal generation aims to produce author responses that directly and faithfully address reviewer critiques.
“Rebuttal generation aims to produce author responses that directly and faithfully address reviewer critiques (Gao et al., 2019;Huang et al., 2023).”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Gao et al., 2019
The retrieved passages are bibliographic records and a landing-page citation, with no substantive paper text or section/page addressing rebuttal generation. Searches for the claimed aim and for author responses to reviewer critiques found no passage establishing that aim in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2004) · Cited by 9,718 · see citing works
Degroot, M. H. (1974) · Journal of the American Statistical Association · doi:10.1080/01621459.1974.10480137 · Cited by 4,004 · see citing works
Accurate methods for the statistics of surprise and coincidence
Dunning, T. E. (1993) · Cited by 2,678 · see citing works
Effects of Group Pressure Upon the Modification and Distortion of Judgments
Asch, S. E. (1961) · Documents of Gestalt Psychology · doi:10.1525/9780520313514-017 · Cited by 2,644 · see citing works
Opinion Dynamics and Bounded Confidence Models, Analysis and Simulation
Hegselmann, R., & Krause, U. (2002) · RePEc: Research Papers in Economics · Cited by 2,394 · see citing works
Mixing beliefs among interacting agents
Deffuant, G., Neau, D., Amblard, F., & Weisbuch, G. (2000) · Advances in Complex Systems · doi:10.1142/s0219525900000078 · Cited by 2,165 · see citing works
Consensus Problems on Networks With Antagonistic Interactions
Altafini, C. (2012) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2012.2224251 · Cited by 2,118 · see citing works
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497 · see citing works
How social influence can undermine the wisdom of crowd effect
Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1008636108 · Cited by 1,017 · see citing works
Public goods experiments without confidentiality: a glimpse into fund-raising
Andreoni, J., & Petrie, R. (2003) · Journal of Public Economics · doi:10.1016/s0047-2727(03)00040-9 · Cited by 764 · see citing works
Opinion Dynamics and Learning in Social Networks
Acemoglu, D., & Ozdaglar, A. (2011) · Dynamic Games and Applications · doi:10.1007/s13235-010-0004-1 · Cited by 705 · see citing works
Social Influence and the Collective Dynamics of Opinion Formation
Moussaïd, M., Kämmer, J. E., Analytis, P. P., & Neth, H. (2013) · PLoS ONE · doi:10.1371/journal.pone.0078433 · Cited by 407 · see citing works
Effect on the Quality of Peer Review of Blinding Reviewers and Asking Them to Sign Their Reports
Godlee, F., Gale, C. R., & Martyn, C. N. (1998) · JAMA · doi:10.1001/jama.280.3.237 · Cited by 338 · cited 1× · see citing works
“Birukou et al. (2011) and Kovanis et al. (2017) discuss the shortcomings of the review process in general, such as its inability to detect major flaws in papers (Godlee et al., 1998) and its ineffectiveness in selecting papers that will have high citation counts in the future (Ragone et al., 2013).”
BackgroundA Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 171 · cited 5× · see citing works
“In line with Kang et al. (2018), we first assess the impact of individual aspect scores on the overall score by measuring their Pearson correlation, illustrated in Fig.”
BackgroundMethodResultInfluentialOpinion Dynamics Under Conformity
Buechel, B., Hellmann, T., & Klößner, S. (2012) · Journal of Economic Dynamics and Control · doi:10.1016/j.jedc.2014.12.006 · Cited by 102 · see citing works
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 1× · see citing works
“Hua et al. (2019) compile a corpus consisting of 14.2k reviews from major NLP and machine learning conferences.”
MethodFast and Accurate Prediction of Sentence Specificity
Li, J., & Nenkova, A. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9517 · Cited by 73 · see citing works
Kovanis, M., Trinquart, L., Ravaud, P., & Porcher, R. (2017) · Scientometrics · doi:10.1007/s11192-017-2375-1 · Cited by 69 · see citing works
The arbitrariness of reviews, and advice for school administrators
Langford, J., & Guzdial, M. (2015) · Communications of the ACM · doi:10.1145/2732417 · Cited by 49 · see citing works
On peer review in computer science: analysis of its effectiveness and suggestions for improvement
Ragone, A., Мирыленка, К., Casati, F., & Marchese, M. (2013) · Scientometrics · doi:10.1007/s11192-013-1002-z · Cited by 46 · see citing works
Reviewing Reviews: An Evaluation of Peer Reviews of Journal Article Submissions
Falkenberg, L. J., & Soranno, P. A. (2018) · Limnology and Oceanography Bulletin · doi:10.1002/lob.10217 · Cited by 38 · see citing works
Opinion dynamics and wisdom under out-group discrimination
Eger, S. (2016) · Mathematical Social Sciences · doi:10.1016/j.mathsocsci.2016.02.005 · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
2 of the 24 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Can We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96 · see citing works
APE: Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning
Cheng, L., Bing, L., Yu, Q., Lu, W., & Si, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.569 · Cited by 58 · see citing works
Automated scholarly paper review: Concepts, technologies, and challenges
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101830 · Cited by 37 · see citing works
Personal Bias in Prediction of Emotions Elicited by Textual Opinions
Miłkowski, P., Gruza, M., Kanclerz, K., Kazienko, P., Grimling, D., & Kocoń, J. (2021) · doi:10.18653/v1/2021.acl-srw.26 · Cited by 34 · see citing works
(2020) · doi:10.18653/v1/2020.findings-emnlp.112 · Cited by 32 · see citing works
Controversy and Conformity: from Generalized to Personalized Aggressiveness Detection
Kanclerz, K., Figas, A., Gruza, M., Kajdanowicz, T. J., Kocoń, J., Puchalska, D., & Kazienko, P. (2021) · doi:10.18653/v1/2021.acl-long.460 · Cited by 32 · see citing works
Kumar, S., Ghosal, T., Bharti, P. K., & Ekbal, A. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00081 · Cited by 19 · see citing works
Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation
Deng, Z., Peng, H., Xia, C., Li, J., He, L., & Yu, P. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.555 · Cited by 19 · see citing works
The influence of opening up peer review on the citations of journal articles
Ni, J., Zhao, Z.-Y., Shao, Y., Liu, S., Li, W., Zhuang, Y., Qu, J.-M., Cao, Y., Lian, N., & Li, J. (2021) · Scientometrics · doi:10.1007/s11192-021-04182-9 · Cited by 19 · see citing works
DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Kennard, N., O’Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., & McCallum, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.89 · Cited by 17 · see citing works
Bharti, P. K., Ghosal, T., Agarwal, M., & Ekbal, A. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00375-0 · Cited by 19 · see citing works
A large scale randomized controlled trial on herding in peer-review discussions
Stelmakh, I., Rastogi, C., Shah, N. B., Singh, A., & Daum'e, H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0287443 · Cited by 26 · see citing works
Wu, W., Xi, H., & Zhang, C. (2024) · Scientometrics · doi:10.1007/s11192-024-05070-8 · Cited by 12 · see citing works
MOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11 · see citing works
“Most of the papers in PeerRead are collected together with their review comments, and some of them also have aspect specific scores such as appropriateness, clarity, and originality.”
BackgroundPolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews
Bharti, P. K., Navlakha, M., Agarwal, M., & Ekbal, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09662-3 · Cited by 12 · see citing works
What have we learned from OpenReview?
Wang, G., Peng, Q., Zhang, Y., & Zhang, M. (2022) · World Wide Web · doi:10.1007/s11280-022-01109-z · Cited by 9 · see citing works
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
Self-citationBharti, P. K., Agarwal, M., & Ekbal, A. (2024) · Scientometrics · doi:10.1007/s11192-024-04938-z · Cited by 6 · see citing works
Determinants of grader agreement: an analysis of multiple short answer corpora
Padó, U., & Padó, S. (2021) · Language Resources and Evaluation · doi:10.1007/s10579-021-09547-3 · Cited by 6 · see citing works
An Analysis of Tasks and Datasets in Peer Reviewing
Staudinger, M., Kusa, W., Piroi, F., & Hanbury, A. (2024) · Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024) · doi:10.18653/v1/2024.sdp-1.24 · Cited by 6 · see citing works
Real-Time, Evidence-Based Alerts for Protection From Phishing Attacks
Baki, S., Qachfar, F. Z., Verma, R. M., Kennedy, R., & N.Jones, D. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3425908 · Cited by 5 · see citing works
Between acceptance and rejection
Fernandes, G. L., & de Melo, P. O. S. V. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530935 · Cited by 5 · see citing works
Did AI get more negative recently?
Beese, D., Altunbaş, B., Güzeler, G., & Eger, S. (2023) · Royal Society Open Science · doi:10.1098/rsos.221159 · Cited by 4 · see citing works
Self-citationSome Ethical Issues in the Review Process of Machine Learning ConferencesPreprint
Russo, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00810 · Cited by 4 · see citing works
Huang, J., Huang, W.-B., Bu, Y., Cao, Q., Shen, H., & Cheng, X. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101427 · Cited by 4 · see citing works
Testing for reviewer anchoring in peer review: A randomized controlled trial
Liu, R., Jecmen, S., Conitzer, V., Fang, F., & Shah, N. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301111 · Cited by 4 · see citing works
Enhancing the examination of obstacles in an automated peer review system
Fernandes, G. L., & de Melo, P. O. S. V. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00382-1 · Cited by 3 · see citing works
Computer Simulations of Scientific Peer Reviewing
Hallé, S. (2021) · IEEE Access · doi:10.1109/access.2021.3103837 · Cited by 3 · see citing works
Dershowitz, N., & Verma, R. M. (2023) · Communications of the ACM · doi:10.1145/3584664 · Cited by 3 · see citing works
Overview of Argumentative Text Understanding for AI Debater Challenge
Yuan, J., Cheng, L., He, R., Li, Y., Bing, L., Wei, Z., Liu, Q., Shen, C., Zhang, S., Sun, C., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_45 · Cited by 3 · see citing works
What Have We Learned from OpenReview?Preprint
Wang, G., Peng, Q., Zhang, Y., & Zhang, M. (2022) · Research Square · doi:10.21203/rs.3.rs-1558902/v1 · Cited by 3 · see citing works
Han, R., Zhou, H., Zhong, J., & Zhang, C. (2025) · Data and Information Management · doi:10.1016/j.dim.2025.100105 · Cited by 2 · see citing works
Fair or flawed? Rethinking grant review with generative AI
Sikimić, V. (2025) · Synthese · doi:10.1007/s11229-025-05366-z · Cited by 2 · see citing works
Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals
Purkayastha, S., Lauscher, A., & Gurevych, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.894 · Cited by 5 · see citing works
Self-citationARGUABLY @ AI Debater-NLPCC 2021 Task 3: Argument Pair Extraction from Peer Review and Rebuttals
Kohli, G. S., Kaur, P., Singh, M., Ghosal, T., & Rana, P. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_48 · Cited by 2 · see citing works
Szumega, J., Bougueroua, L., Gkotse, B., Jouvelot, P., & Ravotti, F. (2023) · arXiv · doi:10.48550/arxiv.2312.04576 · Cited by 1 · see citing works
“They compare existing peer-review datasets and their applications; in the vast majority, the existing items are used to predict acceptance and scores ([Gao et al., 2019, Kang et al., 2018]) or citations ([Plank and van Dalen, 2019]).”
MethodNeurReview: A Neural Architecture Based Conformity Prediction of Peer Reviews
Meng, J. (2022) · IEEE Access · doi:10.1109/access.2022.3224019 · Cited by 1 · see citing works
Gao, S., Zhang, H., Chen, X., Tao, C., Zhao, D., & Yan, R. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3243602 · Cited by 1 · see citing works
RottenReviews: Benchmarking Review Quality with Human and LLM-Based Judgments
Ebrahimi, S., Sadeghian, S., Ghorbanpour, A., Arabzadeh, N., Salamat, S., Li, M., Le, H. S., Bashari, M., & Bagheri, E. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761506 · Cited by 2 · see citing works
ACE: A Context-Enhanced Model for Interactive Argument Pair Identification
Wu, Y., & Liu, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_46 · Cited by 1 · see citing works
Transformative research focus considered harmful
Cooper, M., & Licato, J. (2022) · AI Magazine · doi:10.1002/aaai.12063 · Cited by 1 · see citing works
Arora, H., Shinde, K., & Ghosal, T. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00015 · Cited by 2 · see citing works
Does opening up peer review benefit science in terms of citations?
Ni, J., Zhao, Z.-Y., Shao, Y., Liu, S., Li, W., & Li, J. (2020) · ASIS&T Annual Meeting · doi:10.1002/pra2.258 · Cited by 1 · see citing works
Ranking Scientific Papers Using Preference Learning.Preprint
Dycke, N., Simpson, E., Kuznetsov, I., & Gurevych, I. (2021) · arXiv (Cornell University) · Cited by 1 · see citing works
Self-citationWhat do writing features tell us about AI papers?Preprint
Zhu, Z., Li, B., Xu, Y., & Rudzicz, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06310
What Have We Learned from OpenReview?
Wang, G., Peng, Q., Zhang, Y., & Zhang, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85896-4_6
A Dataset for Discourse Structure in Peer Review Discussions.Preprint
Kennard, N. N., O’Gorman, T., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Das, R., Zamani, H., & McCallum, A. (2021) · arXiv (Cornell University)
Kuznetsov, I., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.633
Self-citationBharti, P. K., Agarwal, M., & Ekbal, A. (2025) · Journal of Information Science · doi:10.1177/01655515251353189
RECO-Rank: A Compensatory Multi-Evidence Ranking Model for Cross-Domain Reviewer RecommendationPreprint
Chen, J., Xu, J., & Fang, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9558981/v1
Does Discussion Matter? Interaction Dynamics in Open Peer Review
Wei, S., & Bi, R. (2026) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3785651.3831562
- Authors
- Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y.
- Venue
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Published
- 2019
- Pages
- 1274-1290
- OpenAlex
- W2922604858
- Cited by
- 48
- References
- 24
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 4 · 2023: 13 · 2022: 7 · 2021: 14 · 2020: 4
- FWCI
- 2.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
