COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews
Open accessAbstract
Comparing research papers is a conventional method to demonstrate progress in experimental research. We present COMPARE, a taxonomy and a dataset of comparison discussions in peer reviews of research papers in the domain of experimental deep learning. From a thorough observation of a large set of review sentences, we build a taxonomy of categories in comparison discussions and present a detailed annotation scheme to analyze this. Overall, we annotate 117 reviews covering 1,800 sentences. We experiment with various methods to identify comparison sentences in peer reviews and report a maximum F1 Score of 0.49. We also pretrain two language models specifically on ML, NLP, and CV paper abstracts and reviews to learn informative representations of peer reviews. The annotated dataset and the pretrained models are available at https://github.com/shruti-singh/COMPARE.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 171 · cited 1× · see citing works
“Several works focus on curation of peer review data [3], and further analysis such as sentiment analysis of reviews and decision prediction [3, 4].”
Background
Flagged first, then by citations
Research works printed but not linked: 14
- S. Chakraborty, P. Goyal, and A. Mukherjee, "Aspect- based sentiment analysis of scientific reviews," Pro- ceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, 2020.
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "RoBERTa: A robustly optimized bert pretraining ap- proach," ArXiv, vol. abs/1907.11692, 2019. · arXiv
- T. Ghosal, R. K. Verma, A. Ekbal, and P. Bhattacharyya, "DeepSentiPeer: Harnessing sentiment in review texts to recommend peer review decisions," in ACL, 2019, pp. 1120-1130.
- X. Hua, M. Nikolov, N. Badugu, and L. Wang, "Argu- ment mining for understanding peer reviews," in NAACL- HLT, Jun. 2019.
- M. Fromm, E. Faerman, M. Berrendorf, S. Bhargava, R. Qi, Y. Zhang, L. Dennert, S. Selle, Y. Mao, and T. Seidl, "Argument mining driven analysis of peer- reviews," ArXiv, vol. abs/2012.07743, 2020. · arXiv
- I. Fortanet, "Evaluative language in peer review referee reports," Journal of English for Academic Purposes, vol. 7, pp. 27-37, 2008.
- H. Gosden, "'why not give us the full story?': functions of referees' comments in peer reviews of scientific re- search papers," Journal of English for Academic Pur- poses, vol. 2, no. 2, pp. 87-101, 2003.
- P. Mungra and P. Webber, "Peer review process in medical research publications: Language and content comments," English for Specific Purposes, vol. 29, no. 1, pp. 43-53, 2010.
- F. C. Day, D. L. Schriger, C. Todd, and R. L. Wears, "The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers," Annals of Emergency medicine, vol. 40, no. 3, pp. 329- 333, 2002.
- D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar et al., "Universal sentence encoder," arXiv preprint arXiv:1803.11175, 2018. · arXiv
- I. Beltagy, K. Lo, and A. Cohan, "Scibert: A pretrained language model for scientific text," in EMNLP/IJCNLP, 2019.
- D. R. Radev, P. Muthukrishnan, V. Qazvinian, and A. Abu-Jbara, "The ACL anthology network corpus," Language Resources and Evaluation, vol. 47, no. 4, pp. 919-944, 2013.
- Y. Hou, C. Jochim, M. Gleize, F. Bonin, and D. Ganguly, "Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction," in Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, 2019.
- S. Jain, M. van Zuylen, H. Hajishirzi, and I. Beltagy, "Scirex: A challenge dataset for document-level infor- mation extraction," in ACL, 2020.
Cited by
Sorted by citations
MOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11 · see citing works
“It is composed of papers of NeurIPS, their review comments, and the number of citations.”
Background
- Authors
- Singh, S., Singh, M., & Goyal, P.
- Affiliations
- Indian Institute of Technology Gandhinagar (IN)
- Indian Institute of Technology Kharagpur (IN)
- Venue
- 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL)
- Publisher
- IEEE
- Published
- 2021-09
- Pages
- 238-241
- ISSN
- 1552-5996 (ISSN-L)
- OpenAlex
- W3193246448
- Cited by
- 7
- References
- 2
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 3 · 2021: 1
- FWCI
- 0.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM/IEEE Joint Conference on Digital Libraries (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- peer review, taxonomy (general classification scheme), annotation scheme, language model pretraining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
