Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond
Open accessAbstract
The shift towards publicly available text sources has enabled language processing at unprecedented scale, yet leaves under-serviced the domains where public and openly licensed data is scarce.Proactively collecting text data for research is a viable strategy to address this scarcity, but lacks systematic methodology taking into account the many ethical, legal and confidentiality-related aspects of data collection.Our work presents a case study on proactive data collection in peer reviewa challenging and under-resourced NLP domain.We outline ethical and legal desiderata for proactive data collection and introduce "Yes-Yes-Yes", the first donation-based peer reviewing data collection workflow that meets these requirements.We report on the implementation of Yes-Yes-Yes at ACL Rolling Review 1 and empirically study the implications of proactive data collection for the dataset size and the biases induced by the donation behavior on the peer reviewing platform.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For donation-based datasets (all except F1000-22), the positive skew is likely due to participation bias.
“For the donation-based datasets (all except F1000-22), this is likely due to participation bias (Dycke et al., 2022);”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Dycke et al., 2022
Section 6.2 (page not supplied) finds that reviewers’ donation decisions did not introduce substantial polarity bias and authors’ donation decisions introduced no substantial further positive bias. It identifies conditioning on paper acceptance, rather than donation participation, as the major source of positive score bias in its dataset. The paper does not assess the other datasets named in the claim. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe computational study of peer review lacks a solid data foundation because reviewing data is rarely public and existing sources are not licensed, preventing reuse and redistribution.
“The computational study of peer review lacks a (1) solid data foundation: reviewing data is rarely public and comes with legal and ethical challenges; existing sources of peer reviewing data and the derivative datasets are not licensed, which legally prevents reuse and redistribution (Dycke et al., 2022).”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Dycke et al., 2022
The Introduction says a solid data foundation is lacking, and §2.3 says publicly available peer-review data is scarce and published datasets lack clear licenses. However, §2.3 says the missing licenses leave reuse conditions unspecified; it does not say they legally prevent reuse and redistribution. (LLM, conf=0.88)
Experimental — accuracy not yet measuredOnly reviews and paper drafts for which both authors and reviewers agreed to donation were included, avoiding any possibility of leaking confidential research ideas.
“we only include reviews and paper drafts for which both authors and reviewers agreed to donation; hereby avoiding any possibility of leaking confidential research ideas from the papers.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Dycke et al. (2022)
Section 3Y: Yes by the Authors says authors may donate a draft alone or a draft with reviews whose reviewers agreed to donation. Section 4.1 Design says public release of reviews also requires author permission and is limited to accepted publications. These safeguards address confidentiality, but the paper does not claim they eliminate every possibility of leaking research ideas; it also does not require reviewer agreement for a draft donated alone. (LLM, conf=0.88)
Experimental — accuracy not yet measuredA major challenge in data collection is the availability of public peer reviews with openly licensed scientific articles.
“A major challenge in data collection is the availability of public peer reviews with openly licensed scientific articles (Dycke et al., 2022).”
From PeerQA: A Scientific Question Answering Dataset from Peer Reviews (2025) · cites this paper as Dycke et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 1× · see citing works
“In line with the recent work in responsible data handling for NLP (Rogers et al., 2021; Bender and Friedman, 2018) and previous discussions on ar X iv :2 20 1.”
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · see citing works
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605 · see citing works
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 171 · cited 5× · see citing works
“The ACL-2017 portion of the PeerRead corpus (Kang et al., 2018) includes reviews of submissions to the 55th Annual Meeting of ACL, for which both authors and reviewers agreed to share the review reports.”
BackgroundInfluentialArgument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 4× · see citing works
“…PeerRead – a corpus composed of openly available reports and drafts – and report experiments on paper acceptance and aspect score prediction; Hua et al. (2019) investigate argumentation in peer review reports; Cheng et al. (2020) study the correspondences between review reports and author…”
BackgroundInfluentialPublons' Global State Of Peer Review 2018
Publons (2018) · doi:10.14322/publons.gspr2018 · Cited by 79 · see citing works
APE: Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning
Cheng, L., Bing, L., Yu, Q., Lu, W., & Si, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.569 · Cited by 58 · see citing works
Bandy, J., & Vincent, N. (2021) · arXiv · doi:10.48550/arxiv.2105.05241 · Cited by 36 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics. · DOI · Paper page
- Nils Dycke, Edwin Simpson, Ilia Kuznetsov, and Iryna Gurevych. 2021. Assisting decision making in scholarly peer review: A preference learning per- spective. arXiv preprint arXiv:2109.01190. · arXiv
- Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. Does my re- buttal matter? insights from a major nlp confer- ence. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 1274-1290.
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural lan- guage inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Associa- tion for Computational Linguistics. · DOI · Paper page
- ACL.
- Claire Keeble, Graham Richard Law, Stuart Barber, Paul D Baxter, et al. 2015. Choosing a method to reduce selection bias: a tool for researchers. Open Journal of Epidemiology, 5(3):155-162.
- K. Krippendorff. 1980. Content Analysis: An Introduc- tion To Its Methodology. Sage Publications, Beverly Hills.
- Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2-17. · DOI · Paper page
- Denis Newman-Griffis, Jill Fain Lehman, Carolyn Rosé, and Harry Hochheiser. 2021. Translational NLP: A new paradigm and general principles for nat- ural language processing research. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4125-4138, Online. Association for Computational Linguistics.
- D. B. Paul and J. M. Baker. 1992. The design for the Wall Street Journal-based CSR corpus. In Pro- ceedings IEEE International Conference on Acous- tics, Speech and Signal Processing, pages 899-902, Banff.
- Barbara Plank. 2016. What to do about non- standard (or non-canonical) language in nlp. arXiv:1608.07836. · DOI
- Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. 2013. On peer review in computer science: Analysis of its effectiveness and suggestions for improvement. Scientometrics, 97(2):317-356.
- Philip Resnik, April Foreman, Michelle Kuchuk, Katherine Musacchio Schafer, and Beau Pinkham. 2021. Naturally occurring language as a source of evidence in suicide prevention. Suicide Life Threat. Behav., 51(1):88-96.
- Anna Rogers and Isabelle Augenstein. 2020. What can we do to improve peer review in NLP? In Find- ings of the Association for Computational Linguis- tics: EMNLP 2020, pages 1256-1262, Online. ACL. · DOI
- Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021. 'just what do you think you're doing, dave?' a check- list for responsible data use in NLP. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4821-4833, Punta Cana, Do- minican Republic. Association for Computational Linguistics. · DOI · Paper page
- Robert Slonim, Carmen Wang, Ellen Garbarino, and Danielle Merrett. 2013. Opting-in: Participation bias in economic experiments. Journal of Economic Behavior & Organization, 90:43-70.
- Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daumé III. 2020a. A large scale ran- domized controlled trial on herding in peer-review discussions. arXiv preprint arXiv:2011.15083. · arXiv
- Ivan Stelmakh, Nihar B Shah, and Aarti Singh. 2020b. Catch me if i can: Detecting strategic behaviour in peer assessment. In ICML Workshop on Incentives in Machine Learning.
- Ivan Stelmakh, Nihar B Shah, Aarti Singh, and Hal Daumé III. 2021. Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. Proceedings of the ACM on Human- Computer Interaction, 5(CSCW1):1-17.
- Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171-212.
- Arkaitz Zubiaga. 2018. A longitudinal assessment of the persistence of twitter datasets. Journal of the Association for Information Science and Technology, 69(8):974-984.
Cited by
Sorted by citations
Bauer, E., Greisel, M., Kuznetsov, I., Berndt, M., Kollar, I., Dresel, M., Fischer, M. R., & Fischer, F. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13336 · Cited by 130 · see citing works
Self-citationLiang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07183 · Cited by 73 · see citing works
NLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.277 · Cited by 19 · see citing works
Self-citationWhat Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
Self-citationAn Analysis of Tasks and Datasets in Peer Reviewing
Staudinger, M., Kusa, W., Piroi, F., & Hanbury, A. (2024) · Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024) · doi:10.18653/v1/2024.sdp-1.24 · Cited by 6 · see citing works
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
- Authors
- Dycke, N., Kuznetsov, I., & Gurevych, I.
- Affiliations
- Technische Universität Darmstadt (DE)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Published
- 2022
- Pages
- 300-318
- OpenAlex
- W4385573189
- Cited by
- 10
- References
- 8
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 7
- FWCI
- 2.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- TUbilio (Technical University of Darmstadt) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Bundesministerium für Bildung und Forschung
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- peer review, research ethics, data privacy, data bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
