CoChat
Sign Up
Cited by 1scientometrics and bibliometrics researchAcademic Writing and PublishingDigital Humanities and Scholarship

CASIMIR: A Corpus of Scientific Articles Enhanced with Multiple Author-Integrated Revisions

Open access
Jourdan, Léane, Boudin, Florian, Hernandez, Nicolas, Dufour, Richard·2024·Proceedings of the Language Resources and Evaluation Conference
DOI 10.63317/4aa2twhyqaqa
Open this paper in CoChat — free accountOpen PDF

Abstract

Writing a scientific article is a challenging task as it is a highly codified and specific genre, consequently proficiency in written communication is essential for effectively conveying research findings and ideas.In this article, we propose an original textual resource on the revision step of the writing process of scientific articles.This new dataset, called CASIMIR, contains the multiple revised versions of 15,646 scientific articles from OpenReview, along with their peer reviews.Pairs of consecutive versions of an article are aligned at sentence-level while keeping paragraph location information as metadata for supporting future revision studies at the discourse level.Each pair of revised sentences is enriched with automatically extracted edits and associated revision intention.To assess the initial quality on the dataset, we conducted a qualitative study of several state-of-the-art text revision approaches and compared various evaluation metrics.Our experiments led us to question the relevance of the current evaluation methods for the text revision task.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.63317/4aa2twhyqaqa

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CASIMIR compiles and analyzes papers and their reviews to understand the intent behind edits, which helps plan and evaluate future revisions.

    “CASIMIR (Jourdan et al., 2024) compiles and analyzes papers and their reviews to understand the intent behind edits, which helps plan and evaluate future revisions.”

    From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Jourdan et al., 2024

    The Introduction says CASIMIR contains multiple article versions, associated peer reviews, and edits labeled with revision intentions. Corpus Analysis (§4) examines edits and counts reviews, while Experiments (§5) uses the corpus to evaluate revision models. The paper does not say it analyzes review content to determine edit intentions or show that the corpus helps plan future revisions. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. Stephen Bailey. 2014. Academic writing: A hand- book for international students. Routledge.
  2. Samir Bourekkache. 2022. English for specific pur- poses: writing scientific research papers. case study: Phd students in the computer science de- partment. Master's thesis, University of Biskra, Algeria.
  3. Leshem Choshen and Omri Abend. 2018. Au- tomatic metric validation for grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1372-1382, Melbourne, Australia. Association for Computational Linguistics. · DOI
  4. Wanyu Du, Zae Myung Kim, Vipul Runderstanda- heja, Dhruv Kumar, and Dongyeop Kang. 2022a. Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 96-108, Dublin, Ireland. Associa- tion for Computational Linguistics. · DOI
  5. Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022b. Understanding iterative revision from human-written text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3573-3590, Dublin, Ireland. Association for Computational Linguistics. · DOI
  6. Jane Dwivedi-Yu, Timo Schick, Zhengbao Jiang, Maria Lomeli, Patrick Lewis, Gautier Izacard, Edouard Grave, Sebastian Riedel, and Fabio Petroni. 2022. Editeval: An instruction-based benchmark for text improvements. arXiv. · DOI
  7. Manaal Faruqui, Ellie Pavlick, Ian Tenney, and Di- panjan Das. 2018. WikiAtomicEdits: A multi- lingual corpus of Wikipedia edits for modeling language and discourse. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing, pages 305-315, Brus- sels, Belgium. Association for Computational Linguistics. · DOI
  8. Arthur C. Graesser, Danielle S. McNamara, and Jonna M. Kulikowich. 2011. Coh-metrix: Pro- viding multilevel analyses of text characteristics. Educational Researcher, 40(5):223-234. · DOI
  9. Takumi Ito, Tatsuki Kuribayashi, Hayato Kobayashi, Ana Brassard, Masato Hagiwara, Jun Suzuki, and Kentaro Inui. 2019. Diamonds in the rough: Generating fluent sentences from early-stage drafts for academic writing assis- tance. In Proceedings of the 12th International Conference on Natural Language Generation, pages 40-53, Tokyo, Japan. Association for Computational Linguistics. · DOI
  10. Elena D Kallestinova. 2011. How to write your first research paper. The Yale journal of biology and medicine, 84(3):181.
  11. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collec- tion, insights and NLP applications. In Pro- ceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers), pages 1647- 1661, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
  12. Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. Revise and Resubmit: An Intertextual Model of Text-based Collabora- tion in Peer Review. Computational Linguistics, 48(4):949-986. · DOI · Paper page
  13. Ekaning Dewanti Laksmi. 2006. "scaffolding" stu- dents' writing in efl class: Implementing process approach. TEFLIN Journal, 17(2):144-156. · DOI
  14. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summa- rization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguis- tics.
  15. Lei Liu and Min Zhu. 2022. Bertalign: Im- proved word embedding-based sentence align- ment for Chinese-English parallel corpora of lit- erary texts. Digital Scholarship in the Humani- ties, 38(2):621-634. · DOI
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for auto- matic evaluation of machine translation. In Pro- ceedings of the 40th Annual Meeting of the As- sociation for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. As- sociation for Computational Linguistics. · DOI · Paper page
  17. Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. CoEdIT: Text editing by task-specific instruction tuning. In Findings of the Association for Computational Linguis- tics: EMNLP 2023, pages 5274-5291, Singa- pore. Association for Computational Linguistics. · DOI
  18. Zejiang Shen, Kyle Lo, Lucy Lu Wang, Bailey Kuehl, Daniel S. Weld, and Doug Downey. 2022. VILA: Improving structured content extraction from scientific PDFs using visual layout groups. Transactions of the Association for Computa- tional Linguistics, 10:376-392. · DOI
  19. John M. Swales. 1990. Genre Analysis: English in academic and research settings. The Cam- bridge applied linguistics series. The press syn- dicate of the University of Cambridge.
  20. Zeqiu Wu, Michel Galley, Chris Brockett, Yizhe Zhang, and Bill Dolan. 2021. Automatic docu- ment sketching: Generating drafts from analo- gous texts. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2102-2113, Online. Association for Com- putational Linguistics. · DOI
  21. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimiz- ing statistical machine translation for text simplifi- cation. Transactions of the Association for Com- putational Linguistics, 4:401-415. · DOI
  22. Diyi Yang, Aaron Halfaker, Robert Kraut, and Ed- uard Hovy. 2017. Identifying semantic edit in- tentions from revisions in Wikipedia. In Pro- ceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2000-2010, Copenhagen, Denmark. As- sociation for Computational Linguistics. · DOI
  23. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kil- ian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Repre- sentations.

Cited by

Sorted by citations

Authors
Jourdan, L., Boudin, F., Hernandez, N., & Dufour, R.
Affiliations
École Centrale de Nantes (FR)
ORCID
Venue
Proceedings of the Language Resources and Evaluation Conference
Publisher
European Language Resources Association (ELRA) and ICCL
Published
2024-05
Pages
2883-2892
ISSN
2522-2686
DOI
10.63317/4aa2twhyqaqa
OpenAlex
W7131790615
Cited by
0
References
5
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Language Resources and Evaluation Conference (LREC) (publishedVersion, cc-by)
Funders
  • Agence Nationale de la Recherche (ANR-22-CE38-0004)
  • Grand Équipement National De Calcul Intensif
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
Keywords
scientific articles, collaborative writing, revision history

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review