CoChat
Sign Up
PreprintCited by 4Sentiment Analysis and Opinion MiningTopic ModelingAdvanced Text Analysis Techniques

Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation

Open access
Zeng, Qi, Sidhu, Mankeerat, Blume, Ansel, Chan, Hou Pong, Wang, Lu, Ji, Heng·2023·arXiv
DOI 10.48550/arxiv.2305.14647arXiv 2305.14647
Open this paper in CoChat — free accountOpen PDF

Abstract

Opinions in scientific research papers can be divergent, leading to controversies among reviewers. However, most existing datasets for opinion summarization are centered around product reviews and assume that the analyzed opinions are non-controversial, failing to account for the variability seen in other contexts such as academic papers, political debates, or social media discussions. To address this gap, we propose the task of scientific opinion summarization, where research paper reviews are synthesized into meta-reviews. To facilitate this task, we introduce the ORSUM dataset covering 15,062 paper meta-reviews and 57,536 paper reviews from 47 conferences. Furthermore, we propose the Checklist-guided Iterative Introspection approach, which breaks down scientific opinion summarization into several stages, iteratively refining the summary under the guidance of questions from a checklist. Our experiments show that (1) human-written summaries do not always satisfy all necessary criteria such as depth of discussion, and identifying consensus and controversy for the specific domain, and (2) the combination of task decomposition and iterative self-refinement shows strong potential for enhancing the opinions and can be applied to other complex text generation using black-box LLMs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.14647

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Checklist-guided generation and decision-aware meta-reviewing offer partial solutions by explicitly linking critique structure to recommendations.

    “Checklist-guided generation (Zeng et al., 2023) and decision-aware meta-reviewing (Kumar et al., 2024a) offer partial solutions by explicitly linking critique structure to recommendations.”

    From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Zeng et al., 2023

  • Zeng et al. (2023) cast the task as scientific opinion summarization.

    “Zeng et al. (2023) cast the task as scientific opinion summarization.”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Zeng et al. (2023)

  • Zeng et al. (2023) propose a checklist-based approach to prompt an LLM to write summaries that fulfill meta-review requirements of self-consistency, faithfulness, and active engagement.

    “They propose a checklist-based approach to prompt an LLM to write summaries that fulfill meta-review requirements of self-consistency, faithfulness, and active engagement.”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Zeng et al. (2023)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 44

  1. Reinald Kim Amplayo and Mirella Lapata. Informative and controllable opinion sum- marization. CoRR, abs/1909.02322, 2019. · arXiv
  2. Reinald Kim Amplayo and Mirella Lapata. Unsupervised opinion summarization with noising and denoising. In Dan Jurafsky, Joyce Chai, Na- talie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1934-1945. Association for Computational Linguis- tics, 2020.
  3. Amplayo et al., 2021a] Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. Aspect-controllable opin- ion summarization. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Process- ing, pages 6578-6593, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  4. Amplayo et al., 2021b] Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. Unsupervised opinion summarization with content planning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Ar- tificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 12489- 12497. AAAI Press, 2021.
  5. Stefanos Angelidis and Mirella Lapata. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Ellen Riloff, David Chiang, Julia Hock- enmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3675-3686. Association for Computational Linguistics, 2018.
  6. Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. Extractive opinion summarization in quantized transformer spaces. CoRR, abs/2012.04443, 2020. · arXiv
  7. Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. Extractive opinion summarization in quantized transformer spaces. Transactions of the Association for Computational Linguistics, 9:277-293, 2021.
  8. Iz Beltagy, Matthew E. Peters, and Ar- man Cohan. Longformer: The long-document trans- former. CoRR, abs/2004.05150, 2020. · arXiv
  9. Adithya Bhaskar, Alexander R. Fab- bri, and Greg Durrett. Zero-shot opinion summarization with GPT-3. CoRR, abs/2211.15914, 2022. · arXiv
  10. Chaitanya Bhatia, Tribikram Pradhan, and Sukomal Pal. Metagen: An academic meta-review generation system. In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1653-1656. ACM, 2020.
  11. Brazinskas et al., 2020a] Arthur Brazinskas, Mirella Lap- ata, and Ivan Titov. Few-shot learning for opinion sum- marization. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2020, Online, November 16-20, 2020, pages 4119-4135. Association for Computational Linguis- tics, 2020.
  12. Brazinskas et al., 2020b] Arthur Brazinskas, Mirella Lap- ata, and Ivan Titov. Unsupervised opinion summariza- tion as copycat-review generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5151-5169. Association for Computational Linguistics, 2020.
  13. Arthur Brazinskas, Ramesh Nalla- pati, Mohit Bansal, and Markus Dreyer. Efficient few-shot fine-tuning for opinion summarization. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz, editors, Findings of the Association for Compu- tational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1509-1523. Association for Computational Linguistics, 2022.
  14. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win- ter, Christopher Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In Hugo Larochelle, Marc'Aurelio Ran- zato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  15. Hou Pong Chan, Wang Chen, and Irwin King. A unified dual-view model for review summariza- tion and sentiment classification with inconsistency loss. In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM SIGIR con- ference on research and development in Information Re- trieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1191-1200. ACM, 2020.
  16. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue sum- marization dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Associa- tion for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 5062-5074. Association for Computational Linguistics, 2021.
  17. Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. APE: argument pair extraction from peer review and rebuttal via multi-task learning. In Bon- nie Webber, Trevor Cohn, Yulan He, and Yang Liu, ed- itors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7000-7011. Asso- ciation for Computational Linguistics, 2020.
  18. Somnath Basu Roy Chowdhury, Chao Zhao, and Snigdha Chaturvedi. Unsupervised ex- tractive opinion summarization using sparse coding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1209-1225. Association for Computational Linguistics, 2022.
  19. Eric Chu and Peter J. Liu. Meansum: A neural model for unsupervised multi-document abstrac- tive summarization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1223- 1232. PMLR, 2019.
  20. Zhongfen Deng, Hao Peng, Congying Xia, Jianxin Li, Lifang He, and Philip S. Yu. Hierarchi- cal bi-directional self-attention networks for paper review rating recommendation. In Donia Scott, Núria Bel, and Chengqing Zong, editors, Proceedings of the 28th Interna- tional Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6302-6314. International Committee on Computa- tional Linguistics, 2020.
  21. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. NLPeer: A unified resource for the compu- tational study of peer review. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 5049- 5073, Toronto, Canada, July 2023. Association for Com- putational Linguistics.
  22. Hady Elsahar, Maximin Coavoux, Jos Rozen, and Matthias Gallé. Self-supervised and controlled multi-document opinion summarization. In Proceedings of the 16th Conference of the European Chapter of the As- sociation for Computational Linguistics: Main Volume, pages 1646-1662, Online, April 2021. Association for Computational Linguistics.
  23. Korbinian Friedl, Georgios Rizos, Lukas Stappen, Madina Hasan, Lucia Specia, Thomas Hain, and Björn W. Schuller. Uncertainty aware review hallucina- tion for science article classification. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Find- ings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, vol- ume ACL/IJCNLP 2021 of Findings of ACL, pages 5004- 5009. Association for Computational Linguistics, 2021.
  24. Fu et al., 2023] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. CoRR, abs/2302.04166, 2023. · arXiv
  25. Gao et al., 2023] Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human- like summarization evaluation with chatgpt. CoRR, abs/2304.02554, 2023. · arXiv
  26. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of GPT-3. CoRR, abs/2209.12356, 2022. · arXiv
  27. Gu et al., 2022] Nianlong Gu, Elliott Ash, and Richard Hahnloser. MemSum: Extractive summarization of long documents using multi-step episodic Markov decision pro- cesses. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6507-6522, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  28. Hu and Liu, 2006] Minqing Hu and Bing Liu. Opinion extraction and summarization on the web. In Proceed- ings, The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachusetts, USA, pages 1621-1624. AAAI Press, 2006.
  29. Hayate Iso, Xiaolan Wang, Yoshihiko Suhara, Stefanos Angelidis, and Wang-Chiew Tan. Con- vex aggregation for opinion summarization. CoRR, abs/2104.01371, 2021. · arXiv
  30. Masaru Isonuma, Junichiro Mori, Danushka Bollegala, and Ichiro Sakata. Unsupervised ab- stractive opinion summarization by generating sentences with tree-structured topic guidance. Transactions of the Association for Computational Linguistics, 9:945-961, 2021.
  31. Wojciech Kryscinski, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. CoRR, abs/1910.12840, 2019. · arXiv
  32. Asheesh Kumar, Tirthankar Ghosal, and Asif Ekbal. A deep neural architecture for decision- aware meta-review generation. In J. Stephen Downie, Dana McKay, Hussein Suleman, David M. Nichols, and Faryaneh Poursardar, editors, ACM/IEEE Joint Confer- ence on Digital Libraries, JCDL 2021, Champaign, IL, USA, September 27-30, 2021, pages 222-225. IEEE, 2021.
  33. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. Summac: Re- visiting nli-based models for inconsistency detection in summarization. CoRR, abs/2111.09525, 2021. · arXiv
  34. Li et al., 2017] Piji Li, Zihao Wang, Zhaochun Ren, Lidong Bing, and Wai Lam. Neural rating regression with abstrac- tive tips generation for recommendation. In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, editors, Proceedings of the 40th Inter- national ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 345-354. ACM, 2017.
  35. Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  36. Yang Liu, Dan Iter, Yichong Xu, Shuo- hang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human align- ment. CoRR, abs/2303.16634, 2023. · arXiv
  37. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. Mred: A meta- review dataset for structure-controllable text generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio, editors, Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2521-2535. Association for Computational Linguis- tics, 2022.
  38. Ming Shen, Jie Ma, Shuai Wang, Yog- arshi Vyas, Kalpit Dixit, Miguel Ballesteros, and Yas- sine Benajiba. Simple yet effective synthetic dataset con- struction for unsupervised opinion summarization. CoRR, abs/2303.11660, 2023. · arXiv
  39. Yoshihiko Suhara, Xiaolan Wang, Ste- fanos Angelidis, and Wang-Chiew Tan. Opiniondigest: A simple framework for opinion summarization. In Dan Ju- rafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pages 5789-5798. Association for Computational Linguistics, 2020.
  40. Lu Wang and Wang Ling. Neural network-based abstract generation for opinions and argu- ments. In Kevin Knight, Ani Nenkova, and Owen Ram- bow, editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 47- 57. The Association for Computational Linguistics, 2016.
  41. Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. Re- viewrobot: Explainable paper review generation based on knowledge synthesis. In Brian Davis, Yvette Graham, John D. Kelleher, and Yaji Sripada, editors, Proceedings of the 13th International Conference on Natural Language Generation, INLG 2020, Dublin, Ireland, December 15- 18, 2020, pages 384-397. Association for Computational Linguistics, 2020.
  42. Wen Xiao and Giuseppe Carenini. Systematically exploring redundancy re- duction in summarizing long documents. In Kam-Fai Wong, Kevin Knight, and Hua Wu, editors, Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Pro- cessing, AACL/IJCNLP 2020, Suzhou, China, December 4-7, 2020, pages 516-528. Association for Computational Linguistics, 2020.
  43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
  44. Wei Zhao, Michael Strube, and Steffen Eger. Discoscore: Evaluating text generation with BERT and discourse coherence. CoRR, abs/2201.11176, 2022. · arXiv

Cited by

Sorted by citations

Authors
Zeng, Q., Sidhu, M., Blume, A., Chan, H. P., Wang, L., & Ji, H.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Alibaba DAMO Academy (China) (CN)
  • University of Michigan (US)
Corresponding author
Sidhu, Mankeerat
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
20-38
ISSN
0302-9743 (ISSN-L)
DOI
10.48550/arxiv.2305.14647
arXiv
2305.14647
OpenAlex
W4407411788
Cited by
2
References
0
Citations by year
2026: 2
FWCI
1.90
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2019897, HR001121C0165, 2034562, 2229873)
  • U.S. Department of Education (2229873)
  • Defense Advanced Research Projects Agency (FA8750, FA8750-18-2-0014, HR001121C0165)
  • Institute of Education Sciences (2229873)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
task decomposition, large language models, text summarization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review