CoChat
Sign Up
Cited by 2Sentiment Analysis and Opinion MiningTopic ModelingAdvanced Text Analysis Techniques

Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation

Open access
Zeng, Qi, Sidhu, Mankeerat, Blume, Ansel, Chan, Hou Pong, Wang, Lu, Ji, Heng·2025·Lecture Notes in Computer Science
DOI 10.1007/978-981-97-9536-9_2
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-981-97-9536-9_2

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These approaches measure success by matching past reviews via metrics such as ROUGE rather than detecting errors.

    “These approaches measure success by matching past reviews via metrics such as ROUGE [64] rather than detecting errors.”

    From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [64]

    Section 6.1 says ROUGE-L measures similarity between generated and reference texts, supporting the reference-matching part of the claim. But the same section says the paper also uses FactCC and SummaC to assess factual consistency, including inconsistency detection; §7.3 additionally describes human evaluation of hallucinations. Thus “rather than detecting errors” overstates what the source says about its evaluation. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 38 · Cited by 2

Research works printed but not linked: 41

  1. Reinald Kim Amplayo and Mirella Lapata. Informative and controllable opinion sum- marization. CoRR, abs/1909.02322, 2019. · arXiv
  2. Reinald Kim Amplayo and Mirella Lapata. Unsupervised opinion summarization with noising and denoising. In Dan Jurafsky, Joyce Chai, Na- talie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1934-1945. Association for Computational Linguis- tics, 2020.
  3. Amplayo et al., 2021a] Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. Aspect-controllable opin- ion summarization. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Process- ing, pages 6578-6593, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  4. Amplayo et al., 2021b] Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. Unsupervised opinion summarization with content planning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Ar- tificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 12489- 12497. AAAI Press, 2021.
  5. Stefanos Angelidis and Mirella Lapata. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Ellen Riloff, David Chiang, Julia Hock- enmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3675-3686. Association for Computational Linguistics, 2018.
  6. Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. Extractive opinion summarization in quantized transformer spaces. CoRR, abs/2012.04443, 2020. · arXiv
  7. Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. Extractive opinion summarization in quantized transformer spaces. Transactions of the Association for Computational Linguistics, 9:277-293, 2021.
  8. Iz Beltagy, Matthew E. Peters, and Ar- man Cohan. Longformer: The long-document trans- former. CoRR, abs/2004.05150, 2020. · arXiv
  9. Adithya Bhaskar, Alexander R. Fab- bri, and Greg Durrett. Zero-shot opinion summarization with GPT-3. CoRR, abs/2211.15914, 2022. · arXiv
  10. Chaitanya Bhatia, Tribikram Pradhan, and Sukomal Pal. Metagen: An academic meta-review generation system. In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1653-1656. ACM, 2020.
  11. Brazinskas et al., 2020a] Arthur Brazinskas, Mirella Lap- ata, and Ivan Titov. Few-shot learning for opinion sum- marization. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2020, Online, November 16-20, 2020, pages 4119-4135. Association for Computational Linguis- tics, 2020.
  12. Brazinskas et al., 2020b] Arthur Brazinskas, Mirella Lap- ata, and Ivan Titov. Unsupervised opinion summariza- tion as copycat-review generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5151-5169. Association for Computational Linguistics, 2020.
  13. Arthur Brazinskas, Ramesh Nalla- pati, Mohit Bansal, and Markus Dreyer. Efficient few-shot fine-tuning for opinion summarization. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz, editors, Findings of the Association for Compu- tational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1509-1523. Association for Computational Linguistics, 2022.
  14. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win- ter, Christopher Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In Hugo Larochelle, Marc'Aurelio Ran- zato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  15. Hou Pong Chan, Wang Chen, and Irwin King. A unified dual-view model for review summariza- tion and sentiment classification with inconsistency loss. In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM SIGIR con- ference on research and development in Information Re- trieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1191-1200. ACM, 2020.
  16. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue sum- marization dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Associa- tion for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 5062-5074. Association for Computational Linguistics, 2021.
  17. Somnath Basu Roy Chowdhury, Chao Zhao, and Snigdha Chaturvedi. Unsupervised ex- tractive opinion summarization using sparse coding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1209-1225. Association for Computational Linguistics, 2022.
  18. Eric Chu and Peter J. Liu. Meansum: A neural model for unsupervised multi-document abstrac- tive summarization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1223- 1232. PMLR, 2019.
  19. Zhongfen Deng, Hao Peng, Congying Xia, Jianxin Li, Lifang He, and Philip S. Yu. Hierarchi- cal bi-directional self-attention networks for paper review rating recommendation. In Donia Scott, Núria Bel, and Chengqing Zong, editors, Proceedings of the 28th Interna- tional Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6302-6314. International Committee on Computa- tional Linguistics, 2020.
  20. Hady Elsahar, Maximin Coavoux, Jos Rozen, and Matthias Gallé. Self-supervised and controlled multi-document opinion summarization. In Proceedings of the 16th Conference of the European Chapter of the As- sociation for Computational Linguistics: Main Volume, pages 1646-1662, Online, April 2021. Association for Computational Linguistics.
  21. Korbinian Friedl, Georgios Rizos, Lukas Stappen, Madina Hasan, Lucia Specia, Thomas Hain, and Björn W. Schuller. Uncertainty aware review hallucina- tion for science article classification. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Find- ings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, vol- ume ACL/IJCNLP 2021 of Findings of ACL, pages 5004- 5009. Association for Computational Linguistics, 2021.
  22. Fu et al., 2023] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. CoRR, abs/2302.04166, 2023. · arXiv
  23. Gao et al., 2023] Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human- like summarization evaluation with chatgpt. CoRR, abs/2304.02554, 2023. · arXiv
  24. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of GPT-3. CoRR, abs/2209.12356, 2022. · arXiv
  25. Gu et al., 2022] Nianlong Gu, Elliott Ash, and Richard Hahnloser. MemSum: Extractive summarization of long documents using multi-step episodic Markov decision pro- cesses. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6507-6522, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  26. Hu and Liu, 2006] Minqing Hu and Bing Liu. Opinion extraction and summarization on the web. In Proceed- ings, The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachusetts, USA, pages 1621-1624. AAAI Press, 2006.
  27. Hayate Iso, Xiaolan Wang, Yoshihiko Suhara, Stefanos Angelidis, and Wang-Chiew Tan. Con- vex aggregation for opinion summarization. CoRR, abs/2104.01371, 2021. · arXiv
  28. Masaru Isonuma, Junichiro Mori, Danushka Bollegala, and Ichiro Sakata. Unsupervised ab- stractive opinion summarization by generating sentences with tree-structured topic guidance. Transactions of the Association for Computational Linguistics, 9:945-961, 2021.
  29. Wojciech Kryscinski, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. CoRR, abs/1910.12840, 2019. · arXiv
  30. Asheesh Kumar, Tirthankar Ghosal, and Asif Ekbal. A deep neural architecture for decision- aware meta-review generation. In J. Stephen Downie, Dana McKay, Hussein Suleman, David M. Nichols, and Faryaneh Poursardar, editors, ACM/IEEE Joint Confer- ence on Digital Libraries, JCDL 2021, Champaign, IL, USA, September 27-30, 2021, pages 222-225. IEEE, 2021.
  31. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. Summac: Re- visiting nli-based models for inconsistency detection in summarization. CoRR, abs/2111.09525, 2021. · arXiv
  32. Li et al., 2017] Piji Li, Zihao Wang, Zhaochun Ren, Lidong Bing, and Wai Lam. Neural rating regression with abstrac- tive tips generation for recommendation. In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, editors, Proceedings of the 40th Inter- national ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 345-354. ACM, 2017.
  33. Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  34. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. Mred: A meta- review dataset for structure-controllable text generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio, editors, Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2521-2535. Association for Computational Linguis- tics, 2022.
  35. Ming Shen, Jie Ma, Shuai Wang, Yog- arshi Vyas, Kalpit Dixit, Miguel Ballesteros, and Yas- sine Benajiba. Simple yet effective synthetic dataset con- struction for unsupervised opinion summarization. CoRR, abs/2303.11660, 2023. · arXiv
  36. Yoshihiko Suhara, Xiaolan Wang, Ste- fanos Angelidis, and Wang-Chiew Tan. Opiniondigest: A simple framework for opinion summarization. In Dan Ju- rafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pages 5789-5798. Association for Computational Linguistics, 2020.
  37. Lu Wang and Wang Ling. Neural network-based abstract generation for opinions and argu- ments. In Kevin Knight, Ani Nenkova, and Owen Ram- bow, editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 47- 57. The Association for Computational Linguistics, 2016.
  38. Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. Re- viewrobot: Explainable paper review generation based on knowledge synthesis. In Brian Davis, Yvette Graham, John D. Kelleher, and Yaji Sripada, editors, Proceedings of the 13th International Conference on Natural Language Generation, INLG 2020, Dublin, Ireland, December 15- 18, 2020, pages 384-397. Association for Computational Linguistics, 2020.
  39. Wen Xiao and Giuseppe Carenini. Systematically exploring redundancy re- duction in summarizing long documents. In Kam-Fai Wong, Kevin Knight, and Hua Wu, editors, Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Pro- cessing, AACL/IJCNLP 2020, Suzhou, China, December 4-7, 2020, pages 516-528. Association for Computational Linguistics, 2020.
  40. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
  41. Wei Zhao, Michael Strube, and Steffen Eger. Discoscore: Evaluating text generation with BERT and discourse coherence. CoRR, abs/2201.11176, 2022. · arXiv

Authors
Zeng, Q., Sidhu, M., Blume, A., Chan, H. P., Wang, L., & Ji, H.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Alibaba DAMO Academy (China) (CN)
  • University of Michigan (US)
Corresponding author
Sidhu, Mankeerat
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer Nature Singapore
Published
2025
Pages
20-38
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-981-97-9536-9_2
OpenAlex
W4407411788
Cited by
2
References
38
Citations by year
2026: 2
FWCI
1.90
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2019897, HR001121C0165, 2034562, 2229873)
  • U.S. Department of Education (2229873)
  • Defense Advanced Research Projects Agency (FA8750, FA8750-18-2-0014, HR001121C0165)
  • Institute of Education Sciences (2229873)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
task decomposition, large language models, text summarization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review