Cited by 14Speech and dialogue systemsTopic ModelingSpeech Recognition and Synthesis

Dialogue Summarization with Mixture of Experts based on Large Language Models

Open access
Tian, Yuanhe, Xia, Fei, Song, Yan·2024·Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Abstract

Dialogue summarization is an important task that requires to generate highlights for a conversation from different aspects (e.g., content of various speakers).While several studies successfully employ large language models (LLMs) and achieve satisfying results, they are limited by using one model at a time or treat it as a black box, which makes it hard to discriminatively learn essential content in a dialogue from different aspects, therefore may lead to anticipation bias and potential loss of information in the produced summaries.In this paper, we propose an LLM-based approach with roleoriented routing and fusion generation to utilize mixture of experts (MoE) for dialogue summarization.Specifically, the role-oriented routing is an LLM-based module that selects appropriate experts to process different information; fusion generation is another LLM-based module to locate salient information and produce finalized dialogue summaries.The proposed approach offers an alternative solution to employing multiple LLMs for dialogue summarization by leveraging their capabilities of in-context processing and generation in an effective manner.We run experiments on widely used benchmark datasets for this task, where the results demonstrate the superiority of our approach in producing informative and accurate dialogue summarization. 1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.acl-long.385

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. Guimin Chen, Yuanhe Tian, and Yan Song. 2020. Joint Aspect Extraction and Sentiment Analysis with Di- rectional Graph Convolutional Networks. In Pro- ceedings of the 28th International Conference on Computational Linguistics, pages 272-279.
  2. Jiaao Chen, Mohan Dodda, and Diyi Yang. 2023a. Human-in-the-loop Abstractive Dialogue Summa- rization. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 9176-9190, Toronto, Canada.
  3. Jiaao Chen and Diyi Yang. 2020b. Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118, Online.
  4. Jiaao Chen and Diyi Yang. 2021. Structure-aware ab- stractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 1380-1391, Online.
  5. Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. 2023b. Analog Bits: Generating Discrete Data us- ing Diffusion Models with Self-Conditioning. In The Eleventh International Conference on Learning Representations.
  6. Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. 2023c. Lifelong Language Pretraining with Distribution- Specialized Experts. In Proceedings of the 40th Inter- national Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 5383-5395.
  7. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A Real-Life Scenario Dialogue Summarization Dataset. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021, pages 5062-5074, Online.
  8. Tanya Chowdhury, Sachin Kumar, and Tanmoy Chakraborty. 2020. Neural abstractive summariza- tion with structural attention. In Proceedings of the Twenty-Ninth International Joint Conference on Arti- ficial Intelligence, IJCAI 2020, pages 3716-3722.
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota.
  10. Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2020. ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4729-4740.
  11. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier- Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569.
  12. Shen Gao, Xin Cheng, Mingzhe Li, Xiuying Chen, Jin- peng Li, Dongyan Zhao, and Rui Yan. 2023. Di- alogue Summarization with Static-Dynamic Struc- ture Fusion Graph. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13858- 13873, Toronto, Canada.
  13. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum Corpus: A Human- annotated Dialogue Dataset for Abstractive Summa- rization. EMNLP-IJCNLP 2019, page 70.
  14. Iryna Gurevych and Michael Strube. 2004. Semantic Similarity Applied to Spoken Dialogue Summariza- tion. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguis- tics, pages 764-770.
  15. Yilun Hua, Zhaoyuan Deng, and Kathleen McKeown. 2023. Improving Long Dialogue Summarization with Semantic Graph Representation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13851-13883, Toronto, Canada.
  16. Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87.
  17. Qi Jia, Yizhu Liu, Haifeng Tang, and Kenny Zhu. 2022. Post-Training Dialogue Summarization using Pseudo-Paraphrasing. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1660-1669, Seattle, United States.
  18. Anirudh Joshi, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2020. Dr. Summarize: Global Sum- marization of Medical Dialogue by Exploiting Local Structures. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 3755- 3763, Online.
  19. Ryuji Kano, Yasuhide Miura, Tomoki Taniguchi, and Tomoko Ohkuma. 2020. Identifying Implicit Quotes for Unsupervised Extractive Summarization of Con- versations. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 291-302.
  20. Kundan Krishna, Sopan Khosla, Jeffrey Bigham, and Zachary C. Lipton. 2021. Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 4958-4972, Online.
  21. Yuejie Lei, Fujia Zheng, Yuanmeng Yan, Keqing He, and Weiran Xu. 2021. A Finer-grain Universal Dia- logue Semantic Structures based Model For Abstrac- tive Dialogue Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1354-1364, Punta Cana, Dominican Re- public.
  22. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In Inter- national Conference on Learning Representations, pages 1-19.
  23. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online.
  24. Yu Li, Baolin Peng, Pengcheng He, Michel Galley, Zhou Yu, and Jianfeng Gao. 2023b. DIONYSUS: A Pre- trained Model for Low-Resource Dialogue Summa- rization. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1368-1386, Toronto, Canada.
  25. Xinnian Liang, Chao Bian, Shuangzhi Wu, and Zhoujun Li. 2022. Towards Modeling Role-Aware Centrality for Dialogue Summarization. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the As- sociation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 43-50, Online.
  26. Xinnian Liang, Shuangzhi Wu, Chenhao Cui, Jiaqi Bai, Chao Bian, and Zhoujun Li. 2023. Enhancing Dia- logue Summarization with Topic-Aware Global-and Local-Level Centrality. In Proceedings of the 17th Conference of the European Chapter of the Associ- ation for Computational Linguistics, pages 27-38, Dubrovnik, Croatia.
  27. Chin-Yew Lin. 2004. Rouge: A Package for Automatic Evaluation of Summaries. In Text summarization branches out, pages 74-81.
  28. Haitao Lin, Liqun Ma, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2021. CSDS: A Fine-Grained Chinese Dataset for Customer Ser- vice Dialogue Summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4436-4451.
  29. Haitao Lin, Junnan Zhu, Lu Xiang, Feifei Zhai, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2023. Topic-Oriented Dialogue Summarization. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 31:1797-1810.
  30. Haitao Lin, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2022. Other Roles Matter! Enhancing Role-Oriented Dialogue Summa- rization via Role Interactions. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics.
  31. Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021. Topic-Aware Contrastive Learning for Ab- stractive Dialogue Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1229-1243, Punta Cana, Dominican Re- public.
  32. Xingxian Liu and Yajing Xu. 2023. Learning to Rank Utterances for Query-Focused Meeting Summariza- tion. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 8496-8505, Toronto, Canada.
  33. Yang Liu and Mirella Lapata. 2019. Text Summariza- tion with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740.
  34. Siru Ouyang, Jiaao Chen, Jiawei Han, and Diyi Yang. 2023. Compositional Data Augmentation for Ab- stractive Conversation Summarization. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1471-1488, Toronto, Canada.
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadel- phia, Pennsylvania, USA.
  36. MengNan Qi, Hao Liu, YuZhuo Fu, and Ting Liu. 2021. Improving Abstractive Dialogue Summarization with Hierarchical Pretraining and Topic Segment. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 1121-1130, Punta Cana, Dominican Republic.
  37. Abigail See, Peter J Liu, and Christopher D Man- ning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1073-1083.
  38. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Net- works: The Sparsely-Gated Mixture-of-Experts Layer. pages 1-19.
  39. Yan Song. 2022. Composing Ci with Reinforced Non- autoregressive Text Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 7219-7229.
  40. Yan Song, Chia-Jung Lee, and Fei Xia. 2017. Learn- ing Word Representations with Regularization from Prior Knowledge. In Proceedings of the 21st Confer- ence on Computational Natural Language Learning (CoNLL 2017), pages 143-152.
  41. Yan Song, Shuming Shi, and Jing Li. 2018. Joint Learn- ing Embeddings for Chinese Words and Their Com- ponents via Ladder Structured Networks. In Proceed- ings of the 27th International Joint Conference on Artificial Intelligence, pages 4375-4381.
  42. Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020. Summarizing Medical Conversations via Identifying Important Utterances. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 717-729.
  43. Yuanhe Tian, Fei Xia, and Yan Song. 2024. Learning Multimodal Contrast with Cross-modal Memory and Reinforced Contrast Recognition. In The 62nd An- nual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.
  44. Nan Wang, Yan Song, and Fei Xia. 2018. Coding struc- tures and actions with the costa scheme in medical conversations. In Proceedings of the BioNLP 2018 workshop, pages 76-86.
  45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems, 35:24824-24837.
  46. Xintong Yu, Hongming Zhang, Yangqiu Song, Yan Song, and Changshui Zhang. 2019. What you see is what you get: Visual pronoun coreference resolution in dialogues. arXiv preprint arXiv:1909.00421. · arXiv
  47. Hongming Zhang, Yan Song, and Yangqiu Song. 2019a. Incorporating Context and External Knowledge for Pronoun Coreference Resolution. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 872-881.
  48. Hongming Zhang, Yan Song, Yangqiu Song, and Dong Yu. 2019b. Knowledge-aware Pronoun Coreference Resolution. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 867-876, Florence, Italy.
  49. Litian Zhang, Xiaoming Zhang, and Junshu Pan. 2022. Hierarchical Cross-modality Semantic Correlation Learning Model for Multimodal Summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11676-11684.
  50. Longxiang Zhang, Renato Negrinho, Arindam Ghosh, Vasudevan Jagannathan, Hamid Reza Hassanzadeh, Thomas Schaaf, and Matthew R. Gormley. 2021. Leveraging Pretrained Models for Automatic Sum- marization of Doctor-Patient Conversations. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 3693-3712, Punta Cana, Dominican Republic.
  51. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019c. BERTScore: Evalu- ating Text Generation with BERT. In International Conference on Learning Representations.
  52. Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. DialogLM: Pre-trained Model for Long Dialogue Understanding and Sum- marization. In Thirty-Sixth AAAI Conference on Arti- ficial Intelligence, AAAI 2022, pages 11765-11773.
  53. Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xue- dong Huang. 2020. A Hierarchical Network for Ab- stractive Meeting Summarization with Cross-Domain Pretraining. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 194- 203, Online.
  54. Rongxin Zhu, Jianzhong Qi, and Jey Han Lau. 2023. Annotating and Detecting Fine-grained Factual Er- rors for Dialogue Summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 6825-6845.
  55. Yicheng Zou, Kaitao Song, Xu Tan, Zhongkai Fu, Qi Zhang, Dongsheng Li, and Tao Gui. 2023. To- wards Understanding Omission in Dialogue Summa- rization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14268-14286.
  56. Yicheng Zou, Lujun Zhao, Yangyang Kang, Jun Lin, Minlong Peng, Zhuoren Jiang, Changlong Sun, Qi Zhang, Xuanjing Huang, and Xiaozhong Liu. 2021. Topic-Oriented Spoken Dialogue Summa- rization for Customer Service with Saliency-Aware Topic Modeling. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 14665-14673.

Cited by

Sorted by citations

Authors
Tian, Y., Xia, F., & Song, Y.
Affiliations
University of Science and Technology of China (CN)
Venue
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Published
2024
Pages
7143-7155
ISSN
0736-587X (ISSN-L)
OpenAlex
W4402672051
Cited by
14
References
13
Citations by year
2026: 3 · 2025: 9 · 2024: 2
FWCI
4.13
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
Funders
National Key Research and Development Program of China (2021YFF0901600)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
dialogue summarization, mixture of experts, large language models, multi-speaker conversations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review