Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles
Open accessAbstract
Previous research in multi-document news summarization has typically concentrated on collating information that all sources agree upon. However, the summarization of diverse information dispersed across multiple articles about an event remains underexplored. In this paper, we propose a new task of summarizing diverse information encountered in multiple news articles encompassing the same event. To facilitate this task, we outlined a data collection schema for identifying diverse information and curated a dataset named DiverseSumm. The dataset includes 245 news stories, with each story comprising 10 news articles and paired with a human-validated reference. Next, to enable consistent automatic evaluation, we conducted a comprehensive analysis to pinpoint the position and verbosity biases when utilizing Large Language Model (LLM)-based metrics for evaluating the coverage and faithfulness of summaries. Through correlation analyses, we outline the best practices for effectively using automatic LLM-based metrics on the DiverseSumm dataset. Finally, we study how LLMs summarize multiple news articles by analyzing which type of diverse information LLMs are capable of identifying. Our analyses suggest that despite the extraordinary capabilities of LLMs in single-document summarization, the proposed task remains a complex challenge for them mainly due to their limited coverage, with GPT-4 only able to cover under 40% of the diverse information on average.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Long-context instructions spanning several tens of thousands of tokens are known to deteriorate model performance.
“The summary task is the only task we include that tests long-context capabilities, with instructions spanning several tens of thousands of tokens, which is known to deteriorate model performance [29, 32, 33].”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [29]
This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content.
“This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content [29, 50, 40].”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [29]
Previous studies on verbosity bias have focused on specific cases: Zheng et al. limited their problem setting to questions answered with lists, and Huang et al. conducted experiments on summarization tasks.
“While previous studies have explored the concept of verbosity bias, they have tended to focus on specific cases.Zheng et al. (2023) limits their problem setting to questions answered with lists in their experiment on verbosity bias, and Huang et al. (2023) conducted experiments on summarization tasks.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Huang et al. (2023)
Huang et al. found that GPT-4 prefers short responses in faithfulness and coverage for summarization, although this is seen strongly only in single-answer grading, and not in comparison grading.
“They found that GPT-4 actually prefers short responses in faithfulness and coverage when it comes to summarization, although this is seen strongly only in single-answer grading, and not in comparison grading.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Huang et al. (2023)
Some tasks may have a negative verbosity bias, for example summarization tasks as shown in Huang et al. (2023).
“This distinction is crucial as some tasks may have a negative bias, for example in summarization tasks as shown in Huang et al. (2023).”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Huang et al. (2023)
Cites
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · arXiv · doi:10.48550/arxiv.2301.13298 · Cited by 3 · cited 2×
“Following the recommendations from Krishna et al. (2023), we conducted evaluations at a finer granularity.”
MethodResult
Flagged first, then by citations
Cited by
Sorted by citations
Verbosity Bias in Preference Labeling by Large Language Models
Saito, K., Wachi, A., Wataoka, K., & Akimoto, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.10076 · Cited by 15 · see citing works
LLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 10 · see citing works
“This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content [29, 50, 40].”
Self-citationAttributable by Construction: Claim-Anchored Provenance for Multi-Document Summarization
Guan, S. (2026) · arXiv · doi:10.48550/arxiv.2606.23989
“DiverseSumm (Huang et al., 2024) targets diverse and conflicting information across sources and supplies human-validated references for our conflict study.”
- Authors
- Huang, K.-H., Laban, P., Fabbri, A. R., Choubey, P. K., Joty, S., Xiong, C., & Wu, C.-S.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Salesforce (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2309.09369
- OpenAlex
- W4386875587
- Cited by
- 1
- References
- 1
- Citations by year
- 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
