CoChat
Sign Up
Cited by 1Topic ModelingInformation Retrieval and Search BehaviorMultimodal Machine Learning Applications

DebateQA: Evaluating Question Answering on Debatable Knowledge

Open access
Xu, Rongwu, Qi, Xuan, Qi, Zehan, Xu, Wei, Guo, Zhijiang·2026·Findings of the Association for Computational Linguistics: EACL 2026
DOI 10.18653/v1/2026.findings-eacl.44
Open this paper in CoChat — free accountOpen PDF

Abstract

The rise of large language models (LLMs) has enabled us to seek answers to inherently debatable questions on LLM chatbots, necessitating a reliable way to evaluate their ability.However, traditional QA benchmarks assume fixed answers are inadequate for this purpose.To address this, we introduce DEBATEQA, a dataset of 2,941 debatable questions, each accompanied by multiple human-annotated partial answers that capture a variety of perspectives.We develop two metrics: Perspective Diversity, which evaluates the comprehensiveness of perspectives, and Dispute Awareness, which assesses if the LLM acknowledges the question's debatable nature.Experiments demonstrate that both metrics are aligned with human preferences and stable across different underlying models.Using DEBATEQA with two metrics, we assess 12 prevalent LLMs and retrievalaugmented generation methods.Our findings reveal that while LLMs generally excel at recognizing debatable issues, their ability to provide comprehensive answers encompassing diverse perspectives varies considerably.

AI snapshot

Written from the paper’s full text.

TL;DR

DebateQA introduces a 2,941-question dataset with human-annotated partial answers and two metrics, Perspective Diversity and Dispute Awareness, for evaluating LLM answers to inherently debatable questions. Experiments with 12 LLMs show models generally recognize debate but vary widely in covering diverse perspectives, with GPT-4o strongest and smaller models weakest.

Key findings

  • The final dataset contains 2,941 debatable questions and 10,873 partial answers after removing 767 substandard partial answers; the test split has 1,000 randomly sampled questions.
  • P.D. aligns with human preferences: Spearman ρ = 0.733 with Phi-3 mini 128k, 0.825 with GPT-2 117M, and 0.820 with Qwen2 0.5B. D.A. reaches 0.800 accuracy and 0.868 F1 with Phi-3 medium, versus DELPHI’s Acknowledge metric at 0.220 accuracy and 0.049 F1.
  • In main evaluation, GPT-4o is best on P.D. (3.07/4.03) and D.A. (0.952/0.979), while Qwen2 0.5B is worst on P.D. (6.56/10.87) and D.A. (0.792/0.836); larger models generally outperform smaller ones, and open-source Llama3 70B and Gemma2 9B nearly match GPT-4o.
  • ReAct consistently outperforms Vanilla RAG, RAG on trustworthy documents improves P.D., more specific prompts improve both metrics, and higher temperature/top-p and longer responses generally improve performance.

Methods

  • Dataset curation: sourced 4,473 questions from DELPHI (2,281), CONFLICTINGQA (434), and the web (1,758), deduplicated to 3,216; retrieved documents from trustworthy TLDs and kept questions with at least three documents (2,982); used GPT-4 to extract POVs and expand them into explanations; three annotators performed two binary tasks with Fleiss’ Kappa κ = 0.66 and κ = 0.60, removing 767 partial answers.
  • Evaluation metrics: P.D. is the aggregate conditional perplexity of partial answers given a model answer, with lower scores better, using Qwen2 0.5B and GPT-2 117M as evaluators; D.A. is a binary prompt-based judgment from an instruction-tuned LLM (Phi-3 medium 128k, Qwen2 1.5B) of whether the answer acknowledges debate.
  • Experiments: evaluated 12 LLMs on DebateQA-test, including GPT-4o, GPT-4o mini, GPT-3.5 Turbo, Claude 3.5 Sonnet, Llama3 8B/70B, Qwen2 0.5B/1.5B/7B, Phi-3 mini/small 128k, and Gemma2 9B; also tested Vanilla RAG, ReAct, prompt variants, decoding hyperparameters, and response length.

Limitations

  • Stated: P.D. can diverge from human judgment when model answers contain non-perspective content or when legitimate perspectives are omitted because partial answers are extracted only from top Google-search documents.
  • Stated: the zero-shot D.A. variant can fail when the evaluator model does not output 0 or 1 as instructed.
  • Evident: evaluation is limited to 12 LLMs and selected RAG, prompt, and decoding settings; D.A. is binary and prompt-dependent, and P.D. depends on the evaluator model and the coverage of the partial-answer set.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2026.findings-eacl.44

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 15 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Traditional QA benchmarks are typically designed to provide fixed answers to questions.

    “Traditional question-answering (QA) benchmarks are typically designed to provide fixed answers to questions, as in datasets like SQuAD (Rajpurkar et al., 2016) and Natural Questions (Kwiatkowski et al., 2019).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Rajpurkar et al., 2016 source
    • cites Kwiatkowski et al., 2019 source
  • Premisecitation

    Even in scenarios where multiple legitimate answers are possible, such as TruthfulQA, answers are often presented as multiple-choice questions, limiting the space of responses.

    “Even in scenarios where multiple legitimate answers are possible, such as TruthfulQA (Lin et al., 2022), the answers are often presented as multiple-choice questions, thus limiting the space of responses.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Lin et al., 2022 source

      Section 2.2 (p. 4) says each TruthfulQA question has a set of true reference answers, and Section 3.2 (p. 5) describes a multiple-choice task restricted to reference answers. But Section 3.2 calls that task an additional variation: TruthfulQA’s main task generates full-sentence answers, so the claim overstates how the benchmark limits responses. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingown data

    LLMs with RAG do not consistently improve in answering debatable questions.

    “LLMs with RAG do not consistently improve in answering debatable questions”
    Quote in the paperSubstantial claim
  • Findingown data

    There is a positive correlation between P.D. and D.A. for the tested LLMs.

    “The figure indicates a positive correlation between P.D. and D.A., suggesting that models with greater perspective diversity are also more aware of the debate in question, aligning with our expectations.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The worst-performing Qwen2 0.5B's answers are around 3 to 9× worse than GPT-4o in recovering information in partial answers.

    “For instance, the worstperforming Qwen2 0.5B's answers are around 3 to 9×foot_2 worse than GPT-4o in terms of recovering the information in partial answers.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 23
  • Classified as probably not a substantial claim. 31
  • The quote only partly states this claim. 3

  • May combine two claims.

    Findingown data

    DEBATEQA sources questions from DELPHI (2,281), CONFLICTINGQA (434), and 1,758 manually sourced from the Web.

    “First, we repurpose two existing datasets: we select 2,281 annotated controversial questions from DELPHI (Sun et al., 2023) and a full set of 434 questions from CONFLICTINGQA (Wan et al., 2024).To enrich the existing data, we further manually sourced 1,758 additional debatable questions from the Web (see Table 8 for detailed sources).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Sun et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wan et al., 2024 source

      Section 3 identifies CONFLICTINGQA as the cited paper’s dataset of contentious questions, supporting its identification as a source of questions. The passages do not give a total of 434 questions or establish that 434 is the full set (searched: “434” and Table 2 statistics). The DELPHI and Web counts are not attributed to Wan et al. in the citing sentence. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The LLM is required to leverage only information in relevant documents to generate explanations, minimizing hallucinations.

    “The LLM is required to leverage only the information contained within these relevant documents to generate the explanation, minimizing the risk of hallucinations (Zhang et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zhang et al., 2023 source

      Section 5.4.2, “Resorting to External Knowledge” (page unavailable), says that obtaining relevant knowledge and using it to guide generation can mitigate hallucinations. It does not say that an LLM must use only information in the retrieved documents when generating an explanation; that restriction is the citing paper’s added requirement. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent long-form QA evaluations such as ELI5 and ASQA allow for more elaborate answers.

    “Recent years have seen the emergence of long-form QA evaluations, such as ELI5 (Fan et al., 2019) and ASQA (Stelmakh et al., 2022a), which allow for more elaborate answers.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Fan et al., 2019 source
    • cites Stelmakh et al., 2022a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Questions with fewer than three documents are discarded, resulting in 2,982 questions, each supported by 3-5 most relevant documents.

    “We discard questions that have fewer than three documents, resulting in 2,982 questions, each supported by 3-5 of the most relevant documents.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    More specific prompts significantly improved performance for all five tested models.

    “We find that even the relatively simple p comprehensive prompt significantly improved the performance for all five models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Deduplication removes duplicates, resulting in 3,216 questions.

    “We then run a deduplication algorithm (see § A.1 for details) to remove any duplicate questions, resulting in 3,216 questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    DEBATEQA is split into a test split of 1,000 randomly sampled questions and a dev set of the remaining instances.

    “To reduce computational costs for upcoming evaluation, we split DE-BATEQA into two splits: the test split with 1,000 randomly sampled questions and the dev set containing the remaining instances.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    P.D. significantly outperforms DELPHI's metric and G-Eval powered by GPT-4o in aligning with human preferences.

    “We observe that P.D. significantly outperforms DELPHI's metric and G-Eval powered by GPT-4o.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human pairwise preferences provide a valid ground truth for evaluating answer quality.

  • May combine two claims.

    Findingown data

    Average cosine similarity between document chunks and questions is 0.56, with no significant outliers, indicating high relevance and minimal noise.

    “We calculate the cosine similarity between document chunks and questions.As depicted in Figure 11, the average cosine similarity for document trunks is 0.56 and there are no significant outliers, indicating high relevance and minimal noise in the documents, confirming their overall quality for serving as the basis for upcoming steps.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Larger models generally outperform small ones on both P.D. and D.A. metrics.

    “Larger models generally outperform small ones.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Each partial answer in DEBATEQA addresses the question from a distinct perspective and consists of a POV and an explanation.

    “Each partial answer addresses the question from a distinct perspective and consists of two parts: a short point-of-view statement (POV) and a longform explanation (Explan) that fully expands the POV.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Models with moderate capabilities tend to generate structurally disjointed responses, blending various perspectives into single clauses, which diminishes the clarity and readability of the discourse.

    “Poor structural organization.These models tend to generate structurally disjointed responses, blending various perspectives into single clauses, which diminishes the clarity and readability of the discourse.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Qwen2 7B is representative of models with moderate capabilities.

  • May combine two claims.

    Findingown data

    Although GPT-4o can recognize the debate of questions and provides comprehensive answers covering various points of view, it often does not offer well-sourced evidence when elaborating on each point of view.

    “Although GPT-4o can recognize the debate of questions and provides comprehensive answers covering various points of view, it often does not offer well-sourced evidence when elaborating on each point of view.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    After extracting POVs, questions with fewer than three perspectives were filtered out, resulting in 2,941 questions.

    “After extracting the POVs, we filter out questions with fewer than three perspectives, ensuring that the remaining questions are sufficiently debatable, resulting in 2,941 questions.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Questions with fewer than three perspectives are not sufficiently debatable.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The two examples in Table 19 illustrate multiple deficiencies in Qwen2 7B's responses.

    “Table 19: Two examples that illustrate multiple deficiencies in Qwen2 7B's responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The average number of partial answers in DEBATEQA is approximately 3.7.

    “given the fact that the average number of partial answers in DEBATEQA ≈ 3.7”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The deduplication process resulted in a final dataset of 3,216 unique entries.

    “This process results in a final dataset consisting of 3,216 unique entries.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    After annotation, 767 partial answers deemed substandard by two or more annotators were removed, resulting in a final dataset of 10,873 partial answers.

    “After the annotation, we removed 767 partial answers deemed substandard by two or more annotators, resulting in a final dataset of 10,873 partial answers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The LLMs were configured as described in §7.1 to solicit answers, resulting in 500 answers.

    “We configure the LLMs as described in § 7.1 to solicit answers, resulting in 500 answers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Questions with fewer than three documents were filtered out, leaving 2,982 questions.

    “We filter questions with fewer than three documents, as we consider these lack sufficient trustworthy evidence, leaving us with 2,982 questions.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Questions with fewer than three documents lack sufficient trustworthy evidence.

  • May combine two claims.

    Findingown data

    Models with moderate capabilities lack neutrality in their responses; while recognizing the contentious nature of questions, they tend to provide unbalanced coverage and exhibit bias in language use.

    “Perspective imbalance.There is a noticeable lack of neutrality in the responses.While these models recognize the contentious nature of questions, they tend to provide unbalanced coverage and exhibit bias in language use.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Qwen2 7B is representative of models with moderate capabilities.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The questions in DEBATEQA are collected from three sources.

    “The questions in DEBATEQA are collected from three sources, as detailed in Table 9.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Despite P.D.'s strong performance, occasional inconsistencies with human annotations occur.

    “Despite P.D.'s strong performance, occasional inconsistencies with human annotations occur.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    A proper answer to a debatable question should inform the receiver that the issue is debatable.

    “Firstly, the answer should inform the receiver that the issue at hand is debatable (Misco, 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Misco, 2011

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    DELPHI curates a dataset with controversial questions but emphasizes identifying controversy rather than evaluating models' responses.

    “The most related work to us is DEL-PHI (Sun et al., 2023), which curates a dataset with controversial questions, however, their emphasis is on identifying controversy rather than delving deeper into evaluating models' responses.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sun et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Documents are sourced from authoritative top-level domains (TLDs) to ensure reliability.

    “To ensure the reliability of our partial answers, we source documents from authoritative top-level domains (TLDs), as listed in Table 10.This treatment helps in maintaining the reliability of the sources.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Documents from authoritative top-level domains (.edu, .org, .gov) are reliable sources for debatable questions.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    A proper answer to a debatable question should be comprehensive and include diverse perspectives.

    “Secondly, a proper answer should be comprehensive and include diverse perspectives, which are crucial for maintaining an atmosphere of neutrality, especially in public-related inquiries (Habermas, 1991).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Habermas, 1991

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductcitation

    Interannotator agreement for the two annotation tasks is κ=0.66 and κ=0.60, indicating substantial agreement.

    “Interannotator agreement (IAA) is measured using Fleiss' Kappa (Fleiss et al., 1981), yielding scores of κ = 0.66 and κ = 0.60 for the two annotation tasks, all indicating substantial agreement.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fleiss et al., 1981

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Traditional QA benchmarks assume fixed answers and are inadequate for debatable questions.

    “However, traditional QA benchmarks assume fixed answers are inadequate for this purpose.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Debatable questions inherently lack fixed answers.

    “reflecting human beings' natural curiosity about topics that inherently lack fixed answers (Lowry and Johnson, 1981;Brady, 2009).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Lowry and Johnson, 1981 source
    • cites Brady, 2009 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Three professional annotators from a local data annotation company were recruited to verify the partial answers.

    “We recruit three professional annotators from a local data annotation company to verify the partial answers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Retrieval was only done on authoritative domains listed in Table 10 to assure trustworthiness of documents.

    “We only do retrieval on authoritative domains in Table 10 to assure the trustworthiness of the documents.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: .gov and .edu domains are not open for personal registration and can only be registered by government or educational institutions.
    • Rests on an assumption: Although .org, .pro, and .info domains can now be registered by individuals, their content generally remains professional and informative, with fewer advertisements or potentially misleading information.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    To ensure distinctiveness of extracted POVs, pairwise cosine similarity is calculated, and a 0.75 cosine similarity threshold is used to discern unique POVs.

    “To ensure the distinctiveness of extracted POVs, i.e., they are non-overlapping, we calculate the pairwise cosine similarity for each question's POVs using gte-Qwen2-1.5B-instruct.We set a 0.75 cosine similarity threshold to discern unique POVs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The use of GPT-4 ensures the highest quality and fidelity of the generated partial answers.

    “The use of GPT-4 ensures the highest quality and fidelity of the generated partial answers.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: At the time of dataset curation (April to May 2024), the strongest model available was the gpt-4-turbo variant of GPT-4.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Longer answers provide more comprehensive information, enhancing P.D. scores.

    “This is likely due to longer answers providing more comprehensive information, enhancing P.D. scores.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Our metric is susceptible to non-perspective content, whereas human evaluators are less likely to be influenced by it, resulting in humans rating such answers higher than our metric does.

    “Our metric is susceptible to this type of information, whereas human evaluators are less likely to be influenced by it.This results in humans rating this answer higher than our metric does.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Three annotators recorded their preferences among the model answers.

    “Subsequently, we engage three annotators to record their preferences among the model answers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Equation 7 shows that the change in probability ∆P1 can be approximated using the change in P.D. ∆P.D., and the effect is exponential.

    “Equation 7 shows that the change in probability ∆P 1 can be approximated using the change in P.D. ∆P.D., and the effect is exponential.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The approximation exp(x) ≈ 1 + x holds for the values of x involved.
    • The authors state this as an assumption: All ∆P_i, i = 1...n are the same value.

  • May combine two claims.

    Interpretationown data

    Source quality in RAG for debatable QA is significant; utilizing trustworthy documents improves LLM response quality.

    “This highlights the significance of source quality in RAG for debatable QA, emphasizing that utilizing trustworthy documents improves LLM response quality in responding to sensitive topics.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The payment for this job is above the local minimum wage.

    “The payment for this job is above the local minimum wage.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Responses were gathered from five LLMs to 100 randomly chosen test questions from DEBATEQA-test.

    “Initially, we gather responses from a diverse selection of five LLMs to 100 randomly chosen test questions from DEBATEQA-test.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    After merging questions from the three sources, a deduplication algorithm removed duplicates from the original set of 4,473 questions.

    “After merging questions from the three sources, we employ a simple deduplication algorithm to remove duplicates from the original set of 4,473 questions”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    Acknowledging the debate improves D.A. scores.

    “Furthermore, when tasked with longer answers, models are more prone to acknowledge the debate, which improves D.A. scores.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    If a model's answer includes an omitted point, including this point will not improve the model's assessment of our metric, but human annotators may think the answer is more comprehensive, leading to discrepancies.

    “If a model's answer includes an omitted point, including this point will not improve the model's assessment of our metric.However, human annotators do not have a predetermined pool of perspectives, so including this point will likely make the annotator think the answer is more comprehensive, leading to discrepancies in judgments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    In a real debate, debaters need to support their arguments with evidence.

    “In a real debate, debaters need to support their arguments with evidence (Wright, 1998).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wright, 1998

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

45 of 45

  • The paper relies on this without support.

    Documents from authoritative top-level domains (.edu, .org, .gov) are reliable sources for debatable questions.

    “To ensure the reliability of our partial answers, we source documents from authoritative top-level domains (TLDs), as listed in Table 10.This treatment helps in maintaining the reliability of the sources.”

  • The paper relies on this without support.

    Human pairwise preferences provide a valid ground truth for evaluating answer quality.

    “Using pairwise preferences and Spearman's ρ correlation (Zar, 2005), we find strong agreement among three annotators with ρ > 0.8, as shown in Figure 2.This consensus allows us to assess the alignment of evaluator judgments with human preferences effectively.”

  • The paper relies on this without support.

    .gov and .edu domains are not open for personal registration and can only be registered by government or educational institutions.

    “Among the selected TLDs, .govand .edudomains are not open for personal registration and can only be registered by government or educational institutions.”

  • The paper relies on this without support.

    Although .org, .pro, and .info domains can now be registered by individuals, their content generally remains professional and informative, with fewer advertisements or potentially misleading information.

    “Although .org,.pro,and .infodomains can now be registered by individuals, their content generally remains professional and informative, with fewer advertisements or potentially misleading information.”

  • The paper relies on this without support.

    At the time of dataset curation (April to May 2024), the strongest model available was the gpt-4-turbo variant of GPT-4.

    “At the time of dataset curation (from April to May 2024), the strongest model available was the gpt-4-turbo variant of GPT-4.”

  • The authors state this as an assumption.

    Questions with fewer than three documents lack sufficient trustworthy evidence.

    “We filter questions with fewer than three documents, as we consider these lack sufficient trustworthy evidence, leaving us with 2,982 questions.”

  • The authors state this as an assumption.

    Questions with fewer than three perspectives are not sufficiently debatable.

    “After extracting the POVs, we filter out questions with fewer than three perspectives, ensuring that the remaining questions are sufficiently debatable, resulting in 2,941 questions.”

  • The paper relies on this without support.

    Qwen2 7B is representative of models with moderate capabilities.

    “Taking Qwen2 7B (Qwen, 2024a) as a case study in Table 19, we pinpoint three main deficiencies typically found in the responses of models with moderate capabilities:”

  • The authors state this as an assumption.

    The approximation exp(x) ≈ 1 + x holds for the values of x involved.

    “Applying the approximation exp(x) ≈ 1 + x for small x, we get:”

  • The authors state this as an assumption.

    All ∆P_i, i = 1...n are the same value.

    “Applying the approximation that all ∆P i , i = 1 • • • n are the same value for simplicity, we get:”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Zero-shot D.A. case study names a different model from the results table

    Minor

    The case study attributes the D.A.-ZS result to Phi-3 mini 128k, whereas the D.A.-ZS row in Table 3 identifies its backbone as Phi-3 medium.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 32

  1. Roy Bar-Haim, Dalia Krieger, Orith Toledo- Ronen, Lilach Edelstein, Yonatan Bilu, Alon Halfon, Yoav Katz, Amir Menczel, Ranit Aharonov, and Noam Slonim. 2019. From sur- rogacy to adoption; from bitcoin to cryptocur- rency: Debate topic expansion. In Proceed- ings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Flo- rence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 977-990. Association for Computational Linguistics. · DOI
  2. Johannes Bjerva, Nikita Bhutani, Behzad Gol- shan, Wang-Chiew Tan, and Isabelle Augen- stein. 2020. Subjqa: A dataset for subjectivity and review comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 5480- 5494. Association for Computational Linguis- tics. · DOI
  3. Kai Chen, Zihao He, Rong-Ching Chang, Jonathan May, and Kristina Lerman. 2023. Anger breeds controversy: Analyzing contro- versy and emotions on reddit. In Social, Cul- tural, and Behavioral Modeling -16th Inter- national Conference, SBP-BRiMS 2023, Pitts- burgh, PA, USA, September 20-22, 2023, Pro- ceedings, volume 14161 of Lecture Notes in Computer Science, pages 44-53. Springer. · DOI
  4. David Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 15607-15631. Association for Computa- tional Linguistics. · DOI · Paper page
  5. Yoonjung Choi, Yuchul Jung, and Sung-Hyon Myaeng. 2010. Identifying controversial issues and their sub-topics in news articles. In Intel- ligence and Security Informatics, Pacific Asia Workshop, PAISI 2010, Hyderabad, India, June 21, 2010. Proceedings, volume 6122 of Lec- ture Notes in Computer Science, pages 140- 153. Springer. · DOI · Paper page
  6. Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Im- proved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2587-2601. Association for Computa- tional Linguistics. · DOI · Paper page
  7. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: long form question answering. In Proceedings of the 57th Conference of the As- sociation for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 3558-3567. As- sociation for Computational Linguistics. · DOI · Paper page
  8. Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, et al. 1981. The measurement of inter- rater agreement. Statistical methods for rates and proportions, 2(212-236):22-23.
  9. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6465- 6488. Association for Computational Linguis- tics. · DOI · Paper page
  10. Jack Hessel and Lillian Lee. 2019. Something's brewing! early prediction of controversy- causing posts from discussion features. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Tech- nologies, volume 1.
  11. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain ques- tion answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781. As- sociation for Computational Linguistics. · DOI · Paper page
  12. Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit Iyyer. 2022. Rankgen: Improving text generation with large ranking models. In Pro- ceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emi- rates, December 7-11, 2022, pages 199-232. Association for Computational Linguistics. · DOI · Paper page
  13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polo- sukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question an- swering research. Trans. Assoc. Comput. Lin- guistics, 7:452-466. · DOI · Paper page
  14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceed- ings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. As- sociation for Computational Linguistics. · DOI · Paper page
  15. Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen- tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural In- formation Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  16. Chin-Yew Lin. 2004. Rouge: A package for auto- matic evaluation of summaries. In Text summa- rization branches out, pages 74-81.
  17. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214-3252. Association for Com- putational Linguistics. · DOI · Paper page
  18. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G- eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singa- pore, December 6-10, 2023, pages 2511-2522. Association for Computational Linguistics. · DOI · Paper page
  19. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Ha- jishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 12076-12100. As- sociation for Computational Linguistics. · DOI · Paper page
  20. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: An- swering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Process- ing, EMNLP 2020, Online, November 16-20, 2020, pages 5783-5797. Association for Com- putational Linguistics. · DOI
  21. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. Disentqa: Disentangling para- metric and contextual knowledge with coun- terfactual question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pages 10056-10070. Association for Computational Linguistics. · DOI
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311- 318. ACL. · DOI · Paper page
  23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised mul- titask learners. OpenAI blog, 1(8):9.
  24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopy- rev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Em- pirical Methods in Natural Language Process- ing, EMNLP 2016, Austin, Texas, USA, Novem- ber 1-4, 2016, pages 2383-2392. The Associa- tion for Computational Linguistics. · DOI · Paper page
  25. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics. · DOI · Paper page
  26. Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Cohen-Karlik, Lena Dankin, Lilach Edelstein, Liat Ein-Dor, Roni Friedman-Melamed, Assaf Gavron, Ariel Gera, Martin Gleize, Shai Gretz, Dan Gutfreund, Alon Halfon, Daniel Hershcovich, Ron Hoory, Yufang Hou, Shay Hummel, Michal Jacovi, Charles Jochim, Yoav Kantor, Yoav Katz, David Konopnicki, Zvi Kons, Lili Kotlerman, Dalia Krieger, Dan Lahav, Tamar Lavee, Ran Levy, Naftali Liberman, Yosi Mass, Amir Menczel, Shachar Mirkin, Guy Moshkowich, Shila Ofek- Koifman, Matan Orbach, Ella Rabinovich, Ruty Rinott, Slava Shechtman, Dafna Sheinwald, Eyal Shnarch, Ilya Shnayderman, Aya Sof- fer, Artem Spector, Benjamin Sznajder, Assaf Toledo, Orith Toledo-Ronen, Elad Venezian, and Ranit Aharonov. 2021. An autonomous de- bating system. Nat., 591(7850):379-384. · DOI
  27. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022a. ASQA: factoid ques- tions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 8273-8288. Association for Computational Linguistics. · DOI
  28. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022b. ASQA: factoid ques- tions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 8273-8288. Association for Computational Linguistics. · DOI
  29. David Q. Sun, Artem Abzaliev, Hadas Kotek, Christopher Klein, Zidi Xiu, and Jason D. Williams. 2023. DELPHI: data for evaluating llms' performance in handling controversial is- sues. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing: EMNLP 2023 -Industry Track, Sin- gapore, December 6-10, 2023, pages 820-827. Association for Computational Linguistics. · DOI
  30. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eu- nsol Choi. 2023. A critical evaluation of eval- uations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 3225-3245. Association for Computational Linguistics. · DOI · Paper page
  31. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali, Rwanda, May 1- 5, 2023. OpenReview.net.
  32. Jerrold H. Zar. 2005. Spearman rank correlation. · DOI

Cited by

Sorted by citations

Authors
Xu, R., Qi, X., Qi, Z., Xu, W., & Guo, Z.
Affiliations
  • Tsinghua University (CN)
  • University of Cambridge (GB)
Venue
Findings of the Association for Computational Linguistics: EACL 2026
Publisher
Association for Computational Linguistics
Published
2026
Pages
854-885
DOI
10.18653/v1/2026.findings-eacl.44
OpenAlex
W7140110514
Cited by
1
References
0
Citations by year
2025: 1
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EACL (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
question answering evaluation, large language models, retrieval-augmented generation, evaluation metrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review