CoChat
Sign Up
PreprintCited by 27Topic ModelingInformation Retrieval and Search BehaviorNatural Language Processing Techniques

YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering

Open access
D'Souza, Jennifer, Giglou, Hamed Babaei, Münch, Quentin·2025·arXiv
DOI 10.48550/arxiv.2505.14279arXiv 2505.14279
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) drive scientific question-answering on modern search engines, yet their evaluation robustness remains underexplored. We introduce YESciEval, an open-source framework that combines fine-grained rubric-based assessment with reinforcement learning to mitigate optimism bias in LLM evaluators. We release multidisciplinary scienceQ&A datasets, including adversarial variants, with evaluation scores from multiple LLMs. Independent of proprietary models and human feedback, our approach enables scalable, cost-free evaluation. By advancing reliable LLM-as-a-judge models, this work supports AI alignment and fosters robust, transparent evaluation essential for scientific inquiry.

AI snapshot

Written from the paper’s full text.

TL;DR

YESciEval is an open-source framework for robust LLM-as-a-judge evaluation of scientific question answering, combining nine rubric-based criteria with supervised fine-tuning and reinforcement learning to reduce optimism bias. It releases benign and adversarial scienceQ&A datasets and shows that a small open-source model can be aligned to distinguish benign from perturbed responses without proprietary models or human feedback.

Key findings

  • Vanilla Llama 8B showed excessive optimism, scoring above 4 on the Correctness rubric even for extreme adversarial cases; fine-tuning on benign data alone amplified optimism, RL on adversarial data alone made it overly pessimistic, but RL on both benign and adversarial samples stabilized it, scoring around 1 in extreme cases and around 3 in subtle cases.
  • Across benign evaluations, no LLM evaluator showed bias toward its own generated synthesis; instead, all evaluators consistently preferred synthesis outputs generated by Qwen.
  • ORKGSyn generally yielded higher alignment scores than BioASQ; strong correlations emerged within Llama 8B/70B and between Mistral and Qwen.
  • The study produced 37,584 evaluation scores for ORKGSyn and 7,884 for BioASQ, equating to 9,396 and 1,971 evaluations per model, with zero monetary cost versus at least 1,000 euros or dollars for proprietary GPT-based evaluations.

Methods

  • The framework defines two tasks: LLMgen generates a synthesized answer from a research question and top N relevant paper abstracts, and LLMeval scores responses on nine rubrics grouped into Linguistic & Stylistic Quality, Logical & Structural Integrity, and Content Accuracy & Informativeness, each on a 1–5 Likert scale.
  • Data came from ORKGSyn (multidisciplinary, N=5 papers per question) and BioASQ (biomedical, 73 summary questions with up to 40 PubMed abstracts), with adversarial variants created through subtle and extreme rubric-specific perturbations.
  • Four open-source LLMs (Llama 3.1 8B & 70B, Qwen 2.5 72B, Mistral Large 128B) served as generators and evaluators; alignment used supervised fine-tuning with QLoRA on LLaMA-3.1-8B followed by reinforcement learning via Contrastive Preference Optimization on benign and adversarial preference data.

Limitations

  • The paper states that chain-of-thought reasoning was not explored and is left for future work; RL was chosen over few-shot in-context learning because nine rubrics would require long prompts and curating high-quality bad examples would need human annotations, contradicting the zero-cost goal.
  • The paper states that the approach remains constrained by current open-source LLM capabilities, which may not fully match commercial alternatives, and that generalizing beyond scienceQ&A is future work.
  • Evident from the text: adversarial redundancy generation required extensive manual cleanup—over 100 Llama 8B responses and 18 Llama 70B responses were manually revised—which may affect reproducibility and scalability of the adversarial dataset construction.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.14279

Cites

76 of 76 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

18 of 27 cited sources checked · 9 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Existing benchmarks primarily assess multiple-choice reasoning or human preference alignment.

    “Existing benchmarks, such as Hugging Face leaderboard tasks (Wang et al., 2024;Rein et al., 2023) and alignment-focused chat-based evaluations (Reddy et al., 2019;Zheng et al., 2023;Köpf et al., 2024), primarily assess multiple-choice rea-soning or human preference alignment.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Wang et al., 2024 source
    • cites Rein et al., 2023 source
    • cites Reddy et al., 2019 source

      The Introduction (p. 2) describes CoQA as a conversational question-answering benchmark with free-form answers, not a multiple-choice or human-preference-alignment evaluation. The Evaluation Metric section (p. 9) says it scores answers by word-overlap F1. Thus, while CoQA is conversational, the paper characterizes its assessment differently from the citing sentence. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    Traditional n-gram-based metrics may not fully capture compositional and domain-specific reasoning in scienceQ&A.

    “The domain-agnostic and free-form nature of scienceQ&A limits the applicability of traditional n-gram-based metrics (e.g., BLEU (2002a), ROUGE (2004a)), which may not fully capture compositional and domain-specific reasoning (Krishna et al., 2021).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Krishna et al., 2021 source

      The “Takeaway (better automatic metrics needed)” section says ROUGE-L cannot reliably distinguish valid from invalid long-form answers, partly because answers are open-ended and the output space is large. The paper studies ELI5 long-form QA, not scienceQ&A, and does not establish that n-gram metrics fail specifically to capture compositional or domain-specific reasoning. No page number is available in our copy. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Premisecitation

    Of 41 NLP Q&A datasets reviewed by Wang (2022), only BioASQ aligns with scienceQ&A.

    “Of 41 NLP Q&A datasets reviewed by Wang (2022), only BioASQ aligns with scienceQ&A.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wang, 2022 source

      Section 2.3 (page unavailable) reviews BioASQ as a biomedical question-answering dataset, supporting its inclusion in Wang’s survey. The same section also discusses PubMedQA and QASPER; it does not assess which datasets align with scienceQ&A. A search for “41” did not substantiate the claimed dataset count. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    LLM-as-a-judge evaluations exhibit biases.

    “but also exhibit biases (Gudibande et al., 2023;Ye et al., 2024)”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Gudibande et al., 2023 source

      Section 4.4 reports that GPT-4 evaluations showed the same trends as crowdworker evaluations and says this implies LLMs *may* replicate human-like cognitive biases. That supports a concern about bias in LLM-as-a-judge evaluations, but the source presents it as a possibility for future investigation, not an established finding that such evaluations exhibit biases. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Ye et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Optimism bias is inherent in LLMs.

    “Second, we address the optimism bias (Villaflor et al., 2022) inherent in LLMs”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Villaflor et al., 2022 source

      Section 2.1 (p. 2) describes optimism bias in Decision Transformer and Trajectory Transformer, two sequence-modeling methods for offline reinforcement learning. It does not establish that optimism bias is inherent in large language models; the claim extends the finding to a different class of models. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 41
  • May combine two claims. 28
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation

    LLM-as-a-judge evaluations show parity with human judgment.

    “Recent efforts using LLMs as evaluators (LLM-as-a-judge (Zheng et al., 2023)) show parity with human judgment (Chiang and Lee, 2023b)”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chiang and Lee, 2023b source

      Section 3.3.1 reports statistically significant but weak-to-strong positive correlations between LLM and English-teacher ratings of stories. Section 3.3 also says their absolute scores differ. The findings support some alignment in judgment, but not the broader claim of parity with human judgment. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Interpretationcitation

    MT-Bench and Chatbot Arena assess open-domain, multi-turn dialogue abilities, but their focus remains distinct from our objective.

    “These benchmarks evaluate capabilities in writing, reasoning, extraction, and domain knowledge (e.g., STEM and humanities/social sciences), but their focus remains distinct from our objective.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zheng et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    DBLP-QuAD contains 10,000 QA pairs generated via SPARQL queries over the DBLP scholarly knowledge graph.

    “One notable dataset is DBLP-QuAD (Banerjee et al., 2023), which contains 10,000 QA pairs generated via SPARQL queries over the DBLP scholarly knowledge graph (KG).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Banerjee et al., 2023 source

  • May combine two claims.

    Premisecitation

    RL does not require large datasets; models learn from comparisons rather than absolute labels, reducing redundancy and cost.

    “Unlike traditional finetuning, RL does not require large datasets; instead, models learn from comparisons rather than absolute labels, reducing redundancy and cost (Ziegler et al., 2019).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ziegler et al., 2019 source

      Section 3.1.2 (p. 7) reports that very little human data was needed for the paper’s stylistic-continuation tasks, and Section 2.4 (p. 4) describes collecting comparison labels. The paper does not establish that RL generally needs less data than traditional fine-tuning or that using comparisons reduces redundancy and cost. Appendix B (p. 17) discusses difficulties with absolute ratings, but does not make that cost claim. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Most prior evaluation work relies on proprietary GPT models, raising issues of transparency and high costs at scale.

    “Most prior evaluation work relies on proprietary GPT models (Wang et al., 2023a;Dubois et al., 2023;Liu et al., 2023;Fu et al., 2024), raising issues of transparency and high costs at scale (Kim et al., 2023(Kim et al., , 2024)).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Kim et al., 2023 source
    • cites Kim et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SPECTER2 and SciBERT embeddings are designed for scientific texts and have shown superior performance in domain-specific applications.

    “However, we employed SPECTER2 (Singh et al., 2022) and SciB-ERT (Beltagy et al., 2019) embeddings, designed for scientific texts and have shown superior performance in domain-specific applications.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Singh et al., 2022 source
    • cites Beltagy et al., 2019 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MoverScore's combination of BERT's contextual embeddings and IDF weighting correlates highly with human judgment.

    “The combination of BERT's contextual embeddings and IDF weighting has been shown by Zhao et al. (2019) to correlate highly with human judgment.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zhao et al. (2019) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Among the 41 reviewed datasets, only the 2023 release of BioASQ was found to be directly relevant to our research objective.

    “Among the 41 reviewed datasets, only the 2023 release of BioASQ was found to be directly relevant to our research objective.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    A ROUGE-L score of 0.4 indicates a moderate similarity in sentence structure and coherence.

    “Additionally, a ROUGE-L score of 0.4 indicates a moderate similarity in sentence structure and coherence.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingnone given

    In the ORKGSynthesis dataset, NIST correlations are notably higher across all models.

    “In the ORKGSynthesis dataset, correlations are notably higher across all models.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    This process culminated in the creation of adversarial datasets with subtle and extreme degrees of manipulation for each underlying dataset, resulting in a total of four datasets.

    “This process culminated in the creation of adversarial datasets with subtle and extreme degrees of manipulation for each underlying dataset, resulting in a total of four datasets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The stabilized Llama 8B assigned relatively high scores for benign syntheses while distinguishing adversarial perturbations, scoring around 1 in extreme cases and around 3 in subtle cases.

    “It assigned relatively high scores for benign syntheses while distinguishing adversarial perturbations, scoring around 1 in extreme cases and around 3 in subtle cases, demonstrating rubric-specific discrimination.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Mistral-Large achieves the highest average BLUE score on the BioASQ dataset, while on the ORKGSynthesis dataset, Meta's 70B LLM performs best.

    “In terms of individual performance, Mistral-Large achieves the highest average BLUE score on the BioASQ dataset, while on the ORKGSynthesis dataset, Meta's 70B LLM performs best.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The four models evaluated all instances from ORKGSyn and BioASQ across the nine rubrics, resulting in 37,584 evaluation scores for ORKGSyn and 7,884 for BioASQ.

    “This resulted in 37,584 evaluation scores for ORKGSyn (348 × 3 × 9 × 4) and 7,884 for BioASQ (73 × 3 × 9 × 4)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The results reveal patterns of correlation among specific models.

    “Interestingly, the results reveal patterns of correlation among specific models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    This pattern is consistent for both subtle and extreme adversarial datasets, though the ORKGSyn dataset receives even higher scores compared to BioASQ.

    “This pattern is consistent for both subtle and extreme adversarial datasets, though the ORKGSyn dataset receives even higher scores compared to BioASQ.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The ORKGSyn corpus is a highly multidisciplinary scienceQ&A dataset.

    “The ORKGSyn corpus is a highly multidisciplinary scienceQ&A dataset.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Subtle Adversarial Dataset: reference texts are minimally altered, making it challenging for models to detect changes.

    “Subtle Adversarial Dataset: Here, reference texts are minimally altered, making it challenging for models to detect changes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The rubrics of GPTScore for summarization tasks align with ours.

    “The rubrics of GPTScore (Fu et al., 2024) for summarization tasks also align with ours.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    ORKGSyn consistently linked each Q to N = 5 papers with abstracts, whereas in BioASQ, N varied between 1 and 40.

    “According to the scienceQ&A task definition, ORKGSyn consistently linked each Q to N = 5 papers with abstracts, whereas in BioASQ, N varied between 1 and 40.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Illustrated results for ROUGE-1 and ROUGE-L exhibit distinct patterns across the two datasets.

    “Illustrated results for ROUGE-1 in Figure 7 and ROUGE-L in Figure 8 exhibit distinct patterns across the two datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    We manually revised over 100 responses generated by the Llama 8B model and 18 by the Llama 70B.

    “We manually revised over 100 responses generated by the Llama 8B model and 18 by the Llama 70B.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On the ORKGSynthesis dataset, MoverScore reveals strong correlations between the outputs of Mistral-Large and Qwen2.5-72B, and between the LLaMA models.

    “In contrast, the ORKGSynthesis dataset reveals strong correlations between the outputs of the Mistral-Large and Qwen2.5-72Bmodels, and between the LLaMA models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On the ORKGSynthesis dataset, the two LLaMA models show strong WMD alignment with one another, as do Mistral-Large and Qwen2.5-72B.

    “In contrast on the ORKGSynthesis dataset, the two LLaMA models show strong alignment with one another, as do Mistral-Large and Qwen2.5-72B.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    On the BioASQ dataset, the LLaMA-3.1-70B model performs slightly worse in MoverScore.

    “As observed with other metrics, the LLaMA-3.1-70Bmodel performs slightly worse on the BioASQ dataset, likely attributable to the model's struggles with generating comprehensive and coherent responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The examples in Table 15 and Table 16 illustrate challenges in vanilla and SFT models in distinguishing the desired behaviors.

    “The examples are shown in Table 15 and Table 16, which illustrate the challenges in vanilla and SFT models in distinguishing the desired behaviors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    BioASQ's biomedical focus led to greater uncertainty and lower correlation scores.

    “In contrast, BioASQ's biomedical focus, a more specialized domain, led to greater uncertainty and lower correlation scores.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The YESciEval approach is independent of proprietary models and human feedback, enabling scalable, cost-free evaluation.

    “Independent of proprietary models and human feedback, our approach enables scalable, cost-free evaluation.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Using proprietary models like OpenAI's GPT would have cost at least 1,000 euros or dollars for these evaluations.

  • May combine two claims.

    Premisecitation

    Human evaluation is costly and difficult to scale.

    “Human evaluation, though more nuanced, is costly and difficult to scale (Krishna et al., 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Krishna et al., 2023 source

  • Classified as probably not a substantial claim.

    Premisecitation

    CPO is a state-of-the-art method that enhances Direct Preference Optimization (DPO) by teaching the model to identify and reject suboptimal examples.

    “CPO is a state-of-the-art method that enhances Direct Preference Optimization (DPO) (Rafailov et al., 2023) by teaching the model to identify and reject suboptimal examples.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Xu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    If the model fails to adjust its scores accordingly, it suggests an inability to critique responses effectively, thus low reliability on the LLM-as-a-judge.

    “If the model fails to adjust its scores accordingly, it suggests an inability to critique responses effectively, thus low reliability on the LLM-as-a-judge.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The adversarial perturbations systematically degrade response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This discrepancy may be attributable to METEOR's emphasis on recall.

    “This discrepancy may be attributable to METEOR's emphasis on recall.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BioKGQA features 85,368 QA pairs generated using multi-noded triples from PrimeKG.

    “We also considered the BioKGQA dataset, proposed by Yan et al. (2024), which features 85,368 QA pairs generated using multi-noded triples from PrimeKG, a comprehensive KG oriented toward precision medicine.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yan et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Higher scores when Qwen2.5-72B is a candidate suggest it produces longer outputs, increasing the likelihood of matches with reference terms.

    “Higher scores when Qwen2.5-72B is a candidate suggest it produces longer outputs, increasing the likelihood of matches with reference terms.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Curating high-quality bad examples for ICL would necessitate human annotations, contradicting our goal of maintaining a fully automated, zero-cost alignment process.

    “Additionally, curating high-quality bad examples would necessitate human annotations, contradicting our goal of maintaining a fully automated, zero-cost alignment process.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Predefined guidelines ensure consistent and rigorous assessment.

    “Each rubric is scored on a Likert scale (1-5) with predefined guidelines, ensuring consistent and rigorous assessment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The nine rubrics minimize overlap while capturing key facets of an ideal LLM response to science Q&A.

  • May combine two claims.

    Interpretationreasoning

    An ICL approach would require significantly longer prompt contexts, potentially exceeding the model's ability to effectively utilize the information.

    “Given our nine rubrics, an ICL approach would require significantly longer prompt contexts, potentially exceeding the model's ability to effectively utilize the information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    This work advances the field by proposing a structured evaluation methodology that moves beyond traditional ROUGE/BLEU metrics and proprietary model dependencies.

    “This work advances the field by proposing a structured evaluation methodology that moves beyond traditional ROUGE/BLEU metrics and proprietary model dependencies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Our adversarial attacks, ranging from subtle to extreme, deliberately violate English syntax, ensuring inherently low-quality text and obviating the need for human raters.

    “In contrast, our adversarial attacks, ranging from subtle to extreme, deliberately violate English syntax, ensuring inherently low-quality text and obviating the need for human raters.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    BioKGQA primarily focuses on fact-based answers, making it less suitable for exploring nuanced or multi-dimensional evaluation.

    “While its method of leveraging KG structures for QA generation is innovative, the dataset primarily focuses on fact-based answers, making it less suitable for exploring nuanced or multi-dimensional evaluation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    BERTScore scores near 0.9 reflect a high degree of semantic similarity between outputs.

    “The other models, including those evaluated on the ORKGSynthesis dataset, achieve scores near 0.9, which reflects a high degree of semantic similarity between their outputs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    GPTScore includes dialogue rubrics aligned with instruction-following chat-based evaluations such as likeability, flexibility, inquisitiveness, and engagement, which are out of scope for scienceQ&A.

    “However, GPTScore also includes dialogue rubrics aligned with instruction-following chat-based evaluations such as likeability, flexibility, inquisitiveness, and engagement, which are out of scope for scienceQ&A.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    This design directly tests whether the LLM can overcome its optimism bias and robustly assign the expected low scores.

    “This design directly tests whether the LLM can overcome its optimism bias and robustly assign the expected low scores.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Adversarial manipulations that introduce errors into text will cause LLM judges to assign lower evaluation scores.

  • May combine two claims.

    Interpretationreasoning

    The paper introduces a generative scienceQ&A dataset, filling a gap in current benchmarking efforts.

    “In contrast, we introduce a generative scienceQ&A dataset, filling a gap in current benchmarking efforts.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: A systematic evaluation framework for scienceQ&A remains absent.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Our work targets a holistic evaluation of scienceQ&A, defining a multifaceted qualitative framework of nine rubrics that comprehensively assess LLM outputs.

    “Inspired by this, our work targets a holistic evaluation of scienceQ&A, defining a multifaceted qualitative framework of nine rubrics that comprehensively assess LLM outputs.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Nine evaluation criteria are systematically targeted during adversarial dataset creation, with distinct manipulations tailored to degrade the corresponding aspect of synthesis quality.

    “Nine evaluation criteria are systematically targeted during adversarial dataset creation, with distinct manipulations tailored to degrade the corresponding aspect of synthesis quality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The SciQA benchmark includes 100 handcrafted complex QA pairs alongside 2,465 automatically generated ones.

    “Another dataset we evaluated is the SciQA benchmark (Auer et al., 2023), which includes 100 handcrafted complex QA pairs alongside 2,465 automatically generated ones.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Auer et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BERTScore highly correlates with human judgment.

    “Studies by Zhang et al. (2020) demonstrate that BERTScore highly correlates with human judgment, making it a valuable tool for evaluating machine-generated text.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al. (2020)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Our approach reduces reliance on proprietary models but remains constrained by the capabilities of current open-source LLMs, which may not fully match the performance of commercial alternatives.

    “Finally, while our approach reduces reliance on proprietary models, it remains constrained by the capabilities of current open-source LLMs, which may not fully match the performance of commercial alternatives.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    This is because unigram-based evaluations inherently capture a broader overlap by disregarding strict positional constraints or dependencies on higher-order matches.

    “This is because unigram-based evaluations inherently capture a broader overlap by disregarding strict positional constraints or dependencies on higher-order matches.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Qwen likely demonstrates superior text generation abilities.

    “As the largest model, Qwen likely demonstrates superior text generation abilities”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The YESciEval approach is a demonstrated reproducible, cost-free framework for evaluating NLG in scienceQ&A.

    “The YESciEval approach is a demonstrated reproducible, cost-free framework for evaluating NLG in scienceQ&A, advancing AI alignment, robustness, and the broader agenda of LLM plausibility.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Using proprietary models like OpenAI's GPT would have cost at least 1,000 euros or dollars for these evaluations.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Adversarial datasets are designed to exhibit a marked deterioration in quality, and we expect evaluation scores to reflect this decline.

    “While original syntheses may not always be flawless, adversarial datasets are designed to exhibit a marked deterioration in quality, and we expect evaluation scores to reflect this decline.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Adversarial manipulations that introduce errors into text will cause LLM judges to assign lower evaluation scores.

  • May combine two claims.

    Premisereasoning

    WER normalizes the number of edits by reference length, so shorter reference texts amplify the impact of discrepancies and lead to inflated WER values.

    “Since WER normalizes the number of edits by reference length, shorter reference texts amplify the impact of any discrepancies, leading to inflated WER values.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The examples were selected randomly from experimental models' LLM eval outputs in test sets.

    “The examples were selected randomly from experimental models LLM eval outputs in test sets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    LLaMA-3.1-70B's poor WMD performance on BioASQ is consistent with its tendency to generate incoherent responses.

    “As expected, LLaMA-3.1-70Bexhibits poor performance on the BioASQ dataset, consistent with its tendency to generate incoherent responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    LLaMA-3.1-70B's slightly worse MoverScore on BioASQ is likely attributable to its struggles with generating comprehensive and coherent responses.

    “As observed with other metrics, the LLaMA-3.1-70Bmodel performs slightly worse on the BioASQ dataset, likely attributable to the model's struggles with generating comprehensive and coherent responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The uniformity in MoverScore can be interpreted as evidence that the models generate outputs with similar semantic content and structure, despite potential stylistic differences.

    “This uniformity can be interpreted as evidence that the models generate outputs with similar semantic content and structure, despite potential stylistic differences.”
    Quote in the paperNot a substantial claim
58 of 58

  • The paper relies on this without support.

    A systematic evaluation framework for scienceQ&A remains absent.

    “Despite the growing use of LLMs for sci-enceQ&A, a systematic evaluation framework remains absent, limiting iterative development and transparency.”

  • The paper relies on this without support.

    LLMs' real-world robustness in scienceQ&A remains largely unexplored.

    “their realworld robustness in scienceQ&A remains largely unexplored.”

  • The paper relies on this without support.

    Human feedback is often infeasible to obtain.

    “While human feedback is invaluable for LLM alignment, it is often infeasible to obtain.”

  • The paper relies on this without support.

    Using proprietary models like OpenAI's GPT would have cost at least 1,000 euros or dollars for these evaluations.

    “whereas using proprietary models like Ope-nAI's GPT would have cost at least 1,000 euros or dollars for these evaluations.”

  • The paper relies on this without support.

    The nine rubrics minimize overlap while capturing key facets of an ideal LLM response to science Q&A.

    “Chosen for simplicity, memorability, and precise definability, these rubrics minimize overlap while capturing key facets of an ideal LLM response to science Q&A.”

  • The paper relies on this without support.

    The adversarial perturbations systematically degrade response quality.

    “These perturbations systematically degrade response quality, ensuring rigorous evaluation of LLM-as-a-judge reliability.”

  • The authors state this as an assumption.

    Adversarial manipulations that introduce errors into text will cause LLM judges to assign lower evaluation scores.

    “we expect evaluation scores to reflect this decline.”

  • The paper relies on this without support.

    Human annotations for evaluating scientific QA are expensive.

    “eliminate reliance on expensive human annotations”

  • The paper relies on this without support.

    Proprietary LLM-generated reference answers are costly.

    “proprietary LLM-generated reference answers”

  • The paper relies on this without support.

    The ORKGSyn and BioASQ datasets are suitable for evaluating generative scienceQ&A.

    “The ORKGSyn corpus is a highly multidisciplinary scienceQ&A dataset.”

  • The paper relies on this without support.

    WER cannot account for semantic equivalence or tolerate variations in word order, which limits its applicability in evaluating generative models.

    “While WER provides a straightforward measure of surface-level similarity, its inability to account for semantic equivalence or tolerate variations in word order limits its applicability in evaluating gen-erative models.”
11 of 11

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • SFT GPU count conflicts across setup sections

    Major

    The training details say two GPUs were used in the SFT batch-size description, whereas the hardware section says SFT used only one H100 GPU.

  • ORKGSyn training and test counts fall short of the stated instance count

    Minor

    Table 2 lists 234 ORKGSyn training instances and 105 test instances, totaling 339. The evaluation count instead uses 348 instances, leaving nine unaccounted for in the stated training and test split.

    Reported 348 · recomputed 339

  • Qwen is called the largest model despite Mistral being larger

    Minor

    The paper identifies Qwen as a 72B-parameter model and Mistral Large as a 128B-parameter model. Its explanation of Qwen's preferred outputs as a consequence of being the largest model is therefore inconsistent with its own model specifications.

  • Model-size range gives two different maxima

    Minor

    The introduction says the selected models span up to 123B parameters, but the model-selection section lists a selected 128B model.

4 of 4

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

16 of the 92 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556-6576.
  2. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957-966.
  3. PMLR.
  4. VI Levenshtein. 1966. Binary codes capable of correct- ing deletions, insertions, and reversals. Proceedings of the Soviet physics doklady.
  5. Chin-Yew Lin. 2004a. Rouge: A package for auto- matic evaluation of summaries. In Text summariza- tion branches out, pages 74-81.
  6. Chin-Yew Lin. 2004b. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

Cited by

Sorted by citations

Related

Authors
D'Souza, J., Giglou, H. B., & Münch, Q.
Affiliations
  • Technische Informationsbibliothek (TIB) (DE)
  • Leibniz University Hannover (DE)
ORCID
Giglou, Hamed Babaei
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.14279
arXiv
2505.14279
OpenAlex
W4416557032
Cited by
0
References
92
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
LLM-as-a-judge, scientific question answering, evaluation robustness, optimism bias, rubric-based assessment, adversarial evaluation, LLM evaluation, AI alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review