CoChat
Sign Up
Cited by 4Topic ModelingNatural Language Processing TechniquesSpeech and dialogue systems

Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

Open access
Siro, Clemencia, Aliannejadi, Pourya, Aliannejadi, Mohammad·2026·Findings of the Association for Computational Linguistics: EACL 2026
DOI 10.18653/v1/2026.findings-eacl.335
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are increasingly used as evaluators for natural language generation, applying human-defined rubrics to assess system outputs.However, human rubrics are often static and misaligned with how models internally represent language quality.We introduce GER-Eval (Generating Evaluation Rubrics for Evaluation) to investigate whether LLMs can design and apply their own evaluation rubrics.We evaluate the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria.LLMs reliably generate interpretable and task-aware evaluation dimensions and apply them within models, but their scoring reliability degrades in factual and knowledge-intensive settings.Closed-source models such as GPT-4o achieve higher agreement and cross-model generalization than open-weight models such as Llama.Our findings position evaluation as a learned linguistic capability of LLMs, consistent within models but fragmented across them, and call for new methods that jointly model human and LLM evaluative language to improve reliability and interpretability.

AI snapshot

Written from the paper’s full text.

TL;DR

GER-Eval tests whether LLMs can generate and apply their own evaluation rubrics, comparing rubric coherence, scoring reliability, and alignment with human criteria across four NLG datasets. LLMs produce interpretable, task-aware rubrics and apply them consistently within models, but reliability and human alignment drop sharply in factual or knowledge-intensive settings.

Key findings

  • LLMs generated five to eight rubrics per dataset; GPT-4o and GPT-4o-mini produced around 90% unique rubrics, and alignment with human rubrics exceeded 80% across most datasets. HelpSteer2 reached full alignment (Align. = 100) in few-shot settings, while SumPubMed was lowest, often below 60%.
  • Models were internally consistent when applying their own rubrics: zero-shot vs. few-shot agreement typically fell between 70–90%, with correlations above 0.8 for most configurations. GPT-4o and Mixtral were most robust.
  • With human rubrics, correlations with human scores reached ρ ≈ 0.8–0.9 for HelpSteer2 and USR, but fell to around 0.5 for SummEval and below 0.3 for SumPubMed.
  • Rubric transfer was task-dependent: USR showed strong inter-model reliability (ICC ≈ 0.7–0.8, α ≈ 0.74–0.81, ρ ≈ 0.7–0.9), while SumPubMed transfer was weak (ICC < 0.2, ρ < 0.3), with factual and terminology rubrics showing near-zero or negative correlations.

Methods

  • GER-Eval is a two-stage diagnostic framework: LLMs first generate task-specific rubrics with names, descriptions, scales, and scoring instructions; then they apply those rubrics to candidate outputs, with or without demonstrations.
  • Experiments covered four datasets—USR, SummEval, SumPubMed, and HelpSteer2—spanning conversational, summarization, instruction-following, and biomedical tasks, using 50 data points per dataset.
  • Models included GPT-4o, GPT-4o-mini, Mixtral-8x22B, Qwen2.5-72B, and Llama-3.3-70B. Rubric generation used three prompting conditions (Task-only, Task + Contexts, Task + Contrastive Examples); scoring used Rubric-only and Rubric + Demonstrations, with human-defined vs. LLM-generated rubrics compared.

Limitations

  • Stated: the study is limited to a subset of closed-source and open-weight LLMs on English-language datasets, focuses on textual outputs and rubric-based evaluations, does not independently verify factual accuracy beyond human reference scores, and treats LLMs as static evaluators without modeling temporal drift or adaptation.
  • Evident from the text: the human-rubric alignment metric relies on GPT-4o as an LLM-based tagger, so the reported overlap between generated and human rubrics is itself model-mediated.
  • Evident from the text: the experiments use 50 data points per dataset, which may limit generalization of the reported reliability and alignment estimates.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2026.findings-eacl.335

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

17 of 20 cited sources checked · 4 supported · 2 flagged · accuracy not yet measured

  • Interpretationown data

    Evaluation rubrics are model-dependent.

    “suggesting that evaluation rubrics are modeldependent.”
    • Also stated in: § 5 Discussion · hedged
      “Rubric transfer across models shows the lowest consistency, even when similar rubrics are used, implying that evaluative semantics diverge across architectures.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Jones et al., 2025 source

      Section 4.2 (p. 8) shows that Mistral and Llama 3 interpret subjective phrases differently, supporting the idea that evaluations may need to account for the model being tested. But Sections 3.3 (p. 5) and 4.3 (p. 9) describe using a judge, including the same GPT-4 judge across evaluations; they do not establish that the evaluation rubrics themselves are model-dependent. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    LLMs fail to produce stable preferences across reruns.

    “Studies have also shown that LLMs favor their own generations (Koo et al., 2023), or fail to produce stable preferences across reruns (Stureborg et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Stureborg et al., 2024 source
  • Findingown data

    On HelpSteer2, several models reach full alignment (100%) in few-shot settings.

    “On HelpSteer2, where quality involves tone and empathy, several models reach full alignment (i.e., Align.(%) = 100 in Table 1) in few-shot settings, suggesting that interpersonal aspects are easier to internalize.”
    Quote in the paperNot a substantial claim
  • Findingown data

    LLM scoring reliability degrades in factual and knowledge-intensive settings.

    “LLMs reliably generate interpretable and task-aware evaluation dimensions and apply them within models, but their scoring reliability degrades in factual and knowledge-intensive settings.”
    • Also stated in: § 1 Introduction · plain
      “In contrast, all models perform less reliably in factual or knowledge-intensive settings.”
    • Also stated in: § 4.2 Rubric Application · plain
      “In contrast, agreement drops on SummEval and especially SumPubMed, where tasks require factual reasoning or domain knowledge.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human scores on the datasets are valid gold standards for evaluating LLM-based scores.
  • Findingown data

    Few-shot prompting generally increases the number of rubrics.

    “Fewshot prompting generally increases the number of rubrics, while GPT-4o and GPT-4o-mini generate more unique rubrics (around 90%), indicating broader yet non-redundant coverage.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 22
  • May combine two claims. 31

  • Classified as probably not a substantial claim. May combine two claims.

    Premiseown data

    The evaluated LLMs include GPT-4o, GPT-4o-mini, Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B, covering proprietary and open-weight model families.

    “Our selection includes GPT-4o and GPT-4o-mini (OpenAI), Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B,enabling analysis across proprietary and openweight model families.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites OpenAI, 2023 source

      The Introduction (p. 1) presents GPT-4, and the Capabilities section (p. 7) evaluates GPT-4 alongside GPT-3.5 and other models. Searches for GPT-4o-mini, the listed model names, and open-weight model families found no passage verifying the citing authors’ model selection or its proprietary/open-weight coverage. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Jiang et al., 2024 source

      The Introduction identifies the model presented in this paper as Mixtral 8x7B, not Mixtral-8×22B. The Results likewise evaluate Mixtral 8x7B alongside Llama 2 and GPT-3.5; they do not identify the cited paper as a source for the named Mixtral-8×22B model. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Yang et al., 2024 source

      The Qwen2-72B section names Mixtral-8x22B as an open-weight comparison model, and Table 11 (section 6) includes GPT-4o under proprietary LLMs. The report does not document the citing authors’ model selection and does not identify GPT-4o-mini, Llama-3.3-70B, or Qwen2.5-72B; its comparisons instead name Llama-3-70B and Qwen2-72B. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Llama 3.3-70B-Instruct serves as a widely adopted open-weight baseline for instruction-following and judgment tasks, supporting long-context evaluation.

    “Llama 3.3-70B-Instruct (Touvron et al., 2023) serves as a widely adopted open-weight baseline for instruction-following and judgment tasks, supporting long-context evaluation.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Touvron et al., 2023 source

      The paper describes its LLaMA models as ranging from 7B to 65B parameters (opening section, p. 1) and identifies its instruction-finetuned model as LLaMA-I (Section 4, p. 7; Appendix D, p. 22). It therefore does not identify the cited model as Llama 3.3-70B-Instruct; the passages also provide no basis for the claims about its widespread adoption or long-context evaluation. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    HelpSteer2 represents the instruction-following domain, where the task is to generate helpful responses to user prompts, with outputs being model-generated chat responses annotated by humans on five attributes of helpfulness.

    “HelpSteer2 (Wang et al., 2024) represents the instruction-following domain, where the task is to generate helpful responses to user prompts.The outputs are model-generated chat responses, annotated by humans on five attributes of helpfulness.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2024 source

  • May combine two claims.

    Premisecitation

    LLM-based evaluations are sensitive to prompt phrasing, label order, response length, and position.

    “LLM-based evaluations are sensitive to prompt phrasing (Zhou et al., 2023), label order (Wang et al., 2023), response length (Stureborg et al., 2024), and position (Shi et al., 2024;Kudugunta et al., 2023).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Zhou et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wang et al., 2023 source

      Section 2.2, “Revealing the Positional Bias,” shows that swapping candidate responses between the Assistant 1 and Assistant 2 positions changes LLM evaluation results. That supports sensitivity to response order, but the paper does not establish sensitivity to the order of evaluation labels—the specific point attributed to Wang et al. here. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Stureborg et al., 2024 source

  • Classified as probably not a substantial claim.

    Premisecitation

    GPT-4o-mini is a lower-cost, reduced-capacity variant used to study how evaluation reliability changes under constrained model capacity.

    “GPT-4o-mini (OpenAI, 2023) is a lowercost, reduced-capacity variant used to study how evaluation reliability changes under constrained model capacity.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites OpenAI, 2023 source

      The Introduction (p. 1) and Scope (p. 2) describe GPT-4, while the scaling discussion (pp. 3–4) compares it with smaller models. Searches for “GPT-4o-mini” and the claimed lower-cost, reduced-capacity evaluation role found no passage addressing that model or its use to study evaluation reliability. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SumPubMed is situated in the biomedical research domain, where the task is long-form scientific summarization, with outputs being system-generated summaries of PubMed articles and expert judgments on readability, coherence, redundancy, and informativeness.

    “SumPubMed (Gupta et al., 2021) is situated in the biomedical research domain, where the task is long-form scientific summarization.The outputs are system-generated summaries of PubMed articles, with expert judgments on readability, coherence, redundancy, and informativeness.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Gupta et al., 2021 source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SummEval is a news domain dataset where the task is abstractive summarization, with outputs being system-generated summaries of CNN/Daily Mail articles and human ratings on coherence, consistency, fluency, and relevance.

    “SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Fabbri et al., 2021 source

      The Introduction (p. 2) describes released model-generated summaries of CNN/DailyMail articles with human judgments, and the human-annotation table (p. 8) lists coherence, consistency, fluency, and relevance ratings. However, Section 3.2 (p. 5) says the included models use both extractive and abstractive approaches, so describing SummEval’s task as solely abstractive summarization is too narrow. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Mixtral 8×22B represents open-weight mixture-of-experts models, allowing examination of rubric generation and application under sparse activation settings (64K context).

    “Mixtral 8×22B (Jiang et al., 2024) represents open-weight mixture-of-experts models, allowing us to examine rubric generation and application under sparse activation settings (64K context).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Jiang et al., 2024 source

      The Introduction identifies Mixtral as an open-weight, sparsely activated mixture-of-experts model, supporting the general model description. But the Introduction and Abstract specify Mixtral 8×7B with a 32K-token context, not Mixtral 8×22B with a 64K context. Rubric generation and application are the citing authors’ proposed uses, not findings attributed to this source. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs favor their own generations.

    “Studies have also shown that LLMs favor their own generations (Koo et al., 2023), or fail to produce stable preferences across reruns (Stureborg et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Koo et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Qwen 2.5-72B-Instruct is a strong open-weight dense model with long-context support (128K), commonly used in recent evaluation and benchmarking work.

    “Qwen 2.5-72B-Instruct (Yang et al., 2024) is a strong open-weight dense model with longcontext support (128K), commonly used in recent evaluation and benchmarking work.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Yang et al., 2024 source

      The Qwen2-72B-Instruct section evaluates Qwen2-72B-Instruct, not the cited sentence’s Qwen 2.5-72B-Instruct. Section 3.2 describes long-context support for Qwen2, but our copy does not address whether Qwen 2.5-72B-Instruct has 128K support or is commonly used in recent benchmarks (searched for the model name and 128K context). (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    USR is an open-domain dialogue dataset where the task is dialogue response generation, with outputs being continuations of dialogue contexts and expert evaluators providing ratings across six dimensions of dialogue quality.

    “USR (Mehri and Eskenazi, 2020) is an opendomain dialogue dataset, where the task is dialogue response generation.The outputs are continuations of dialogue contexts, and expert evaluators provide ratings across six dimensions of dialogue quality.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Mehri and Eskenazi, 2020 source

      Section 3 (Human Quality Annotation; page unavailable) describes responses to open-domain dialogue contexts rated by human annotators on six quality dimensions, and Section 3.4 identifies the annotators as dialogue researchers. But the Introduction and Conclusions identify USR as an evaluation metric, not the name of a dialogue dataset; the annotations were collected using Topical-Chat and PersonaChat. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GPT-4o serves as a high-capacity proprietary judge model, frequently adopted as a reference in evaluation pipelines, with a 128K context window.

    “GPT-4o (OpenAI, 2023) serves as a highcapacity proprietary judge model, frequently adopted as a reference in evaluation pipelines, with a 128K context window.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites OpenAI, 2023 source

      The Introduction (p. 1) identifies the model in this report as GPT-4, not GPT-4o. Searches of our copy for GPT-4o, judge-model use, and a 128K context window found no passage addressing those claims. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    LLMs can generate coherent and interpretable evaluation rubrics that capture task-relevant quality dimensions such as coherence, fluency, and informativeness, and show consistency across prompting conditions.

    “LLMs reliably generate interpretable and task-aware evaluation dimensions and apply them within models, but their scoring reliability degrades in factual and knowledge-intensive settings.”
    • Also stated in: § 1 Introduction · plain
      “Our experiments across four datasets on NLG tasks show that LLMs can generate coherent and interpretable rubrics that capture task-relevant quality dimensions such as coherence, fluency, and informativeness, showing consistency across prompting conditions.”
    • Also stated in: § 4.1 Rubric Generation · plain
      “Overall, the models generated five to eight rubrics per dataset, showing an ability to construct structured rubrics.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Agreement drops on SummEval and especially SumPubMed.

    “In contrast, agreement drops on SummEval and especially SumPubMed, where tasks require factual reasoning or domain knowledge.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Models capture human preferences for clarity, fluency, and engagement, but struggle with rubrics linked to factuality.

    “Models capture human preferences for clarity, fluency, and engagement, but struggle with rubrics linked to factuality.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human scores on the datasets are valid gold standards for evaluating LLM-based scores.

  • May combine two claims.

    Findingown data

    For USR, rubrics generated by both GPT-4o and Llama achieve strong inter-model reliability (ICC ≈ 0.7-0.8, α ≈ 0.74-0.81) and high cross-model correlations (ρ ≈ 0.7-0.9).

    “For USR, rubrics generated by both GPT-4o and Llama achieve strong inter-model reliability (ICC ≈ 0.7-0.8,α ≈ 0.74-0.81)and high cross-model correlations (ρ ≈ 0.7-0.9).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4o remains robust across all setups, showing stable alignment and low rubrics redundancy.

    “GPT-4o remains robust across all setups, showing stable alignment and low rubrics redundancy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    When using human rubrics, model-human alignment decreases, and variation across domains becomes more pronounced.

    “When using human rubrics, model-human alignment decreases, and variation across domains becomes more pronounced.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human scores on the datasets are valid gold standards for evaluating LLM-based scores.

  • May combine two claims.

    Findingown data

    Few-shot prompting slightly improves reliability in dialogue-oriented datasets, but has limited or inconsistent benefit for summarization.

    “Few-shot prompting slightly improves reliability in dialogue-oriented datasets, but has limited or inconsistent benefit for summarization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Correlations with human scores reach ρ ≈ 0.8-0.9 for HelpSteer2 and USR, but fall to around 0.5 for SummEval and below 0.3 for SumPubMed.

    “As shown in Tables 4, 5, and 13, correlations with human scores reach ρ ≈ 0.8-0.9 for HelpSteer2 and USR, but fall to around 0.5 for SummEval and below 0.3 for SumPubMed.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human scores on the datasets are valid gold standards for evaluating LLM-based scores.

  • May combine two claims.

    Findingown data

    Agreement between zero-shot and few-shot settings typically falls between 70-90%, with correlations above 0.8 for most configurations.

    “Agreement between zero-shot and few-shot settings (Table 3) typically falls between 70-90%, with correlations above 0.8 for most configurations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All outputs were automatically validated; malformed responses were re-prompted up to three times before being discarded.

    “All outputs were automatically validated; malformed responses were re-prompted up to three times before being discarded.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The work treats LLMs as static evaluators, without modeling temporal drift or adaptation over time.

    “Finally, our work treats LLMs as static evaluators, without modeling temporal drift or adaptation over time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Results may not generalize to smaller, multilingual, or domain-specialized models.

    “While these models cover diverse families, results may not generalize to smaller, multilingual, or domain-specialized models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Prompting tweaks and calibration strategies often address superficial variances rather than the deeper challenge that LLMs are instructed to interpret static human-defined rubrics that may not align with their internal language quality representation.

    “While prompting tweaks and calibration strategies have been proposed (Jang et al., 2023;Kudugunta et al., 2023), these methods often address superficial variances rather than the deeper challenge, i.e., LLMs are instructed to interpret static human-defined rubrics that may not align with their internal language quality representation.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Jang et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Kudugunta et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: Human rubrics are often static and misaligned with how models internally represent language quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In all cases, prompt content was normalized to ensure consistency across APIs.

    “In all cases, we normalized the prompt content to ensure consistency across APIs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Evaluation may be a learned linguistic behavior: models can coherently define and apply rubrics yet fail to generalize across architectures.

    “The results suggest that evaluation may itself be a learned linguistic behavior: models can coherently define and apply rubrics yet fail to generalize across architectures, indicating that their notion of quality arises from internal representations rather than fixed external standards.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The analysis focuses on textual outputs and rubric-based evaluations, without examining how multimodal inputs or interactive feedback might alter model behavior.

    “The analysis focuses on textual outputs and rubric-based evaluations, without examining how multimodal inputs or interactive feedback might alter model behavior.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    GER-Eval is a two-stage framework that decouples rubric generation from rubric application, enabling controlled analyses of where the assumption holds and fails across tasks, prompt conditions, and model families.

    “By decoupling rubric generation from rubric application, we enable controlled analyses of where this assumption holds and where it fails across tasks, prompt conditions, and model families.”
    • Also stated in: § 3 GER-Eval: A Framework for LLM-Generated Evaluation Rubrics · plain
      “GER-Eval is a two-stage framework that decouples rubric generation from rubric application, enabling us to analyze two complementary questions: (i) How LLMs conceptualize evaluation rubrics?; and (ii) How consistently do LLMs apply their own evaluation rubrics?”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human rubrics are often static and misaligned with how models internally represent language quality.
    • Rests on an assumption: The assumption that LLM self-defined rubrics are reliable and lead to improved and consistent scores has not yet been systematically tested and verified in the literature.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All models were run with temperature set to 0 for scoring tasks to ensure determinism, using greedy decoding.

    “All models were run with temperature set to 0 for scoring tasks to ensure determinism, using greedy decoding.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Together, the datasets span diverse domains and varying dependence on factual correctness, allowing testing of both generality and failure modes of rubric generation and application.

    “Together, they span diverse domains and varying dependence on factual correctness, allowing us to test both generality and failure modes of rubric generation and application.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    We manually inspected a set of merged and nonmerged pairs to verify that high-similarity pairs correspond to genuine semantic duplicates (e.g., Brevity vs. Conciseness), and used the embedding-based merging thereafter.

    “We manually inspected a set of merged and nonmerged pairs to verify that high-similarity pairs correspond to genuine semantic duplicates (e.g., Brevity vs. Conciseness), and used the embeddingbased merging thereafter.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Together, these datasets cover news, conversational, instructional, and scientific domains, offering a broad set of text generation tasks on which to test the generalizability of GER-Eval.

    “Together, these datasets cover news, conversational, instructional, and scientific domains, offering a broad set of text generation tasks on which to test the generalizability of GER-Eval.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    LLM-based evaluation is reliable for assessing linguistic quality but unreliable for factual reasoning.

    “These findings position LLM-based evaluation as reliable for assessing linguistic quality; however, unreliable for factual reasoning.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human scores on the datasets are valid gold standards for evaluating LLM-based scores.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Random seeds were fixed across runs for reproducibility.

    “Random seeds were fixed across runs for reproducibility.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Factual reasoning and biomedical knowledge remain model-dependent.

    “The drop in both reliability and correlation highlights that factual reasoning and biomedical knowledge remain modeldependent, and that domain-grounded evaluation rubrics are harder to generalize without shared knowledge representations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study does not independently verify factual accuracy beyond human reference scores.

    “Although we include factual and knowledge-intensive tasks, we do not independently verify factual accuracy beyond human reference scores.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Existing research builds on the assumption that LLMs' self-defined rubrics are reliable and lead to improved and consistent scores, as opposed to human-defined rubrics.

    “Existing research builds on the assumption that the LLMs' selfdefined rubrics are reliable and lead to improved and consistent scores, as opposed to human-defined rubrics.”
    • Also stated in: § 2 Related Work · plain
      “Existing research builds on the assumption that the LLMs' self-defined rubrics are reliable and lead to improved and consistent scores, as opposed to human-defined rubrics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    There is a reliability-validity trade-off: LLMs are procedurally stable but semantically limited.

    “This distinction reflects a broader reliability-validity trade-off: LLMs are procedurally stable but semantically limited.”
    Quote in the paperSubstantial claim
39 of 39

  • The paper relies on this without support.

    Human rubrics are often static and misaligned with how models internally represent language quality.

    “However, human rubrics are often static and misaligned with how models internally represent language quality.”
    “LLMs are instructed to interpret static human-defined rubrics that may not align with their internal language quality representation.”

  • The paper relies on this without support.

    The assumption that LLM self-defined rubrics are reliable and lead to improved and consistent scores has not yet been systematically tested and verified in the literature.

    “However, this assumption has not yet been systematically tested and verified in the literature.”
    “This assumption has not yet been systematically tested and verified in the literature.”

  • The paper relies on this without support.

    Human scores on the datasets are valid gold standards for evaluating LLM-based scores.

    “Each dataset provides human scores on multiple quality attributes, enabling direct validity analysis via correlation between LLM-based scores and human judgments.”

  • The paper relies on this without support.

    The human reference scores in the datasets are valid ground truth for evaluating LLM judgments.

    “Spearman correlation of zero-shot and few-shot LLM scores with human scores across datasets (using human rubrics).”

  • The paper relies on this without support.

    The human-defined rubrics are a valid standard for assessing the alignment of LLM-generated rubrics.

    “We estimate overlap with the dataset's human rubric via LLM-based tagging.”

  • The paper relies on this without support.

    The LLM-based tagger (GPT-4o) accurately maps generated criteria to human rubric items.

    “We use GPT-4o (temperature 0) as a tagger conditioned on the task description, the human rubric items (teacher), and the generated criteria (student).”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • HelpSteer2 human-score correlations are overstated

    Major

    The text says correlations with human scores reach about 0.8-0.9 for HelpSteer2, but Table 5 reports HelpSteer2 correlations of 0.31-0.65 across models and scoring conditions. The approximately 0.8 values appear in the USR columns, not HelpSteer2.

  • Most self-rubric correlations do not exceed 0.8

    Major

    Table 3 reports correlations above 0.8 in only 8 of its 20 model-dataset configurations, contradicting the claim that they exceed 0.8 for most configurations.

  • USR transfer reliability is overstated for GPT-4o rubrics

    Minor

    The stated ICC and α ranges are presented as applying to rubrics from both GPT-4o and Llama. Table 7 reports substantially lower values for several GPT-4o rubrics, including Creativity (ICC 0.3979, α 0.3931) and Humor (ICC 0.4929, α 0.4853).

  • SumPubMed is not the lowest-alignment dataset in Table 1

    Minor

    Table 1 shows SumPubMed alignment below 60% in three of ten model-condition entries, versus four of ten for SummEval. SummEval's alignment percentages are also generally lower, contradicting the description of SumPubMed as the lowest and often below 60%.

  • A reported rubric-generation average falls below five

    Minor

    The text says models generated five to eight rubrics per dataset, but Table 2 reports an average of 4.5 per dataset for Llama under the Good/Bad condition. That average also entails at least one dataset with fewer than five rubrics.

5 of 5

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 6

  1. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-409. · DOI · Paper page
  2. Vivek Gupta, Prerna Bharti, Pegah Nokhiz, and Harish Karnick. 2021. SumPubMed: Summarization dataset of PubMed scientific articles. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing: Stu- dent Research Workshop, pages 292-303, Online. Association for Computational Linguistics. · DOI · Paper page
  3. Shikib Mehri and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707, Online. Association for Computational Linguistics. · DOI · Paper page
  4. Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. Branch- solve-merge improves large language model evalu- ation and generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
  5. Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hart- mann, Aditya G. Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm- assisted evaluation of llm outputs with human pref- erences. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST). · DOI · Paper page
  6. Wanjun Zhong, Tao Yu, and Danqi Chen. 2020. Qags: Evaluating factual consistency with qa-based metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 519-529. · DOI

Cited by

Sorted by citations

Authors
Siro, C., Aliannejadi, P., & Aliannejadi, M.
Affiliations
  • Centrum Wiskunde & Informatica (NL)
  • Shahid Beheshti University (IR)
ORCID
Venue
Findings of the Association for Computational Linguistics: EACL 2026
Publisher
Association for Computational Linguistics
Published
2026
Pages
6371-6389
DOI
10.18653/v1/2026.findings-eacl.335
OpenAlex
W7140126704
Cited by
4
References
0
Citations by year
2026: 4
FWCI
35.54
Type
conference paper
Open access
Licence
cc-by
Locations
Findings of the Association for Computational Linguistics: EACL (publishedVersion, cc-by)
Funders
  • Universiteit van Amsterdam
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWA.1518.22.105)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
large language models, assessment rubrics, natural language generation evaluation, semantic coherence, scoring reliability, human-LLM alignment, cross-model generalization, GPT-4o, llamas, interpretable evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review