CoChat
Sign Up
Cited by 3Topic ModelingNatural Language Processing TechniquesInformation Retrieval and Search Behavior

MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions

Open access
Jinji, Michinori, Atarashi, Kyohei, Takeuchi, Koh, Kashima, Hisashi·2026·Findings of the Association for Computational Linguistics: ACL 2026
DOI 10.18653/v1/2026.findings-acl.1351
Open this paper in CoChat — free accountOpen PDF

Abstract

LLM-as-a-Judge, which uses LLMs to evaluate responses to open-ended questions, has seen significant growth in recent years.It has been adopted as a scalable alternative to manual human evaluation, such as crowdsourcing, which is often time-consuming and costly.However, the discrepancy between LLM-generated evaluations and human evaluations remains a critical problem in this field.To bridge this gap, we propose Multi-Aspect Panels of LLM Evaluators (MAPLE), a framework that orchestrates evaluations across multiple criteria using multiple LLMs.MAPLE integrates criterion-wise pairwise evaluations from multiple LLMs by estimating the importance of criteria and the reliability of individual evaluators.We conduct experiments with both open-source and closedsource models.Our results demonstrate that MAPLE achieves superior alignment with human evaluations compared to baselines, highlighting the importance of employing multiagent and multi-criteria evaluation strategies.

AI snapshot

Written from the paper’s full text.

TL;DR

MAPLE combines multi-criteria rubric decomposition with a heterogeneous panel of LLM evaluators, aggregating criterion-wise pairwise comparisons via an extended Crowd-BT model that learns evaluator reliability and criterion weights. On an essay-scoring task, it aligns better with human scores than multi-agent or multi-criteria baselines and remains robust to several biased or low-performing evaluators.

Key findings

  • In synthetic reliability tests, MAPLE recovered reliability and criterion-weight rankings when evaluator accuracies ranged from 60%–100% and 10%–100% (CI 1.00 for reliability and criteria; target-score CI 0.997 and 0.998), but failed in the 0%–90% setting (reliability CI 0.00, criteria CI 0.00, target CI 0.544).
  • On ASAP 2.0 essay scoring, MAPLE outperformed Crowd-BT and Majority Vote across all seven themes; for example, MAPLE scored CI 0.794 on “A Cowboy Who Rode the Waves,” 0.832 on “Car-free cities,” and 0.880 on “The electoral college.”
  • Ablation over 70 prompt-resampling runs showed Full MAPLE had the highest mean CI (0.814±0.059) and ranked first in 50/70 runs (71.4%), versus Fixed-Reliability 0.789±0.058, Fixed-Criterion Weights 0.800±0.061, and Collapsed-Criteria 0.793±0.064.
  • MAPLE remained robust with negligible performance loss when up to four adversarial models were added, but declined when five Position-1- or Position-2-biased models were introduced.

Methods

  • MAPLE takes a holistic rubric and target texts, then runs three steps: criteria generation (4–6 analytic criteria), AHP-style pairwise comparisons of texts under each criterion and of criteria against each other, and aggregation via an extended Crowd-BT model estimating criterion weights \(w_c\), criterion-specific target scores \(\theta_{i,c}\), and evaluator reliability \(\eta_k\).
  • Experiments used synthetic data (50 target items, 5 criteria) for reliability analysis and the ASAP 2.0 dataset for essay scoring: seven prompts, US students in grades 6–10, 50 essays sampled per theme, repeated 10 times. Evaluators included Llama-3-8B-instruct, Ministral-8B, Qwen-2.5-7B-instruct, Llama-3.3-70B-instruct, Qwen-3-32B, GPT-5.1, GPT-4o-mini, Gemini-2.5-flash, Gemini-2.5-flash-lite, and Claude-3.5-haiku; GPT-5.2Pro served as the criteria Generator.
  • Performance was measured with the Concordance Index; aggregation minimized the extended Crowd-BT loss by gradient descent with 5,000 epochs and learning rate 0.01.

Limitations

  • The paper states that MAPLE was evaluated only on rubric-based essay scoring, so effectiveness on other open-ended tasks such as creative writing and open-domain question answering remains unvalidated.
  • The paper states that multi-criteria pairwise comparisons increase the number of required evaluations as the number of criteria grows, creating a granularity–cost trade-off.
  • The text also makes evident that aggregation can degrade when biased evaluators form a majority, because it relies on majority agreement; this explains the drop with five position-biased models and the observation that full-model MAPLE rarely ranks first, most often ranking second, third, or fourth.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2026.findings-acl.1351

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 8 cited sources checked · 2 supported · 0 flagged

  • Premisecitation

    The discrepancy between LLM-generated evaluations and human evaluations remains a critical problem in this field.

    “However, the discrepancy between LLM-generated evaluations and human evaluations remains a critical problem in this field.”
    • Also stated in: § 1 Introduction · plain
      “Despite this promise, evaluations produced by LLMs often do not align with human labels, with prior work pointing to factors such as systematic biases and sensitivity to the chosen evaluator model and task (Ye et al., 2025;Chen et al., 2024;Wang et al., 2024;Li et al., 2024).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Ye et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chen et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wang et al., 2024 source
  • Findingown data

    MAPLE using all models consistently ranks high relative to single models, most frequently occupying the 2nd, 3rd, or 4th positions, aggregated across themes and iterations.

    “MAPLE consistently ranks high, most frequently occupying the 2nd, 3rd, or 4th positions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The evaluator LLMs' forced-choice pairwise decisions are reliable enough to estimate criterion weights and essay scores.
    • Rests on an assumption: The ASAP 2.0 holistic scores are a valid ground truth for essay quality.
  • Findingown data

    MAPLE attains strong average performance over individual evaluators.

    “Our results indicate that MAPLE attains strong average performance over individual evaluators and remains robust to lowperforming or biased LLMs.”
    Quote in the paperSubstantial claim
  • Findingown data

    The original pool already includes a Position-2-biased model, Ministral-8B.

    “The original pool already includes a Position-2-biased model, Ministral-8B, as shown in Figure 6.”
    Quote in the paperSubstantial claim
  • Findingown data

    No single model demonstrates consistent superiority over the MAPLE framework across all themes.

    “no single model demonstrates consistent superiority across all themes.”
    • Also stated in: § Additional information · plain
      “no single model consistently surpasses it across all themes”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The evaluator LLMs' forced-choice pairwise decisions are reliable enough to estimate criterion weights and essay scores.
    • Rests on an assumption: The ASAP 2.0 holistic scores are a valid ground truth for essay quality.
  • Classified as probably not a substantial claim. 9
  • May combine two claims. 19

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior studies have reported that LLM evaluations can be competitive with crowdworker annotations.

    “Prior studies have also reported that LLM evaluations can be competitive with crowdworker annotations (Gilardi et al., 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Gilardi et al., 2023 source

  • May combine two claims.

    Findingreasoning

    The full-model performance rarely ranks first and most frequently ranks second, third, or fourth.

    “This observation is corroborated by Figure 4, which shows that the full-model performance rarely ranks first and most frequently ranks second, third, or fourth.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Full MAPLE attains the highest mean CI (0.814) and ranks first in 50 out of 70 runs (71.4%).

    “Full MAPLE attains the highest mean CI (0.814) and ranks first in 50 out of 70 runs (71.4%), indicating both strong average performance and stability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAPLE remains robust with negligible performance loss when up to four adversarial models are added, but declines when five Position-1- or Position-2-biased models are introduced.

    “MAPLE remains robust with negligible performance loss when up to four adversarial models are added, but declines when five Position-1-or Position-2-biased models are introduced.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAPLE achieves higher performance than methods that either aggregate multiple LLMs without explicit criteria or use a single LLM for multi-criteria evaluation.

    “Our results further show that MAPLE achieves higher performance than methods that either (i) aggregate multiple LLMs without explicit criteria or (ii) use a single LLM for multi-criteria evaluation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In some themes, some single models outperform the MAPLE framework, and model performance varies by theme.

    “In some themes, some single models outperform the framework, and model performance varies by theme.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The evaluator LLMs' forced-choice pairwise decisions are reliable enough to estimate criterion weights and essay scores.
    • Rests on an assumption: The ASAP 2.0 holistic scores are a valid ground truth for essay quality.

  • May combine two claims.

    Findingown data

    MAPLE outperforms both Crowd-BT and Majority Vote across all themes.

    “Table 3 presents the comparison results between MAPLE, Crowd-BT, and Majority Vote.MAPLE outperforms both Crowd-BT and Majority Vote across all themes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The Crowd-BT baseline used in Section 5.2 requires K N^2 pairwise comparisons.

    “the Crowd-BT baseline used in Section 5.2 requires K N 2 pairwise comparisons”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    In the 0%-90% setting, performance drops substantially, and the method fails to accurately estimate model reliability or criterion weights.

    “In the 0%-90% setting, performance drops substantially, and the method fails to accurately estimate model reliability or criterion weights.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The Step-3 aggregation (optimization of the extended Crowd-BT model) runs on CPU and typically completes within a few minutes per run.

    “the Step-3 aggregation (optimization of the extended Crowd-BT model) runs on CPU and typically completes within a few minutes per run.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The Pairwise (single model) baseline requires N^2 comparisons.

    “the Pairwise (single model) baseline requires N 2 .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Multi-criteria methods often use only one LLM, so their results depend heavily on that model and its biases.

    “On the other hand, multi-criteria methods often use only one LLM, so their results depend heavily on that model and its biases.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Lu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Jordán et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Multi-agent methods lack clear evaluation criteria and assume all models are equally reliable, which can hurt accuracy and make it unclear which aspects contribute to the final evaluation, especially when low-quality models are involved.

    “In many settings, multi-agent methods lack clear evaluation criteria and assume all models are equally reliable, which can hurt accuracy and make it unclear which aspects contribute to the final evaluation, especially when low-quality models are involved.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Qian et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chan et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This reflects a trade-off between evaluation granularity and cost.

    “This reflects a trade-off between evaluation granularity and cost, and motivates future work on more efficient comparison or sampling strategies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    If malicious models form the majority, the resulting consensus becomes misleading and leads to poor performance.

    “If malicious models form the majority, the resulting consensus becomes misleading and leads to poor performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Injecting additional Position-2-biased models shifts the majority more strongly than injecting Position-1-biased models, resulting in a larger performance drop.

    “Consequently, injecting additional Position-2-biased models shifts the majority more strongly than injecting Position-1-biased models, resulting in a larger performance drop.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All prompts enforce a strict JSON-only output schema.

    “All prompts enforce a strict JSON-only output schema”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Learning criterion importance wc provides additional benefits beyond reliability learning, although its impact is smaller than that of ηk.

    “suggesting that learning criterion importance w c provides additional benefits beyond reliability learning, although its impact is smaller than that of η k .”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The holistic rubric is provided only to the criterion-generation step (Step 1) to derive analytic evaluation criteria, and the evaluator models in Step 2 do not see the holistic rubric.

    “In our pipeline, this rubric is provided only to the criterion-generation step (Step 1) to derive analytic evaluation criteria, and it is shared across all themes used in our experiments.”
    • Also stated in: § Additional information · plain
      “The evaluator models in Step 2 do not see the holistic rubric and instead receive only the generated criterion text to perform pairwise comparisons.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Outperforming Majority Vote indicates that, given the variability in LLM capabilities, weighing opinions based on individual reliability yields better prediction accuracy than treating all models equally.

    “Furthermore, outperforming Majority Vote indicates that, given the variability in LLM capabilities, weighing opinions based on individual reliability yields better prediction accuracy than treating all models equally.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Without ground truth, judges that agree with the majority are given higher weight, while those that disagree are given lower weight.

    “Without ground truth, judges that agree with the majority are given higher weight, while those that disagree are given lower weight.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The proposed method yields the best performance on average, despite occasionally falling behind single models in specific instances, because LLM capabilities are not absolute but vary by task.

    “The fact that the proposed method yields the best performance on average, despite occasionally falling behind single models in specific instances, stems from the fact that LLM capabilities are not absolute but vary by task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    If a model response does not conform to the required format, the same model is re-queried with the same inputs until a valid response is obtained, and only the first valid response is used for aggregation.

    “If a model response does not conform to the required format (e.g., invalid JSON or extra text outside the JSON object), we re-query the same model with the same inputs until a valid response is obtained; only the first valid response is used for aggregation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The full-model performance rarely ranks first because the aggregation fundamentally favors a majority-vote mechanism, meaning it rarely surpasses the absolute best model in the pool.

    “As discussed in experiments for RQ1, this occurs because the aggregation funda-0.6 0.7 0.8 0.9 CI mentally favors a majority-vote mechanism, meaning it rarely surpasses the absolute best model in the pool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The experiments are inference-only.

    “Our experiments are inference-only.”
    Quote in the paperNot a substantial claim
25 of 25

  • The paper relies on this without support.

    Manual human evaluation, such as crowdsourcing, is often time-consuming and costly.

    “It has been adopted as a scalable alternative to manual human evaluation, such as crowdsourcing, which is often time-consuming and costly.”
    “human labeling itself is inherently costly and slow.”

  • The paper relies on this without support.

    In practice, a scenario where malicious models form the majority is unlikely in LLM-as-a-Judge settings.

    “In practice, such a scenario is unlikely in LLM-as-a-Judge settings, and we therefore regard this as a minor limitation.”

  • The paper relies on this without support.

    The analytic criteria generated in Step 1 are independent and faithful to the holistic rubric.

    “Propose **4-6 independent criteria** that are appropriate to the TASK_PROMPT and faithful to the HOLISTIC_RUBRIC.”

  • The paper relies on this without support.

    The evaluator LLMs' forced-choice pairwise decisions are reliable enough to estimate criterion weights and essay scores.

    “For each essay pair and each criterion, we ask an evaluator LLM to output a forced-choice decision, with ties disallowed.”

  • The paper relies on this without support.

    The ASAP 2.0 holistic scores are a valid ground truth for essay quality.

    “Appendix A reproduces the holistic rating form used to score essays in ASAP 2.0.”

  • The paper relies on this without support.

    The extended Crowd-BT model correctly aggregates pairwise comparisons into reliable scores.

    “the Step-3 aggregation (optimization of the extended Crowd-BT model) runs on CPU and typically completes within a few minutes per run.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conclusion claims robustness regardless of evaluators despite a reported failure

    Minor

    The conclusion makes an unqualified robustness claim, but the results report a setting in which performance drops substantially and MAPLE fails to estimate reliability or criterion weights accurately.

1 of 1

Cites

Flagged first, then by citations

2 of the 27 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. Helia Hashemi, Jason Eisner, Corby Rosset, Ben- jamin Van Durme, and Chris Kedzie. 2024. LLM- rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. In Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics. · DOI · Paper page
  2. Joaquín Jordán, Xavier Yin, Melissa Fabros, Gireeja Ranade, and Narges Norouzi. 2026. MAGIC: Multi- agent argumentation and grammar integrated criti- quer. In Proceedings of the 16th Symposium on Edu- cational Advances in Artificial Intelligence. · DOI

Cited by

Sorted by citations

Related

Authors
Jinji, M., Atarashi, K., Takeuchi, K., & Kashima, H.
Affiliations
Kyoto University (JP)
ORCID
Venue
Findings of the Association for Computational Linguistics: ACL 2026
Publisher
Association for Computational Linguistics
Published
2026
Pages
27071-27088
DOI
10.18653/v1/2026.findings-acl.1351
OpenAlex
W7166807741
Cited by
0
References
27
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
Funders
Core Research for Evolutional Science and Technology (JPMJCR21D1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
LLM-as-a-judge, multi-criteria decision making, multi-agent evaluation, criteria weighting, human evaluation alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review