Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge
Open accessAbstract
Effective evaluation of large language models (LLMs) remains a critical bottleneck, as conventional direct scoring often yields inconsistent and opaque judgments. In this work, we adapt the Analytic Hierarchy Process (AHP) to LLM-based evaluation and, more importantly, propose a confidence-aware Fuzzy AHP (FAHP) extension that models epistemic uncertainty via triangular fuzzy numbers modulated by LLM-generated confidence scores. Systematically validated on JudgeBench, our structured approach decomposes assessments into explicit criteria and incorporates uncertainty-aware aggregation, producing more calibrated judgments. Extensive experiments demonstrate that both crisp and fuzzy AHP consistently outperform direct scoring across model scales and dataset splits, with FAHP showing superior stability in uncertain comparison scenarios. Building on these insights, we propose \textbf{DualJudge}, a hybrid framework inspired by Dual-Process Theory that adaptively fuses holistic direct scores with structured AHP outputs via consistency-aware weighting. DualJudge achieves state-of-the-art performance, underscoring the complementary strengths of intuitive and deliberative evaluation paradigms. These results establish uncertainty-aware structured reasoning as a principled pathway toward more reliable LLM assessment. Code is available at https://github.com/hreyulog/AHP_llm_judge.
Verifying this paper
8 of 9 steps done · this page updates itself — you can leave and come back
- Registry recordVerified
- Full textRead the full text
- Reference list24 references found
- Reference checkChecking each cited work — 22 of 22
- StatisticsRecomputed
- Claims61 claims anchored to a quote
- Cited sourcesChecked
- Self-checkChecked
- VerificationFindings verified
AI snapshot
Written from the paper’s full text.
TL;DR
The paper adapts AHP and confidence-aware Fuzzy AHP to LLM-as-a-Judge and proposes DualJudge, a hybrid that fuses direct holistic scores with structured AHP/FAHP scores via consistency-aware weighting. On JudgeBench, structured and hybrid methods generally outperform direct scoring, with DualJudge achieving peak accuracy in 7 of 8 configurations.
Key findings
- Structured evaluation outperformed direct scoring, especially for weaker judges: gpt-oss-20b 1-10 improved from 69.83% direct to 75.35% Fuzzy AHP and 77.60% DualJudge D+F (+7.77 pp); gpt-oss-120b 1-10 improved from 75.81% to 82.10% (+6.29 pp).
- Fuzzy AHP matched or exceeded crisp AHP across model families/scales; e.g., qwen3.5-9b 1-10 improved from 81.44% crisp to 83.55% fuzzy, and qwen3.5-35b 1-5 from 83.60% crisp to 86.24% fuzzy.
- DualJudge (D+C or D+F) achieved peak accuracy in 7 of 8 configurations; gains concentrated in reasoning-intensive domains, e.g., +11.9 pp for gpt-oss-20b on LiveCodeBench.
- For the strongest evaluator qwen3.5-35b under 1-10, direct scoring remained competitive at 87.38%, while structured variants reached 85.47–87.19%.
Methods
- Evaluated on JudgeBench pairwise comparison benchmark, 17 categories from LiveBench and MMLU-Pro; GPT split 350 samples, Claude split 270 samples (620 merged), with judge backbones gpt-oss-20b, gpt-oss-120b, Qwen3.5-9B, Qwen3.5-35B-A3B and 1-10/1-5 scoring scales.
- Crisp AHP derives criterion weights from pairwise comparison matrices via principal eigenvector, with consistency ratio threshold τ=0.15 and automatic consistency repair; FAHP maps comparisons to triangular fuzzy numbers modulated by LLM confidence scores, then defuzzifies.
- DualJudge fuses direct score S_abs and structured score S_ahp via CR-aware adaptive weighting with β=7; performance measured as agreement accuracy with ground-truth pairwise preferences, with all LLM interactions cached.
Limitations
- Stated: increased computational overhead from pairwise comparisons and running both direct and structured pipelines, potentially prohibitive for real-time or large-scale RLHF.
- Stated: dependency on criteria quality because experiments use Category Mode shared criteria per dataset category; dynamic per-sample criterion generation is future work.
- Stated: limited model/dataset scope (gpt-oss and qwen3.5 families, JudgeBench pairwise tasks) and fixed fusion hyperparameters; the text also makes evident that structured methods are not universally superior, since qwen3.5-35b direct scoring remains competitive or better under 1-10.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
5 of 10 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured
- Premisecitation§ 4.2 Results · p. 11
The Claude data split generally contains harder-to-distinguish response pairs.
“which generally contains harder-todistinguish response pairs [18].”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [18] 18. Tan, S., Zhuang, S., Montgomery, K., Tang, W.Y., Cuadron, A., Wang, C., Popa, R.A., Stoica, I.: Judgebench: A benchmark for evaluating llm-based judges (2024), https://arxiv.org/abs/2410.12784 source
- Findingown data§ 4.2 Results · p. 10
The sole exception to Fuzzy AHP leading occurs for qwen3.5-35b (1-10), where both AHP variants underperform Direct scoring.
“The sole exception occurs for qwen3.5-35b (1-10), where both AHP variants underperform Direct scoring”
Quote in the paperNot a substantial claim - Findingown data§ 4.2 Results · p. 11
The hybrid DualJudge approaches (D+C and D+F) achieve peak accuracy in 7 of 8 configurations.
“The hybrid DualJudge approaches (D+C and D+F) achieve peak accuracy in 7 of 8 configurations”
Quote in the paperSubstantial claim - Findingown data§ 4.2 Results · p. 10
Fuzzy AHP consistently matches or exceeds Crisp AHP across model families and scales.
“Fuzzy AHP consistently matches or exceeds Crisp AHP across model families and scales”
- Also stated in: § 5 Discussion · p. 12 · plain
“Fuzzy AHP consistently matches or exceeds Crisp AHP across model families and scales”
- Also stated in: § 7 Conclusion · p. 14 · plain
“with Fuzzy AHP further improving stability over Crisp AHP by explicitly modeling epistemic uncertainty”
Quote in the paperSubstantial claim - Also stated in: § 5 Discussion · p. 12 · plain
- Findingreasoning§ 4.2 Results · p. 11
The 1-5 scale generally yields slightly higher absolute accuracy than 1-10 across methods (e.g., gpt-oss-20b Direct: 71.77% vs. 69.83%).
“The 1-5 scale generally yields slightly higher absolute accuracy than 1-10 across methods (e.g., gpt-oss-20b Direct: 71.77% vs. 69.83%)”
Quote in the paperNot a substantial claim
- Findingreasoning§ 5 Discussion · p. 12
Factual or recall-heavy domains (e.g., history, law) exhibit smaller deltas, as direct scoring already captures salient signals efficiently.
“In contrast, factual or recall-heavy domains (e.g., history, law) exhibit smaller deltas, as direct scoring already captures salient signals efficiently.”
Quote in the paperSubstantial claim - Findingown data§ Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge · p. 1
Structured evaluation methods (AHP and DualJudge) generally outperform direct scoring across model scales and dataset splits.
“Extensive experiments demonstrate that both crisp and fuzzy AHP consistently outperform direct scoring across model scales and dataset splits”
- Also stated in: § 4.2 Results · p. 10 · plain
“Across the aggregated results , structured evaluation methods (AHP and DualJudge) generally outperform direct scoring”
- Also stated in: § 7 Conclusion · p. 14 · boosted
“structured methods (AHP/FAHP) consistently outperform direct scoring”
Quote in the paperSubstantial claim - Also stated in: § 4.2 Results · p. 10 · plain
- Findingreasoning§ 5 Discussion · p. 11
DualJudge consistently outperforms other approaches across model scales and dataset splits.
“The fact that DualJudge consistently outperforms other approaches across model scales and dataset splits”
Quote in the paperSubstantial claim - Findingown data§ 4.2 Results · p. 10
The magnitude of improvement from structured methods is inversely correlated with base model capability.
“with the magnitude of improvement inversely correlated with base model capability”
- Also stated in: § 5 Discussion · p. 12 · hedged
“The magnitude of performance gains is inversely correlated with base model capability”
Quote in the paperSubstantial claim - Also stated in: § 5 Discussion · p. 12 · hedged
- Findingown data§ 4.2 Results · p. 11
Hybrid approaches show the most consistent gains on the Claude data split.
“Notably, hybrid approaches show the most consistent gains on the Claude data split”
Quote in the paperNot a substantial claim - Findingreasoning§ 5 Discussion · p. 12
On cognitively complex benchmarks like LiveCodeBench and MMLU-Pro mathematics/physics, DualJudge yields substantial improvements (e.g., +11.9 pp for gpt-oss-20b on LiveCodeBench).
“On cognitively complex benchmarks like LiveCodeBench and MMLU-Pro mathematics/physics, DualJudge yields substantial improvements (e.g., +11.9 pp for gpt-oss-20b on LiveCodeBench)”
Quote in the paperSubstantial claim - Findingown data§ 4.2 Results · p. 10
The advantage of Fuzzy AHP over Crisp AHP is most pronounced for mid-tier models.
“The advantage is most pronounced for mid-tier models”
- Also stated in: § 5 Discussion · p. 12 · plain
“This advantage is most pronounced for mid-tier models and ambiguous tasks.”
Quote in the paperNot a substantial claim - Also stated in: § 5 Discussion · p. 12 · plain
- Findingown data§ 3.3 DualJudge · p. 9
Since all observed CR values lie within the reliable regime, β primarily modulates relative confidence rather than correcting severely inconsistent cases.
“Since all observed CR values lie within the reliable regime, β primarily modulates relative confidence rather than correcting severely inconsistent cases.”
Quote in the paperSubstantial claim - Interpretationown data§ 5 Discussion · p. 11
The empirical results validate the core premise that hybrid evaluation frameworks can effectively mitigate the inherent instability of LLM-based judgment.
“The empirical results presented in Section 4.2 validate the core premise of our work: that hybrid evaluation frameworks can effectively mitigate the inherent instability of LLM-based judgment.”
Quote in the paperSubstantial claim - Study design and conductreasoning§ 3.2 Crisp and Fuzzy AHP Evaluation · p. 8
The shared matrix principle ensures that differences between AHP and FAHP arise solely from uncertainty modeling rather than inconsistent evidence.
“Shared Matrix Principle Both crisp and fuzzy weights are derived from the same repaired comparison matrix. The fuzzy matrix is reconstructed from repaired scores while preserving confidence signals, ensuring that differences between AHP and FAHP arise solely from uncertainty modeling rather than inconsistent evidence.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 6 Limitations and Future Work · p. 13
The primary limitation of the proposed methodology is increased computational cost.
“The primary limitation of the proposed methodology is increased computational cost.”
Quote in the paperNot a substantial claim - Interpretationown data§ 5 Discussion · p. 12
Structured reasoning acts as a cognitive scaffold for weaker evaluators.
“indicating that structured reasoning acts as a cognitive scaffold.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 Discussion · p. 12
The framework imposes mathematical constraints (e.g., consistency ratio checks) and explicit criterion weighting, providing a more rigorous scaffold for judgment calibration than standard chain-of-thought prompting.
“Unlike standard chain-ofthought prompting, which relies on implicit step generation, our framework imposes mathematical constraints (e.g., consistency ratio checks) and explicit criterion weighting, providing a more rigorous scaffold for judgment calibration.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 Discussion · p. 12
Crisp AHP forces stochastic judgments into discrete Saaty scales, potentially discarding valuable nuance.
“Crisp AHP forces these stochastic judgments into discrete Saaty scales, potentially discarding valuable nuance.”
Quote in the paperSubstantial claim- Rests on an assumption: LLMs do not produce deterministic truths; their token distributions reflect varying degrees of confidence.
- Interpretationown data§ 4.2 Results · p. 11
Structured decomposition is most valuable for cognitively complex, multi-attribute evaluation tasks.
“This pattern reinforces that structured decomposition is most valuable for cognitively complex, multi-attribute evaluation tasks.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 6 Limitations and Future Work · p. 13
For real-time evaluation or large-scale RLHF data processing, the latency of AHP/DualJudge may be prohibitive.
“For real-time evaluation or large-scale RLHF data processing, this latency may be prohibitive.”
Quote in the paperNot a substantial claim - Interpretationown data§ 5 Discussion · p. 11
Combining fast, intuitive reasoning with slower, deliberative reasoning creates something greater than the sum of its parts.
“creates something greater than the sum of its parts”
Quote in the paperNot a substantial claim - Interpretationown data§ Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge · p. 1
The study's results establish uncertainty-aware structured reasoning as a principled pathway toward more reliable LLM assessment.
“These results establish uncertainty-aware structured reasoning as a principled pathway toward more reliable LLM assessment.”
Quote in the paperSubstantial claim - Study design and conductnone given§ 3.2 Crisp and Fuzzy AHP Evaluation · p. 6
Both crisp and fuzzy AHP formulations operate on a shared pairwise comparison structure, differing only in how uncertainty in LLM judgments is modeled and propagated during aggregation.
“Both formulations operate on a shared pairwise comparison structure, differing only in how uncertainty in LLM judgments is modeled and propagated during aggregation.”
Quote in the paperNot a substantial claim - Interpretationown data§ 4.2 Results · p. 11
The optimal fusion strategy may depend on the evaluator's inherent uncertainty calibration.
“suggesting that the optimal fusion strategy may depend on the evaluator's inherent uncertainty calibration.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 6 Limitations and Future Work · p. 13
If predefined criteria fail to capture key aspects of response quality, the structured evaluation may be systematically biased.
“If the predefined criteria fail to capture key aspects of response quality, the structured evaluation may be systematically biased.”
Quote in the paperSubstantial claim - Interpretationown data§ 4.2 Results · p. 10
Fuzzy logic cannot fully compensate when the base evaluator's holistic judgments are already near-optimal.
“indicating that fuzzy logic cannot fully compensate when the base evaluator's holistic judgments are already near-optimal.”
Quote in the paperSubstantial claim - Interpretationown data§ 7 Conclusion · p. 14
By explicitly modeling the evaluation process through decomposed criteria and uncertainty-aware aggregation, we can extract more reliable signals from existing models.
“By explicitly modeling the evaluation process through decomposed criteria and uncertainty-aware aggregation, we can extract more reliable signals from existing models.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 Discussion · p. 12
The adaptive fusion mechanism partially addresses the variability of gains across domains.
“Our adaptive fusion mechanism-which dynamically weights the structured branch based on consistency ratios and confidence-partially addresses this variability.”
Quote in the paperSubstantial claim - Interpretationown data§ 4.2 Results · p. 11
The consistency across model families, scales, and evaluator backends underscores the generalizability of the proposed approach.
“The consistency across model families, scales, and evaluator backends (GPT vs. Claude splits in Appendix) underscores the generalizability of the proposed approach.”
Quote in the paperSubstantial claim
- May combine two claims. 22
- Classified as probably not a substantial claim. 11
May combine two claims.
Premisecitation§ 3.1 Dataset · p. 5JudgeBench aggregates questions from LiveBench and MMLU-Pro, covering 17 fine-grained categories across STEM, humanities, and professional domains.
“JudgeBench aggregates questions from LiveBench [20] and MMLU-Pro [19], covering 17 fine-grained categories across STEM, humanities, and professional domains.”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites [20] 20. White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., Shubh-Agrawal, Sandha, S.S., Naidu, S., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., Goldblum, M.: Livebench: A challenging, contamination-limited llm benchmark (2025), https://arxiv.org/abs/2406. 19314 source
- cites [19] 19. Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., Chen, W.: Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (2024), https://arxiv.org/abs/2406.01574 source
May combine two claims.
Premisecitation§ 3.1 Dataset · p. 5JudgeBench constructs challenging pairwise comparison tasks with minimal quality disparity and reliable ground-truth preference labels, forcing fine-grained criteria-based analysis.
“JudgeBench constructs challenging pairwise comparison tasks: for each query, two model responses with minimal quality disparity are provided, along with a reliable ground-truth preference label. This design forces judge models to perform fine-grained, criteria-based analysis rather than relying on superficial heuristics”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [18] 18. Tan, S., Zhuang, S., Montgomery, K., Tang, W.Y., Cuadron, A., Wang, C., Popa, R.A., Stoica, I.: Judgebench: A benchmark for evaluating llm-based judges (2024), https://arxiv.org/abs/2410.12784 source
Section 3 describes challenging response pairs with objective ground-truth labels, and the Principle section says the incorrect responses contain subtle errors that make the pairs difficult to distinguish. The copy does not establish that each pair has a minimal quality disparity; it frames the challenge as distinguishing factual and logical correctness rather than specifying a minimum quality gap. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
Study design and conductcitation§ 4.1 Experimental Setup · p. 9Two scoring granularities (1-10 and 1-5 scales) are considered to analyze the effect of resolution on structured evaluation stability.
“Two scoring granularities are considered (1-10 and 1-5 scales [9]), allowing us to analyze the effect of resolution on structured evaluation stability.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [9] 9. Li, W., Zhao, M., Dong, W., Cai, J., Wei, Y., Pocress, M., Li, Y., Yuan, W., Wang, X., Hou, R., Lou, K., Zeng, W., Yang, Y., Du, Y., Wang, M.: Grading scale impact on llm-as-a-judge: Human-llm alignment is highest on 0-5 grading scale (2026), https://arxiv.org/abs/2601.03444 source
The paper’s Abstract says it studies scale choice and consistency, but specifies continuous 0–5, 0–10, and 0–100 ranges—not the cited 1–5 and 1–10 scales. The citing authors’ plan to analyze structured evaluation stability is their own, rather than a finding attributed to this source. (LLM, conf=0.88)
Experimental — accuracy not yet measured
May combine two claims.
Findingown data§ 4.2 Results · p. 10For gpt-oss-20b under the 1-10 scale, accuracy improves from 69.83% (Direct) to 75.35% (Fuzzy AHP) and peaks at 77.60% (DualJudge D+F).
“under the 1-10 scale, accuracy improves from 69.83% (Direct) to 75.35% (Fuzzy AHP) and peaks at 77.60% (DualJudge D+F).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 11Task-level analysis reveals that gains concentrate on reasoning-intensive benchmarks: LiveCodeBench and MMLU-Pro mathematics/physics show 8-12 pp improvements for weaker models under DualJudge, while simpler factual categories exhibit smaller deltas.
“Task-level analysis (see Appendix A) reveals that gains concentrate on reasoning-intensive benchmarks: LiveCodeBench and MMLU-Pro mathematics/physics show 8-12 pp improvements for weaker models under DualJudge, while simpler factual categories exhibit smaller deltas.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5 Discussion · p. 12Performance gains are heterogeneous across domains, confirming that no single paradigm is universally optimal.
“Performance gains are heterogeneous across domains, confirming that no single paradigm is universally optimal.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 7 Conclusion · p. 14DualJudge achieves the highest accuracy by adaptively fusing these signals, yielding significant gains particularly in reasoning-intensive domains and for smaller model scales.
“Ultimately, DualJudge achieves the highest accuracy by adaptively fusing these signals, yielding significant gains particularly in reasoning-intensive domains and for smaller model scales.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 10The trend of structured improvement persists for gpt-oss-120b (+4.7-6.3 pp) and qwen3.5-9b (+1.4-2.0 pp).
“The trend persists for gpt-oss-120b (+4.7-6.3 pp) and qwen3.5-9b (+1.4-2.0 pp).”
Quote in the paperSubstantial claimMay combine two claims.
Findingreasoning§ 5 Discussion · p. 12When base evaluators are highly calibrated (e.g., qwen3.5-35b), the overhead of fuzzy decomposition yields diminishing returns, and direct scoring remains competitive.
“However, when base evaluators are highly calibrated (e.g., qwen3.5-35b ), the overhead of fuzzy decomposition yields diminishing returns, and direct scoring remains competitive.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 11For stronger models, improvements from DualJudge diminish but remain positive: qwen3.5-9b gains +1.37-2.75 pp, and qwen3.5-35b shows marginal gains (+0.19 pp) or parity under 1-10.
“For stronger models, improvements diminish but remain positive: qwen3.5-9b gains +1.37-2.75 pp, and qwen3.5-35b shows marginal gains (+0.19 pp) or parity under 1-10.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 11Gains from DualJudge are largest for weaker models: gpt-oss-20b sees +7.77 pp improvement (69.83% → 77.60%) under 1-10 scale, while gpt-oss-120b gains +6.29 pp (75.81% → 82.10%).
“gpt-oss-20b sees a +7.77 pp improvement (69.83% → 77.60%) under the 1-10 scale, while gpt-oss-120b gains +6.29 pp (75.81% → 82.10%).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 10For qwen3.5-35b, the gap between Fuzzy and Crisp AHP narrows but Fuzzy still leads in 3 of 4 settings (e.g., 1-5 scale: 86.24% vs. 83.60%).
“For qwen3.5-35b , the gap narrows but Fuzzy still leads in 3 of 4 settings (e.g., 1-5 scale: 86.24% vs. 83.60%).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 10For gpt-oss-20b (1-5 scale), accuracy improves from 75.48% (Crisp) to 76.45% (Fuzzy), and for qwen3.5-9b (1-10) from 81.44% to 83.55%.
“e.g., gpt-oss-20b (1-5 scale) improves from 75.48% (Crisp) to 76.45% (Fuzzy), and qwen3.5-9b (1-10) from 81.44% to 83.55%.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 11The relative gains from structured methods remain consistent across scales, indicating robustness to scoring protocol.
“However, the relative gains from structured methods remain consistent across scales, indicating robustness to scoring protocol.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 11D+F (Fuzzy fusion) tends to lead for 20B-class models, while D+C (Crisp fusion) is competitive or superior for 9B/35B-class models.
“Notably, D+F(Fuzzy fusion) tends to lead for 20B-class models, while D+C (Crisp fusion) is competitive or superior for 9B/35B-class models”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Results · p. 10For qwen3.5-35b under the 1-10 scale, direct scoring (87.38%) remains competitive, with structured variants achieving 85.47-87.19%.
“for the strongest evaluator qwen3.5-35b under the 1-10 scale, direct scoring (87.38%) remains competitive, with structured variants achieving 85.4787.19%.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3.2 Crisp and Fuzzy AHP Evaluation · p. 7The consistency threshold τ = 0.15 is adopted as the consistency threshold.
“where τ = 0 . 15 [6] is adopted as the consistency threshold.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [6] 6. José, A.S., Alonso, Mª, T., Lamata: Consistency in the analytic hierarchy process: a new approach. Int. J. Uncertain. Fuzziness Knowl. Based Syst. 14 , 445-459 (2006), https://api.semanticscholar.org/CorpusID:18104088
Not checked in the free run.
May combine two claims.
Interpretationown data§ 7 Conclusion · p. 14Reliability is not solely a function of model scale, but also of reasoning architecture.
“reliability is not solely a function of model scale, but also of reasoning architecture.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Study design and conductnone given§ 4.1 Experimental Setup · p. 9All LLM interactions are cached to ensure reproducibility and eliminate sampling variance.
“All LLM interactions are cached to ensure reproducibility and eliminate sampling variance.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ 3.3 DualJudge · p. 9The consistency ratio CR is interpreted as a continuous reliability signal; lower CR values indicate stronger internal agreement and higher confidence in structured evaluation.
“Beyond binary filtering, we further interpret the CR as a continuous reliability signal [16,24]. Lower CR values indicate stronger internal agreement among pairwise comparisons and thus higher confidence in the structured evaluation.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [16] 16. Saaty, T., Vargas, L.: Models, Methods, Concepts & Applications of the Analytic Hierarchy Process. International Series in Operations Research & Management Science, Springer (2012), https://books.google.ru/books?id=6J9XI8I1qjwC source
- cites [24] 24. Zeshui, X., Cuiping, W.: A consistency improving method in the analytic hierarchy process1research supported by nsf of china and shandong.1. European Journal of Operational Research 116 (2), 443-449 (1999). https://doi.org/https://doi.org/10.1016/S0377-2217(98)00109-X , source
Classified as probably not a substantial claim.
Interpretationreasoning§ 3.3 DualJudge · p. 9The structured branch maintains a non-negligible influence even under moderate inconsistency.
“This formulation ensures that the structured branch maintains a non-negligible influence even under moderate inconsistency”
Quote in the paperSubstantial claim- The authors state this as an assumption: Partially consistent structured reasoning still provides useful signals.
Probably not a substantial claim: may not pass S4 (based).
Study design and conductnone given§ 3.1 Dataset · p. 6The use of two independent response sets (GPT Split and Claude Split) ensures that observed improvements are not artifacts of a specific response style or generator bias.
“These splits ensure that observed improvements are not artifacts of a specific response style or generator bias.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 6 Limitations and Future Work · p. 13Optimal fusion parameters may vary across different model families or task complexities.
“optimal parameters may vary across different model families or task complexities.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 5 Discussion · p. 12By mapping pairwise comparisons to triangular fuzzy numbers (TFNs) modulated by confidence scores, Fuzzy AHP preserves uncertainty, allowing the aggregation mechanism to down-weight less reliable judgments.
“By mapping pairwise comparisons to triangular fuzzy numbers (TFNs) modulated by confidence scores, Fuzzy AHP preserves uncertainty, allowing the aggregation mechanism to down-weight less reliable judgments.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 6 Limitations and Future Work · p. 13The generalizability of the findings to single-response scoring or open-weight models remains to be fully verified.
“The generalizability to singleresponse scoring (e.g., reward modeling for RLHF) or open-weight models (e.g., Llama, Qwen) remains to be fully verified.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductcitation§ 4.1 Experimental Setup · p. 9The study evaluates two model scales for each model across both GPT and Claude response splits to ensure robustness to generator variation.
“We evaluate two model scales for each model ( gpt-oss-20b , gpt-oss-120b ) [12] and Qwen3.5-35B-A3B , Qwen3.5-9B across both GPT and Claude response splits to ensure robustness to generator variation.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [12] 12. OpenAI: gpt-oss-120b & gpt-oss-20b model card (2025), https://arxiv.org/abs/ 2508.10925
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationreasoning§ 4.2 Results · p. 11The 1-5 scale yields higher accuracy likely due to reduced granularity easing the judgment burden.
“likely due to reduced granularity easing the judgment burden.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3.2 Crisp and Fuzzy AHP Evaluation · p. 7The repair enforces approximate multiplicative transitivity, which characterizes consistent pairwise comparisons in AHP.
“The repair enforces approximate multiplicative transitivity, a ik ≈ a ij a jk , which characterizes consistent pairwise comparisons in the analytic hierarchy process [15].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [15] 15. Saaty, T.: The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. Advanced book program, McGraw-Hill International Book Company (1980), https://books.google.ru/books?id=Xxi7AAAAIAAJ
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Interpretationnone given§ 6 Limitations and Future Work · p. 13The performance of AHP is inherently bound to the relevance and completeness of the criteria.
“The performance of AHP is inherently bound to the relevance and completeness of these criteria.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 4.2 Results · p. 10Decomposed reasoning provides the most value when base judgments are noisy or poorly calibrated, while highly capable models may already internalize multi-criteria trade-offs.
“This suggests that decomposed reasoning provides the most value when base judgments are noisy or poorly calibrated, while highly capable models may already internalize multi-criteria trade-offs.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 5 Discussion · p. 12Weaker models often lack calibrated internal representations for complex evaluation tasks and benefit significantly from explicit decomposition and consistency enforcement of AHP.
“Weaker models, which often lack calibrated internal representations for complex evaluation tasks, benefit significantly from the explicit decomposition and consistency enforcement of AHP.”
Quote in the paperSubstantial claim
The paper relies on this without support.
The effectiveness of structured hierarchical evaluation as a general paradigm for LLM-as-a-Judge remains insufficiently understood, and systematic comparisons with holistic judging across heterogeneous domains are largely missing.
“Nevertheless, the effectiveness of structured hierarchical evaluation as a general paradigm for LLM-as-a-Judge remains insufficiently understood, and systematic comparisons with holistic judging across heterogeneous domains are largely missing.”
The paper relies on this without support.
LLMs do not produce deterministic truths; their token distributions reflect varying degrees of confidence.
“LLMs do not produce deterministic truths; their token distributions reflect varying degrees of confidence.”
The authors state this as an assumption.
Partially consistent structured reasoning still provides useful signals.
“reflecting the assumption that partially consistent structured reasoning still provides useful signals.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
AHP does not consistently outperform direct scoring
MajorThe paper claims that both AHP variants consistently outperform direct scoring, but the reported 1-10 results for qwen3.5-35b put Direct at 87.38%, above both Crisp AHP (83.65%) and Fuzzy AHP (85.47%).
“both crisp and fuzzy AHP consistently outperform direct scoring across model scales and dataset splits”
“| | 1-10 | 620 | 87.38% | 83.65% | 85.47% | 87.19% | 87.19% |”
DualJudge is described as both simple averaging and adaptive fusion
MajorThe introduction says DualJudge combines its two scores by simple averaging, whereas the methodology assigns the structured branch a consistency-dependent weight controlled by β. Those are different aggregation rules.
“DualJudge aggregates the two scores through simple averaging”
“The fusion weight assigned to the structured branch is defined as <!-- formula-not-decoded --> where β controls the sensitivity to residual inconsistency.”
DualJudge does not outperform direct scoring in every configuration
MinorThe claim of consistent superiority conflicts with the reported qwen3.5-35b 1-10 results: Direct scores 87.38%, while both DualJudge variants score 87.19%.
“DualJudge consistently outperforms other approaches across model scales and dataset splits”
“| | 1-10 | 620 | 87.38% | 83.65% | 85.47% | 87.19% | 87.19% |”
Cites
Zadeh, L. A. (1965) · Information and Control · doi:10.1016/s0019-9958(65)90241-x · Cited by 67,855 · see citing works
Fuzzy Sets and Systems — Theory and Applications
Carter, H. (2011) · doi:10.1057/jors.1982.38 · Cited by 6,573 · cited 1×
“A common formulation represents comparisons using triangular fuzzy numbers (TFN) [4]: where l ij , m ij , and u ij denote lower, modal, and upper bounds of preference intensity, respectively.”
The analytic hierarchy process—what it is and how it is used
Saaty, R. W. (1987) · Mathematical Modelling · doi:10.1016/0270-0255(87)90473-8 · Cited by 5,380 · cited 1× · see citing works
“The AHP, originally proposed by Saaty [15,14], is a classical MCDM framework designed to decompose complex decision problems into hierarchical structures.”
Buckley, J., Feuring, T., & Hayashi, Y. (1999) · FUZZ-IEEE'99. 1999 IEEE International Fuzzy Systems. Conference Proceedings (Cat. No.99CH36315) · doi:10.1016/0165-0114(85)90090-9 · Cited by 3,529 · cited 2× · see citing works
“Weight estimation is then performed through fuzzy aggregation and subsequent defuzzification procedures[1].”
(2021) · Artificial Intelligence for Smarter Power Systems: Fuzzy logic and neural networks · doi:10.1049/pbpo161e_ch3 · Cited by 2,394 · cited 1×
“To address uncertainty in pairwise comparisons, researchers introduced fuzzy set theory [23] into AHP, resulting in the FAHP [1].”
Models, Methods, Concepts & Applications of the Analytic Hierarchy Process
Saaty, T. L., & Vargas, L. G. (2012) · International Series in Operations Research & Management Science · doi:10.1007/978-1-4614-3597-6 · Cited by 2,210 · cited 1× · see citing works
“Beyond binary filtering, we further interpret the CR as a continuous reliability signal [16,24].”
A note on the analysis of subjective judgment matrices
Crawford, G., & Williams, C. (1985) · doi:10.1016/0022-2496(85)90002-1 · Cited by 965 · cited 1×
“…logarithmic deviation from their transitive estimates are locally adjusted while preserving the reciprocity condition: The repair strategy is heuristic and performs a limited number of local corrections within a single repair phase rather than iteratively optimizing the entire matrix[3,24].”
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · cited 2× · see citing works
“[10].”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948 · see citing works
Consistency in the Analytic Hierarchy Process: a New Approach
Alonso, J., & Lamata, M. T. (2006) · Int. J. Uncertain. Fuzziness Knowl. Based Syst. · doi:10.1142/s0218488506004114 · Cited by 914 · cited 1× · see citing works
“15 [6] is adopted as the consistency threshold.”
A consistency improving method in the analytic hierarchy process
Zeshui, X., & Cuiping, W. (1999) · European Journal of Operational Research · doi:10.1016/s0377-2217(98)00109-x · Cited by 297 · cited 2× · see citing works
“…logarithmic deviation from their transitive estimates are locally adjusted while preserving the reciprocity condition: The repair strategy is heuristic and performs a limited number of local corrections within a single repair phase rather than iteratively optimizing the entire matrix[3,24].”
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.01574 · Cited by 131 · cited 1× · see citing works
“JudgeBench aggregates questions from LiveBench [20] and MMLU-Pro [19], covering 17 fine-grained categories across STEM, humanities, and professional domains.”
Large Language Model-Powered Automated Assessment: A Systematic Review
Emirtekin, E. (2025) · Applied Sciences · doi:10.3390/app15105683 · Cited by 87 · cited 2× · see citing works
“The rapid proliferation of LLMs has necessitated scalable evaluation methodologies, leading to the widespread adoption of the LLM-as-a-Judge paradigm [5].”
A Closer Look into Using Large Language Models for Automatic Evaluation
Chiang, C.-H., & Lee, H.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.599 · Cited by 59 · cited 2× · see citing works
“Prior studies show that LLM evaluators are sensitive to prompting formats and reasoning styles, resulting in unstable performance across task types [2].”
LiveBench: A Challenging, Contamination-Limited LLM Benchmark
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.19314 · Cited by 20 · cited 1× · see citing works
“JudgeBench aggregates questions from LiveBench [20] and MMLU-Pro [19], covering 17 fine-grained categories across STEM, humanities, and professional domains.”
AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses
Lu, X., Li, J.-Y., Takeuchi, K., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2410.01246 · Cited by 18 · cited 3×
“Recent work has explored integrating AHP with LLMs for specific evaluation and decision-support scenarios [11,22,21,8], suggesting that hierarchical reasoning can improve interpretability and alignment with human assessment.”
AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses
Lu, X., Li, J., Takeuchi, K., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.101 · Cited by 10 · see citing works
LLM-Assisted AHP for Explainable Cyber Range Evaluation
Kampourakis, V., Kavallieratos, G., Spathoulas, G., Gkioulos, V., & Katsikas, S. K. (2025) · EnCyCriS@ICSE · doi:10.1145/3786160.3788467 · Cited by 7 · cited 2×
“Recent work has explored integrating AHP with LLMs for specific evaluation and decision-support scenarios [11,22,21,8], suggesting that hierarchical reasoning can improve interpretability and alignment with human assessment.”
JudgeBench: A Benchmark for Evaluating LLM-based Judges
Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y., Cuadron, A., Wang, C., Popa, R. A., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2410.12784 · Cited by 4 · cited 1× · see citing works
“In this work, we conducted a comprehensive empirical study on JudgeBench [17], covering 17 categories spanning mathematical reasoning, programming, scientific knowledge, and soft-knowledge tasks.”
Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale
Li, W., Zhao, M., Dong, W., Cai, J., Wei, Y., Pocress, M., Li, Y., Yuan, W., Wang, X., Hou, R., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.03444 · Cited by 4 · cited 2×
“Two scoring granularities are considered (1–10 and 1–5 scales [9]), allowing us to analyze the effect of resolution on structured evaluation stability.”
Doc2AHP: Inferring Structured Multi-Criteria Decision Models via Semantic Trees with LLMs
Wu, H.-J., Zhou, S., Zhang, H.-X., & Chen, W. (2026) · arXiv.org · doi:10.48550/arxiv.2601.16479 · Cited by 2 · cited 2×
“Recent work has explored integrating AHP with LLMs for specific evaluation and decision-support scenarios [11,22,21,8], suggesting that hierarchical reasoning can improve interpretability and alignment with human assessment.”
Xie, J., Li, Y., Yin, X., & Wan, X. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34746 · Cited by 2 · cited 2× · see citing works
“Recent work has explored integrating AHP with LLMs for specific evaluation and decision-support scenarios [11,22,21,8], suggesting that hierarchical reasoning can improve interpretability and alignment with human assessment.”
Flagged first, then by citations
2 of the 24 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- [12]12. OpenAI: gpt-oss-120b & gpt-oss-20b model card (2025), https://arxiv.org/abs/ 2508.10925
- [15]15. Saaty, T.: The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. Advanced book program, McGraw-Hill International Book Company (1980), https://books.google.ru/books?id=Xxi7AAAAIAAJ
- [17]17. Tan, S., Zhuang, S., Montgomery, K., Tang, W.Y., Cuadron, A., Wang, C., Popa, R., Stoica, I.: Judgebench: A benchmark for evaluating LLM-based judges. In: The Thirteenth International Conference on Learning Representations (2025), https: //openreview.net/forum?id=G0dksFayVq
- [4]4. [https://www.sciencedirect.com/science/article/pii/S037722179800109X](https://www.sciencedirect.com/science/article/pii/S037722179800109X)
- Authors
- He, Y., Smirnov, I., Fedrushkov, D., Kovalchuk, S., & Revin, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2604.03742
- OpenAlex
- W7151265763
- Semantic Scholar
- 62aca303753e1431e0d38720d84c0f8e98fba9f4
- Cited by
- 0
- References
- 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Multi-Criteria Decision Making
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- LLM evaluation, fuzzy analytic hierarchy process, epistemic uncertainty, triangular fuzzy numbers, dual-process theory, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
