A Survey on LLM-as-a-Judge

May combine two claims.

Summarize by Multiple Rounds with taking mean score or with taking best score did not improve evaluation performance and even had some adverse effects.

“However, Summarize by Multiple Rounds with taking mean score ( w/ mean@5) or with taking best score ( w/ best-of-5) did not improve the evaluation performance and even had some adverse effects.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In sentiment analysis, numerous biases influencing LLM-based judgments have been identified, prompting the creation of automated frameworks to systematically quantify these biases.

“In sentiment analysis, numerous biases influencing LLM-based judgments have been identified, prompting the creation of automated frameworks to systematically quantify these biases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Beyond the technical sphere, LLMs are increasingly applied in critical domains such as finance, law, and scientific discovery (Ai4Sci).

“Beyond the technical sphere, LLMs are increasingly applied in critical domains, as shown in Figure 14, such as finance, law, and scientific discovery (Ai4Sci) [210], where domain expertise and rigorous evaluation are indispensable.”

• cites [210]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs have been successfully employed as evaluators in several NLP tasks, including sentiment analysis, machine translation, and text summarization.

“LLMs have been successfully employed as evaluators in several NLP tasks, including sentiment analysis, machine translation, and text summarization.”

May combine two claims.

Apart from Concreteness Bias and Content Continuation Bias, the performance of LLMs, except GPT-4-turbo, was generally poor, particularly in Length Bias.

“we observed that, apart from Concreteness Bias and Content Continuation Bias, the performance of LLMs, except GPT-4-turbo, was generally poor, particularly in the Length Bias.”

• AssumptionRests on an assumption: Human annotations in the benchmarks provide a valid ground truth for evaluating LLM-as-a-judge.

May combine two claims.

GPT-4 outperformed other LLMs with a large margin across all meta-evaluation dimensions and showed fewer biases.

“we found GPT-4 outperformed other LLMs with a large margin across all meta-evaluation dimensions and showed fewer biases.”

• AssumptionRests on an assumption: Human annotations in the benchmarks provide a valid ground truth for evaluating LLM-as-a-judge.

May combine two claims.

LLM-as-a-judge has been used to evaluate outputs from single models and to compare multiple models in competitive settings.

“This method has been used to evaluate outputs from single models and to compare multiple models in competitive settings.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In machine learning, LLM-as-a-Judge is used to evaluate NLP tasks, assess social intelligence, and support multi-modal evaluation.

“In machine learning, they are used to evaluate NLP tasks, assess social intelligence, and support multi-modal evaluation.”

May combine two claims.

o1-mini and deepseek-R1, while trailing slightly behind GPT-4-turbo and gemini-2.0-thinking, outperform non-GPT models like Mixtral-8×7B-Instruct-v0.1.

“Similarly, o1-mini and deepseek-R1, while trailing slightly behind GPT-4-turbo and gemini-2.0-thinking, outperform non-GPT models like Mixtral-8 × 7B-Instruct-v0.1”

May combine two claims.

In set 1, poor performance of GPT-3.5-turbo and LLaMA3-8B-Instruct in Length Bias negatively impacted overall performance, whereas in set 2, performance in this dimension was better, aligned with Qwen2.5-7B-Instruct.

“In set 1 , the poor performance of GPT-3.5-turbo and LLaMA3-8B-Instruct in the Length Bias negatively impacted the overall performance, whereas in set 2 , the performance in this dimension was better, which was aligned with Qwen2.5-7B-Instruct.”

Probably not a substantial claim: may not pass S4 (based).

LLMs have emerged as powerful evaluative tools across a wide range of machine learning tasks.

“Large Language Models (LLMs) have emerged as powerful evaluative tools across a wide range of machine learning (ML) tasks, giving rise to the paradigm of 'LLM-as-a-Judge'.”

May combine two claims.

Existing work only covers a subset of evaluation dimensions and lacks a comprehensive framework that includes all relevant aspects.

“Despite these efforts, existing work only covers a subset of evaluation dimensions and lacks a comprehensive framework that includes all relevant aspects.”

• cites [110]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [185]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

GPT-4-turbo excels in resolving ambiguous cases, with an accuracy of 29.67% in the human=TIE scenario, outperforming all other models.

“It also excels in resolving ambiguous cases, with an accuracy of 29.67% in the human=TIE scenario, outperforming all other models.”

• AssumptionRests on an assumption: Human judgments provide a valid ground truth for evaluating LLM judges' alignment.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The role of LLM-as-a-judge in retrieval encompasses both traditional document ranking and dynamic Retrieval-Augmented Generation (RAG) approaches.

“The role of LLM-as-a-judge in retrieval encompasses both traditional document ranking and dynamic Retrieval-Augmented Generation (RAG) approaches.”

May combine two claims.

GPT-4-turbo remains the benchmark for alignment, achieving the highest accuracy rates of 68.47% and 69.44% in the human=model1 and human=model2 scenarios, respectively.

“GPT-4-turbo remains the benchmark for alignment, achieving the highest accuracy rates of 68.47% and 69.44% in the human=model1 and human=model2 scenarios, respectively.”

• AssumptionRests on an assumption: Human judgments provide a valid ground truth for evaluating LLM judges' alignment.

May combine two claims.

Providing with Explanation generally has a negative impact on evaluation performance and bias mitigation.

“However, in terms of evaluation performance and bias mitigation, it generally has a negative impact.”

May combine two claims.

Among reasoning-enhanced models, gemini-2.0-thinking shows strong performance in the human=model2 scenario, achieving an accuracy of 78.27%, surpassing GPT-4-turbo.

“Among reasoning-enhanced models, gemini-2.0-thinking shows strong performance in the human=model2 scenario, achieving an accuracy of 78.27%-surpassing GPT-4turbo-highlighting its capability to identify models that align more closely with human judgment.”

• AssumptionRests on an assumption: Human judgments provide a valid ground truth for evaluating LLM judges' alignment.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

When evaluating models, inherent biases such as position bias and self-enhancement can significantly impact fairness, requiring careful mitigation strategies.

“When evaluating models, inherent biases such as position bias and self-enhancement can significantly impact fairness, requiring careful mitigation strategies.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Task-agnostic biases are more fundamental issues inherent to LLMs and are therefore more difficult to address.

“task-agnostic biases are more fundamental issues inherent to the LLMs themselves and are therefore more difficult to address.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Directly optimizing LLMs' outputs to obtain final results is faster and lower-cost than integrating multiple evaluation results.

“compared to integrating multiple evaluation results, directly optimizing the LLMs' outputs to obtain the final results is faster and lower-cost”

Probably not a substantial claim: may not pass S4 (based).

The basic definition of LLM-as-a-Judge alone does not guarantee the reliability of evaluations.

“However, the basic definition alone does not guarantee the reliability of evaluations.”

May combine two claims.

There are two ways to apply LLM-as-a-Judge for an agent: evaluate the entire process of the intelligent agent, or evaluate it at a specific stage in the agent framework process.

“There are two ways to apply LLM-as-a-Judge for an agent. One is to evaluate the entire process of the intelligent agent [226], and the other is to evaluate it at a specific stage in the agent framework process [43, 128].”

• cites [226]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [43]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [128]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Evaluating complex reasoning tasks poses a particular risk; without robust verification of logical consistency, LLMs may produce seemingly confident but fundamentally flawed evaluations, undermining trust in sophisticated analytical processes.

“Moreover, evaluating complex reasoning tasks poses a particular risk; without robust verification of logical consistency, LLMs may produce seemingly confident but fundamentally flawed evaluations, undermining trust in sophisticated analytical processes.”

May combine two claims.

Internal biases of LLMs may propagate into the LLM-as-a-Judge setting, potentially resulting in unfair evaluation outcomes and influencing downstream LLMs.

“Such internal biases may propagate into the LLM-as-a-Judge setting, potentially resulting in unfair evaluation outcomes and, in turn, influencing the development of downstream LLMs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In data evaluation contexts, the extensive use of LLM-as-a-Judge for pseudo-labeling risks amplifying existing model biases, potentially leading to the generation of unverified and flawed training data.

“In data evaluation contexts, the extensive use of LLM-as-a-Judge for pseudo-labeling risks amplifying existing model biases, potentially leading to the generation of unverified and flawed training data.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Careful prompt engineering is essential to mitigate issues like output variability and inter-rater reliability, ensuring the LLM consistently interprets and responds to evaluation criteria as intended.

“Careful prompt engineering is essential to mitigate issues like output variability and inter-rater reliability, ensuring the LLM consistently interprets and responds to evaluation criteria as intended.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Initial conceptualization in the thinking phase is vital for preempting potential ambiguities and biases that could compromise the fairness and accuracy of subsequent LLM judgments.

“This initial conceptualization is vital for preempting potential ambiguities and biases that could compromise the fairness and accuracy of subsequent LLM judgments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The process of quick practice for LLM-as-a-Judge involves four main stages.

“As shown in Figure 8, The process of quick practice for LLM-as-a-Judge involves four main stages.”

May combine two claims.

Combining multiple rounds of evaluations or multiple LLM evaluators can reduce potential biases of a single LLM evaluator.

“and reduce the potential biases of a single LLM evaluator.”

Classified as probably not a substantial claim.

Objective assessment methods, such as automatic metrics, offer strong scalability and consistency.

“On the other hand, objective assessment methods, such as automatic metrics, offer strong scalability and consistency.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Reliance on smaller open-source models may introduce inconsistencies and misalignment with human judgments.

“Finally, reliance on smaller open-source models, while cost-effective, may introduce inconsistencies and misalignment with human judgments.”

May combine two claims.

Expert-driven assessments are costly, difficult to scale, and susceptible to inconsistency.

“However, these approaches are costly, difficult to scale, and susceptible to inconsistency.”

May combine two claims.

Current prompt and fine-tuning dataset designs often result in evaluation LLMs with poor generalization, making them difficult to compare with strong LLMs like GPT-4.

“The current prompt and fine-tuning dataset designs often result in evaluation LLMs with poor generalization, making them difficult to compare with strong LLMs like GPT-4.”

Classified as probably not a substantial claim.

Self-Enhancement Bias is the phenomenon that LLM evaluators may prefer responses generated by themselves.

“Self-Enhancement Bias describes the phenomenon that LLM evaluators may prefer responses generated by themselves [185, 213].”

• cites [185]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [213]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Model selection focuses on choosing a large-scale model with strong reasoning and instruction-following abilities to ensure reliable evaluations.

“The third stage, model selection (Section 2.2), focuses on choosing a large-scale model with strong reasoning and instruction-following abilities to ensure reliable evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Fine-tuned evaluators often exhibit overfitting and limited generalization beyond their training data.

“Fine-tuned evaluators, while specialized, often exhibit overfitting and limited generalization beyond their training data.”

May combine two claims.

Benchmarks such as MLLM-as-a-Judge assess MLLMs using tasks like scoring, pair comparison, and batch ranking, revealing limitations in alignment with human preferences.

“Benchmarks such as MLLM-as-a-Judge [17] assess these models using tasks like scoring, pair comparison, and batch ranking, revealing limitations in alignment with human preferences.”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Human judgment is the best way to evaluate LLMs, but collecting human annotations is costly, time-consuming, and laborious.

“It is universally known that the best way to evaluate LLMs is human judgment, but collecting human annotations can be costly, time-consuming, and laborious [108, 213].”

• cites [108]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [213]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

With appropriate prompt design, the quality of evaluation and agreement to human judgment can be promising.

“With appropriate prompt design, the quality of evaluation and agreement to human judgment can be promising [32, 155, 207, 213].”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [155]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [207]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [213]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Using LLM as the brain of an agent, an agentic system could evaluate like a human, reducing the need for human involvement and eliminating the trade-off between thoroughness and effort.

“Using LLM as the brain of agent, an agentic system [226] could evaluate like a human, it would reduce the need for human involvement and eliminate the trade-off between thoroughness and effort.”

• cites [226]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The most efficient and generally effective approach to prompt design involves specifying scoring dimensions, emphasizing relative comparisons for improved assessments, and creating effective examples to guide the LLM.

“The most efficient and generally effective approach involves specifying scoring dimensions, emphasizing relative comparisons for improved assessments, and creating effective examples to guide the LLM.”

Probably not a substantial claim: may not pass S4 (based).

Fine-tuned evaluators can inherit subtle biases from their training datasets.

“Moreover, these models can inherit subtle biases from their training datasets, necessitating careful meta-evaluation to ensure fairness.”

May combine two claims.

Fine-tuned evaluator models often demonstrate superior performance on self-designed test sets but have identified limitations in their evaluation capabilities.

“While these fine-tuned models often demonstrate superior performance on self-designed test sets, they are identified several limitations in their evaluation capabilities,which detailed in Section 4.2.”

Probably not a substantial claim: may not pass S4 (based).

Current meta-evaluation primarily focuses on LLM-as-a-judge for models.

“Current meta-evaluation primarily focuses on LLM-as-a-judge for models”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The success of using LLM-as-a-Judge heavily depends on implementation details, including task complexity, prompt design, model selected, and post-processing method.

“The success of using LLMas-a-Judge also heavily depends on the implementation details, including the task complexity, the prompt design, the model selected, and the post-processing method.”

Classified as probably not a substantial claim.

Large language models inherently exhibit diverse forms of bias across different tasks.

“Prior studies have emphasized that large language models inherently exhibit diverse forms of bias across different tasks [25, 34, 138].”

• cites [25]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [138]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The inherent biases and generalization limitations of different models, as well as their black-box nature, underscore the importance of selecting a robust and well-understood backbone to minimize evaluation inconsistencies and unverified judgments.

“The inherent biases and generalization limitations of different models, as well as their black-box nature, underscore the importance of selecting a robust and well-understood backbone to minimize evaluation inconsistencies and unverified judgments.”

May combine two claims.

Widely used subjective methods like expert-driven assessments integrate holistic reasoning and fine-grained contextual understanding, making them the gold standard in comprehensiveness.

“On the one hand, widely used subjective methods like expert-driven assessments [38, 126] integrate holistic reasoning and fine-grained contextual understanding, making them the gold standard in comprehensiveness.”

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [126]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Mitigating task-agnostic biases likely depends on advancements in foundation models.

“Mitigating such biases likely depends on advancements in the foundation models.”

Probably not a substantial claim: may not pass S4 (based).

The absence of a systematic review highlights the lack of formal definitions, fragmented understanding, and inconsistent usage practices in relevant studies.

“The first challenge lies in the absence of a systematic review, which highlights the lack of formal definitions, fragmented understanding, and inconsistent usage practices in the relevant studies.”

May combine two claims.

In-context learning for LLM-as-a-Judge can surface issues that impact reliability, including variability of LLM outputs due to minor prompt changes, inconsistent inter-rater reliability in score-based assessments, ambiguity in response interpretation for Yes/No or multiple-choice questions, and biases such as favoring responses based on position or length.

“When leveraging in-context learning, certain issues can surface, potentially impacting the reliability of evaluations. These include the variability of LLM outputs due to minor prompt changes, which can lead to unstable results.”

• Also stated in: § Reliability Concerns of In-Context Learning · p. 10 · plain

  “Furthermore, score-based assessments often exhibit inconsistent inter-rater reliability, influenced by the inherent randomness of LLM generation and its sensitivity to phrasing.”

• Also stated in: § Reliability Concerns of In-Context Learning · p. 10 · hedged

  “Similarly, evaluation formats like Yes/No or multiple-choice questions are prone to ambiguity in response interpretation. Lastly, LLM-as-a-Judge evaluations may inadvertently reflect biases, such as favoring responses based on their position or length.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Most of the data that need to be evaluated by LLM-as-a-Judge is generated by models or is large-scale crawled data.

“Most of the data need to be evaluated by LLM-as-a-Judge is generated by models, or large-scale crawled data.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Solely relying on LLM-as-a-Judge for data annotation poses challenges, particularly as the value of annotated data diminishes with the rapid improvement of model performance.

“However, solely relying on LLM-as-a-Judge for data annotation poses challenges, particularly as the value of annotated data diminishes with the rapid improvement of model performance.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Self-verification is suitable for all LLMs and requires no complex computing and processing.

“Self-verification is suitable for all LLMs and requires no complex computing and processing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Large Language Models (LLMs) have achieved remarkable success across numerous domains.

“Recently, Large Language Models (LLMs) have achieved remarkable success across numerous domains [178], ranging from technical fields [142, 191, 210] to the humanities [55, 100, 113, 217] and social sciences [45, 127, 164, 177].”

• cites [178]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Persistent issues in MLLM evaluation include biases such as position and verbosity biases and hallucinations, with even advanced models like GPT-4V displaying challenges.

“Persistent issues include biases (e.g., position, verbosity) and hallucinations, with even advanced models like GPT-4V displaying challenges.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The emergence of LLMs presents an unprecedented opportunity to automate the complicated process of data annotation by LLM-as-a-Judge.

“The emergence of LLMs presents an unprecedented opportunity to automate the complicated process of data annotation by LLM-as-a-Judge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Noisy or adversarially perturbed inputs can cause the LLM to misjudge quality, diminishing robustness.

“Noisy or adversarially perturbed inputs can cause the LLM to misjudge quality, thus diminishing its robustness.”

May combine two claims.

Automatic metrics such as BLEU and ROUGE heavily rely on surface-level lexical overlaps and often fail to capture deeper nuances, resulting in poor performance in tasks like story generation or instructional texts.

“However, these metrics, which heavily rely on surface-level lexical overlaps, often fail to capture deeper nuances, resulting in poor performance in tasks like story generation or instructional texts [122].”

• cites [122]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The cost concern still exists when calling the APIs of proprietary models, especially when there is a frequent need for model validation on large-scale data.

“However, the cost concern still exists when calling the APIs of these proprietary models, especially when there is a frequent need for model validation on large-scale data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Best-of-N generation scenarios, where multiple reasoning outputs are produced, the judge determines the most accurate and coherent response.

“For example, in “Best-of-N” generation scenarios, where multiple reasoning outputs are produced, the judge determines the most accurate and coherent response.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A less robust LLM may produce inconsistent scores for identical prompts and inputs due to sampling variance or internal preference drift.

“a less robust LLM may produce inconsistent scores for identical prompts and inputs due to sampling variance or internal preference drift.”

• AssumptionRests on an assumption: Inherent randomness in LLMs' generation causes scores to not fully reflect the LLMs' complete view of the evaluation criteria.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reasoning often depends on judgments to ensure logical coherence, refine intermediate steps, and achieve clarity in its outcomes.

“While reasoning is inherently more demanding and multifaceted than judging, it often depends on judgments to ensure logical coherence, refine intermediate steps, and achieve clarity in its outcomes.”

May combine two claims.

Widespread adoption of LLM-as-a-Judge is hindered by two key challenges: the absence of a systematic review and reliability concerns.

“However, its widespread adoption is hindered by two key challenges.”

• Also stated in: § 1 INTRODUCTION · p. 2 · plain

  “The first challenge lies in the absence of a systematic review, which highlights the lack of formal definitions, fragmented understanding, and inconsistent usage practices in the relevant studies.”

• Also stated in: § 1 INTRODUCTION · p. 2 · plain

  “The second challenge concerns reliability [189], as merely employing LLM-as-aJudge does not ensure accurate evaluations aligned with established standards.”

• cites [189]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The accuracy of the GPT-4-based evaluator has been demonstrated to be high compared to professional human evaluators, showing superior consistency and stability in evaluations.

“The accuracy of the GPT-4-based evaluator has been demonstrated to be high compared to professional human evaluators, showing superior consistency and stability in evaluations.”

• cites [213]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Position Bias is the tendency of LLM evaluators to favor responses in certain positions within the prompt.

“Position Bias is the tendency of LLM evaluators to favor responses in certain positions within the prompt [126, 144, 155, 185].”

• cites [126]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [144]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [155]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [185]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Subtle changes in prompt wording or ordering of context can lead to different judgments for the same input, undermining reliability.

“subtle changes in the prompt wording or the ordering of the context C can lead to different judgments for the same input, a phenomenon that undermines reliability.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The effectiveness of directly optimizing LLMs' outputs still needs further validation.

“although the effectiveness still needs further validation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For agentic frameworks, the sequential application of LLM-as-a-Judge can lead to the accumulation of errors, where early misjudgments cascade into significant inaccuracies in final decisions.

“For agentic frameworks, the sequential application of LLM-as-a-Judge can lead to the accumulation of errors, where early misjudgments cascade into significant inaccuracies in final decisions.”

May combine two claims.

While pair comparison tasks align better with human judgment, scoring and batch ranking require significant improvements for reliable deployment.

“While pair comparison tasks align better with human judgment, scoring and batch ranking require significant improvements for reliable deployment.”

May combine two claims.

Constrained decoding can distort the model's learned distribution, potentially degrade output quality, require significant engineering implementation effort, and introduce computational overhead during inference.

“While this method guarantees syntactically valid outputs, it presents several challenges: it can distort the model's learned distribution and potentially degrade output quality, requires significant engineering implementation effort, and introduces computational overhead during inference.”

Classified as probably not a substantial claim.

Within the broader evaluation pipeline, several factors can compromise the integrity of LLM-based assessments.

“Within the broader evaluation pipeline, several factors can compromise the integrity of LLM-based assessments.”

Probably not a substantial claim: may not pass S4 (based).

General-purpose LLMs' black-box nature and version dependency can hinder the reproducibility of evaluation outputs.

“Concerns arise from the black-box nature and version dependency of general-purpose LLMs, which can hinder the reproducibility of evaluation outputs.”

May combine two claims.

The LLM-as-a-Judge paradigm is gaining traction across domains such as software engineering, education and behavioural science, where it enables scalable qualitative assessment.

“In summary, the LLM-as-a-Judge paradigm is gaining traction across domains such as software engineering, education and behavioural science, where it enables scalable qualitative assessment.”

May combine two claims.

Few-shot learning and prompt tuning improved scoring accuracy, and revisions effectively enhanced essay quality without compromising original meaning.

“Techniques such as few-shot learning and prompt tuning improved scoring accuracy, while revisions effectively enhanced essay quality without compromising original meaning [132].”

• cites [132]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Setting the hyper-parameter temperature to 0 reduces the impact of randomness in LLMs' output.

“To stabilize the evaluation results of LLMs, we set the hyper-parameter temperature to 0 to reduce the impact of randomness in LLMs' output.”

• AssumptionRests on an assumption: Inherent randomness in LLMs' generation causes scores to not fully reflect the LLMs' complete view of the evaluation criteria.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Within NLP, LLM-as-a-judge enables flexible, scalable, and human-aligned evaluation.

“Within NLP, especially for tasks such as text generation, reasoning and retrieval, LLM-as-aJudge enables flexible, scalable, and human-aligned evaluation.”

Probably not a substantial claim: may not pass S2 (checkable).

When multiple LLMs are adopted for joint evaluation, the differences between their evaluation performances must be carefully considered.

“This suggests that when multiple LLMs are adopted for joint evaluation, the differences between their evaluation performances must be carefully considered.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

WizardMath surpasses GPT-3.5 on GSM8K and MATH.

“WizardMath applies Reinforcement Learning from Evol-Instruct Feedback and surpasses GPT-3.5 on GSM8K and MATH [99].”

• cites [99]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Current medical evaluators often rely on exam-style or short-form outputs and struggle with real-world clinical complexity, and hallucination detection remains an open problem.

“Despite rapid progress, current medical evaluators often rely on exam-style or short-form outputs and struggle with real-world clinical complexity; hallucination detection remains an open problem [64].”

• cites [64]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Addressing other biases may require researchers to explore more effective evaluation strategies.

“addressing other biases may require researchers to explore more effective evaluation strategies.”

• AssumptionRests on an assumption: Human annotations in the benchmarks provide a valid ground truth for evaluating LLM-as-a-judge.

Probably not a substantial claim: may not pass S4 (based).

Self-enhancement bias occurs where an LLM is more likely to give a higher score to an answer it generated itself compared to an equivalent answer generated by a different model.

“This is often tied to self-enhancement bias , where an LLM is more likely to give a higher score to an answer it generated itself compared to an equivalent answer generated by a different model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MathVista reveals persistent weaknesses when diagrams are involved in textual-visual reasoning.

“For multimodal scenarios, Lu et al. created MathVista to evaluate textual-visual reasoning, revealing persistent weaknesses when diagrams are involved [97].”

• cites [97]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The reliability of LLM-as-a-Judge is a primary concern, as it directly impacts the consistency and fairness of evaluations.

“The reliability of LLM-as-a-Judge is a primary concern, as it directly impacts the consistency and fairness of evaluations.”

May combine two claims.

The LLM-as-a-Judge paradigm in scientific domains is progressing rapidly, supported by specialised evaluators and benchmarks.

“In summary, the LLM-as-a-Judge paradigm in scientific domains is progressing rapidly, supported by specialised evaluators and benchmarks.”

May combine two claims.

MedHELM introduced a holistic evaluation suite covering eleven clinical tasks and showed that even top commercial models still lag on medication-safety questions.

“Stanford HAI's MedHELM introduced a holistic evaluation suite covering eleven clinical tasks and showed that even top commercial models still lag on medication-safety questions [64].”

• cites [64]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In software engineering, a method using LLMs to evaluate bug report summarizations demonstrated high accuracy in assessing correctness and completeness, even surpassing human evaluators who experienced fatigue.

“In software engineering, a method was proposed for using LLMs to evaluate bug report summarizations, demonstrating high accuracy in assessing correctness and completeness, even surpassing human evaluators who experienced fatigue [69].”

• cites [69]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Taking majority voting results from repeated evaluations helps reduce the impact of randomness in LLMs, thereby addressing bias issues.

“It suggests that taking the majority voting results from repeated evaluations helps reduce the impact of randomness in LLMs, thereby addressing bias issues.”

May combine two claims.

An empirical strategy for pairwise comparison evaluation tasks is to select more powerful LLMs and to adopt two evaluation strategies: swapping the positions of the evaluation contents and taking the majority voting results from multiple rounds of evaluation, which can effectively mitigate biases.

“Based on the current experimental analysis, an empirical strategy for pairwise comparison evaluation tasks is to select more powerful LLMs and to adopt two evaluation strategies: one is swapping the positions of the evaluation contents, the other is taking the majority voting results from multiple rounds of evaluation , which can effectively mitigate biases.”

May combine two claims.

In behavioral sciences, the LLM-as-a-Judge framework for assessing user preferences based on personas revealed limitations in reliability and consistency due to oversimplified personas, but improved significantly through verbal uncertainty estimation, achieving high agreement with human evaluations for high-certainty cases.

“In behavioral sciences, the LLM-as-a-Judge framework was evaluated for assessing user preferences based on personas, revealing limitations in reliability and consistency due to oversimplified personas, but improved significantly through verbal uncertainty estimation, achieving high agreement with human evaluations for high-certainty cases [30].”

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reliability requirements for LLM-as-a-Judge in science centre on hallucination detection and evidence-based rigour to maintain scientific validity.

“Reliability requirements therefore centre on hallucination detection and evidence-based rigour to maintain scientific validity, echoing the survey's discussion of output consistency and adversarial robustness.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Nature Medicine's 2024 review highlighted gaps in automatic metrics and advocated clinician-in-the-loop pipelines for summarization tasks.

“Nature Medicine's 2024 review highlighted gaps in automatic metrics and advocated clinicianin-the-loop pipelines for summarization tasks [151].”

• cites [151]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Reasoning-enhanced LLMs provide meaningful advancements over baseline models but fall short of delivering consistent advantages in alignment-related tasks.

“These results indicate that reasoning-enhanced LLMs provide meaningful advancements over baseline models but fall short of delivering consistent advantages in alignment-related tasks , suggesting room for further optimization in this area.”

May combine two claims.

Qwen2.5-7B-Instruct can be a promising choice as an open-source LLM-as-a-Judge, with potential to serve as a robust base model for specialized evaluators in specific scenarios.

“indicating that it can be a promising choice as an open-source LLM-as-a-Judge, with the potential to serve as a robust base model for specialized evaluators in specific scenarios.”

• AssumptionRests on an assumption: Human annotations in the benchmarks provide a valid ground truth for evaluating LLM-as-a-judge.

May combine two claims.

LLMEval 2 is the largest and most diverse evaluation benchmark for LLM-as-a-judge to date, with 2,553 samples compiled from multiple data sources with human-annotated preferences.

“LLMEval 2 is the largest and most diverse evaluation benchmark for LLM-as-a-judge to date, with 2,553 samples compiled from multiple data sources with human-annotated preferences.”

• cites [207]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Length Bias is the tendency to favor responses of a particular length, such as a preference for more verbose responses.

“Length Bias refers to the tendency to favor responses of a particular length, such as a preference for more verbose responses, which is also known as verbosity bias [49, 110, 185, 213].”

• cites [49]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [110]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [185]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [213]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Math-centric judges are still brittle on open-ended or multimodal problems, and exhaustive step verification is computationally costly.

“Math-centric judges are still brittle on open-ended or multimodal problems, and exhaustive step verification is computationally costly.”

Probably not a substantial claim: may not pass S2 (checkable).

Domain-specific adaptation will be essential to satisfy varied demands and to support reliable, efficient applications of LLM-as-a-Judge.

“Therefore, domain-specific adaptation will be essential to satisfy these varied demands and to support reliable, efficient applications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

EVALBIASBENCH consists of 80 samples, each containing a question, a pair of candidate responses, and a label indicating the correct response without bias influence.

“EVALBIASBENCH consists of 80 samples, each containing a question, a pair of candidate responses, and a label indicating the correct response without bias influence.”

• cites [110]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Integrating large language models with structured knowledge bases is likely to meet high-stakes needs for precise, verifiable judgements.

“Integrating large language models with structured knowledge bases is likely to meet these high-stakes needs for precise, verifiable judgements.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The tendency for GPT-4 to favor its own responses over those of GPT-3.5 can be interpreted as either self-enhancement bias or a proper tendency towards higher quality text.

“Additionally, the tendency for GPT-4 to favor its own responses over those of GPT-3.5 can be interpreted as either self-enhancement bias or a proper tendency towards higher quality text.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Reliability requirements for LLM-as-a-Judge differ by field and often centre on the consistent interpretation of nuanced rules or subjective preferences so as to minimise bias.

“Reliability requirements differ by field. They often centre on the consistent interpretation of nuanced rules or subjective preferences so as to minimise bias, echoing the survey's broader themes of prompt optimisation and self-consistency.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In the case of position bias, lengthening the response could potentially alter the style, fluency, and coherence, or even introduce new biases such as self-enhancement bias.

“For instance, in the case of position bias, lengthening the response could potentially alter the style, fluency, and coherence, or even introduce new biases such as self-enhancement bias.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Concreteness Bias reflects that LLM evaluators favor responses with specific details, including citation of authoritative sources, numerical values, and complex terminologies.

“Concreteness Bias reflects that LLM evaluators favor responses with specific details, including citation of authoritative sources, numerical values, and complex terminologies, which is called authority bias [110] or citation bias [19, 185].”

• cites [110]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [19]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [185]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

The most direct metric to reflect the quality of automatic evaluation is alignment with human evaluation.

“The most direct metric to reflect the quality of automatic evaluation is the alignment with human evaluation.”

• AssumptionRests on an assumption: Human annotations in the benchmarks provide a valid ground truth for evaluating LLM-as-a-judge.

Probably not a substantial claim: may not pass S3 (load-bearing).

Math-Shepherd boosts Mistral-7B accuracy to 84.1% on GSM8K.

“Math-Shepherd introduces an automatic process-reward model that verifies each reasoning step without human annotations and then reinforces the solver, boosting Mistral-7B accuracy to 84.1 % on GSM8K [156].”

• cites [156]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

DART-Math achieves state-of-the-art results across six math benchmarks.

“Tong et al. proposed DART-Math, a difficulty-aware rejection-tuning pipeline that focuses training on hard problems and achieves state-of-the-art results across six math benchmarks [146].”

• cites [146]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs as evaluators demonstrate significant advantages in qualitative assessments that are difficult to quantify, such as evaluating service quality, analyzing user experience feedback, and assessing creative content.

“Moreover, LLMs as evaluators demonstrate significant advantages in qualitative assessments that are difficult to quantify, such as evaluating service quality, analyzing user experience feedback, and assessing creative content like art or literature reviews.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

LLMs' capability to understand and generate nuanced language makes them well-suited for subjective evaluation tasks traditionally requiring human judgment.

“LLMs' capability to understand and generate nuanced language makes them well-suited for subjective evaluation tasks traditionally requiring human judgment.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Insufficient robustness allows trivial manipulations to deceive evaluators and undermine evaluation of text quality.

“Robustness is crucial because insufficient robustness allows trivial manipulations to deceive the evaluators and to undermine the evaluation of text quality.”

May combine two claims.

In content moderation, an LLM-based approach to identify rule violations on platforms like Reddit achieved high true-negative rates but encountered challenges with complex rule interpretation, emphasizing the necessity of human oversight for nuanced cases.

“In content moderation, an LLM-based approach was developed to identify rule violations on platforms like Reddit, achieving high true-negative rates but encountering challenges with complex rule interpretation, emphasizing the necessity of human oversight for nuanced cases [65].”

• cites [65]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

LLM-as-a-judge offers a nuanced, adaptable, and customized assessment for text generation tasks.

“Unlike traditional metrics-based evaluations, LLM-as-a-judge offers a nuanced, adaptable, and customized assessment.”

Classified as probably not a substantial claim.

When evaluating a specific dimension, especially a particular type of bias, it is often challenging to isolate the bias of interest from other confounding factors such as additional biases or general quality-related characteristics.

“When evaluating a specific dimension, especially a particular type of bias, it is often challenging to isolate the bias of interest from other confounding factors such as additional biases or general quality-related characteristics.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The black-box nature and version dependency of commercial models like GPT-4 hinder reproducibility.

“The black-box nature and version dependency of commercial models like GPT-4 hinder reproducibility.”

• AssumptionRests on an assumption: Proprietary LLMs are black boxes whose internal reasoning cannot be inspected.

May combine two claims. The quote only partly states this claim.

Current research on LLM-as-a-Judge in science can be grouped into three major directions: specialized evaluators for clinical reasoning, step-wise reward modeling in mathematics, and comprehensive benchmarking frameworks.

“Current research on LLM-as-a-Judge in science can be grouped into three major directions.”

Classified as probably not a substantial claim.

Simultaneously generating evaluations and explanations may compromise the quality of evaluations.

“While explanations help humans understand LLM decision processes, their simultaneous execution may compromise the quality of evaluations.”

May combine two claims.

The deployment of LLM-as-a-Judge is constrained by context-specific inconsistencies and by the difficulty of replacing fatigued human reviewers.

“Nevertheless, its deployment is constrained by context-specific inconsistencies and by the difficulty of replacing fatigued human reviewers.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

LLM-as-a-Judge can be susceptible to adversarial prompt manipulation and contextual framing biases, potentially causing unintended or unreliable evaluations.

“LLM-as-a-Judge can be susceptible to adversarial prompt manipulation and contextual framing biases, potentially causing unintended or unreliable evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Current research efforts on LLM-as-a-Judge for LLM optimization are still in their infancy.

“Despite these encouraging developments, current research efforts are still in their infancy.”

May combine two claims.

This brittleness undermines the reliability and automation of the entire evaluation system.

“This brittleness undermines the reliability and automation of the entire evaluation system.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This self-generated feedback loop helps the model improve without needing external verification.

“This self-generated feedback loop helps the model improve without needing external verification.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The development of more comprehensive and rigorous benchmarks is critical for advancing the reliability and applicability of LLM-as-a-Judge systems.

“The development of more comprehensive and rigorous benchmarks is critical for advancing the reliability and applicability of LLM-as-a-Judge systems.”

Classified as probably not a substantial claim. May combine two claims.

The evaluation process of current LLM-as-a-Judge systems is often an opaque black box, which significantly limits user trust and constrains application in high-stakes domains such as medicine, law, and education.

“While current LLM-as-a-Judge systems can provide seemingly reasonable scores or conclusions for many tasks, their evaluation process is often an opaque black box. This lack of transparency significantly limits user trust and constrains the application in high-stakes domains such as medicine, law, and education.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The gap between the model's superficial fluency and its true reasoning depth undermines trust in its ability to handle sophisticated analytical processes and highlights the need for more robust reasoning backbones.

“This gap between the model's superficial fluency and its true reasoning depth undermines trust in its ability to handle sophisticated analytical processes and highlights the need for more robust reasoning backbones.”

Classified as probably not a substantial claim.

When evaluating a response that requires a deep understanding of logical consistency, a model might produce a seemingly confident but fundamentally flawed evaluation.

“When evaluating a response that requires a deep understanding of logical consistency, a model might produce a seemingly confident but fundamentally flawed evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

If a model consistently favors a certain style, format, or tone, it could stifle creative and diverse outputs, leading to a homogenization of content.

“If a model consistently favors a certain style, format, or tone, it could stifle creative and diverse outputs, leading to a homogenization of content.”

Classified as probably not a substantial claim.

LLM-as-a-Judge is particularly well-suited for applications in education, peer review, and decision-making systems due to its ability to process diverse content formats and integrate domain-specific knowledge.

“Its ability to process diverse content formats and integrate domain-specific knowledge makes it particularly well-suited for applications in education, peer review, and decision-making systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent models like GPT-4o, Gemini, and LLaVA integrate text, audio, and image processing capabilities within a single framework.

“Instead of employing distinct architectures for processing text, audio, and images, recent models like GPT-4o [107], Gemini [40], and LLaVA [90] integrate these capabilities within a single framework.”

• cites [107]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [90]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Extensive empirical evidence suggests that the performance of LLM-as-a-Judge is not static, with its judgments potentially drifting over time due to model updates or contextual changes.

“Furthermore, extensive empirical evidence suggests that the performance of LLM-as-a-Judge is not static, with its judgments potentially drifting over time due to model updates or contextual changes.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Judging the actions and behaviors of agents in physical or simulated environments presents a novel and complex challenge.

“This presents a novel and complex challenge: judging the actions and behaviors of agents in physical or simulated environments.”

Classified as probably not a substantial claim.

The reliability requirements for evaluation differ significantly across domains.

“The reliability requirements for evaluation differ significantly across domains.”

Probably not a substantial claim: may not pass S4 (based).

RLHF techniques do not eliminate all sources of subjectivity.

“Although techniques RLHF have improved alignment with human judgment, they do not eliminate all sources of subjectivity.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

These attacks are particularly insidious because the manipulated input appears harmless to a human but can cause a significant deviation in the LLM's judgment.

“These attacks are particularly insidious because the manipulated input appears harmless to a human but can cause a significant deviation in the LLM's judgment.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

In a legal context, an LLM judge might accurately summarize a case and suggest a verdict, but it cannot explicitly show which case precedents it referenced or which legal statutes it prioritized in its reasoning process.

“For instance, in a legal context, an LLM judge might accurately summarize a case and suggest a verdict, but it cannot explicitly show which case precedents it referenced or which legal statutes it prioritized in its reasoning process.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Attacks on LLM-as-a-Judge aim to exploit a judge model's biases, inconsistencies, or loopholes to manipulate its decision-making process.

“These attacks aim to exploit a judge model's biases, inconsistencies, or loopholes to manipulate its decision-making process.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

LLM-as-a-Judge is poised to become an integral component of next-generation evaluation systems, augmenting human expertise rather than replacing it.

“Ultimately, LLM-as-a-Judge is poised to become an integral component of next-generation evaluation systems, augmenting human expertise rather than replacing it.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A central component of o1's advancement is the use of LLM-as-a-Judge modules that evaluate reasoning paths at each stage, offering feedback that improves future steps.

“A central component of o1's advancement is the use of LLMas-a-Judge modules that evaluate reasoning paths at each stage, offering feedback that improves future steps.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Advanced LLMs such as GPT-4, Gemini, and LLaMA-2 offer a promising opportunity to revolutionize data annotation.

“Advanced LLMs such as GPT-4 [107], Gemini [40], and LLaMA-2 [149] offer a promising opportunity to revolutionize data annotation.”

• cites [107]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [149]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

In judging a complex scientific paper or legal argument, the model may give a high score for stylistic fluency while failing to identify a subtle but fundamental flaw in the logical chain or causal argument.

“For instance, in judging a complex scientific paper or legal argument, the model may give a high score for stylistic fluency while failing to identify a subtle but fundamental flaw in the logical chain or causal argument.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Current models struggle with evaluating multi-modal inputs, reasoning over structured data, and adapting to domain-specific evaluation standards.

“Finally, generalization across domains and modalities remains a significant hurdle, as current models struggle with evaluating multi-modal inputs, reasoning over structured data, and adapting to domain-specific evaluation standards.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

This judge would act as a crucial feedback mechanism, providing high-dimensional, natural language feedback to guide the embodied agent's learning process.

“This judge would act as a crucial feedback mechanism, providing high-dimensional, natural language feedback to guide the embodied agent's learning process.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In optimization pipelines, an LLM judge can provide nuanced, dynamic feedback to guide model fine-tuning and enhance reasoning chains, serving as a scalable alternative to traditional, fixed metrics.

“In multi-agent systems, an LLM judge can evaluate the quality of inter-agent communication and collaboration in real time, and in optimization pipelines, it can provide nuanced, dynamic feedback to guide model fine-tuning and enhance reasoning chains, thus serving as a scalable alternative to traditional, fixed metrics.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Currently, we have not been able to experimentally rely solely on LLM for a reliable evaluation of various scenarios of data; most of the time, we still rely on human annotation to ensure professionalism and reliability.

“Currently, we have not been able to experimentally rely solely on LLM for a reliable evaluation of various scenarios of data; most of the time, we still rely on human annotation to ensure professionalism and reliability.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Whether in the field of scientific research or industry, we are all still suffering from insufficient target data and domain-specific data, or situations where the data quality is not high enough.

“Whether in the field of scientific research or industry, we are all still suffering from insufficient target data and domain-specific data, or situations where the data quality is not high enough.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Unlike evaluating a text response, judging an embodied agent requires assessing a sequence of actions, their spatiotemporal relationships, and their alignment with a high-level goal, which necessitates a deep understanding of physics and cause and effect.

“Unlike evaluating a text response, judging an embodied agent requires assessing a sequence of actions, their spatiotemporal relationships, and their alignment with a high-level goal, which necessitates a deep understanding of physics and cause and effect.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The more a model systematically evaluates and refines its own thought processes, the more it becomes an effective reasoner.

“The more a model systematically evaluates and refines its own thought processes, the more it becomes an effective reasoner.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLM-as-a-Judge offers a path to rapidly generate massive, high-quality datasets that would be otherwise impossible to obtain.

“LLM-as-a-Judge offers a path to rapidly generate massive, high-quality datasets that would be otherwise impossible to obtain.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Research exploring how MLLM-as-a-Judge could be applied to the evaluation of data or agents remains limited.

“However, research exploring how MLLM-as-a-Judge could be applied to the evaluation of data or agents remains limited.”

May combine two claims.

This allows the model to correct inconsistencies, identify simpler decompositions of complex problems, and progressively refine its outputs.

“This allows the model to correct inconsistencies, identify simpler decompositions of complex problems, and progressively refine its outputs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Jailbreaking is a significant concern for open-source LLM judges, which may not have the same level of fine-tuning for safety as proprietary models.

“This is a significant concern for open-source LLM judges, which may not have the same level of fine-tuning for safety as proprietary models.”

Probably not a substantial claim: may not pass S4 (based).

Data annotation poses significant challenges for current machine-learning models due to the complexity, subjectivity, and diversity of data.

“Despite its wide applications, data annotation poses significant challenges for current machine-learning models due to the complexity, subjectivity, and diversity of data.”

May combine two claims.

LLM-as-a-Judge excels in tasks such as text quality assessment, model evaluation, and automated data annotation.

“By leveraging the reasoning capabilities of large language models, this framework excels in tasks such as text quality assessment, model evaluation, and automated data annotation.”

Classified as probably not a substantial claim. May combine two claims.

Existing benchmarks evaluate a model's performance on a task, whereas new meta-evaluation frameworks would focus on the evaluator itself.

“Unlike existing benchmarks that evaluate a model's performance on a task, these new frameworks would focus on the evaluator itself.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Addressing the challenges of reliability, robustness, and generalization can lead to more trustworthy, adaptive, and comprehensive evaluators, paving the way for their adoption across scientific research, education, industry, and beyond.

“By addressing the challenges of reliability, robustness, and generalization, we can create more trustworthy, adaptive, and comprehensive evaluators, paving the way for their adoption across scientific research, education, industry, and beyond.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Data annotation requires domain expertise and is resource-intensive, particularly when manually labeling large datasets.

“This process requires domain expertise and is resource-intensive, particularly when manually labeling large datasets.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Reasoning and judgment are two deeply connected yet distinct cognitive abilities.

“As shown in Figure 16, reasoning and judgment are two deeply connected yet distinct cognitive abilities.”

Probably not a substantial claim: may not pass S4 (based).

Current multimodal LLMs like GPT-4 Vision still struggle with sophisticated reasoning that integrates different modalities.

“While powerful in text, current multimodal LLMs like GPT-4 Vision still struggle with sophisticated reasoning that integrates different modalities.”

Probably not a substantial claim: may not pass S4 (based).

A response to a controversial topic might be rated as acceptable by a model version from March, but subsequently penalized by a June version due to new safety fine-tuning.

“For example, a response to a controversial topic might be rated as acceptable by a model version from March, but subsequently penalized by a June version due to new safety fine-tuning.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Current research on the reliability of LLM-as-a-Judge primarily relies on empirical benchmarks, but it lacks a solid theoretical foundation.

“Current research on the reliability of LLM-as-a-Judge primarily relies on empirical benchmarks, but it lacks a solid theoretical foundation.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

In virtual environments or gaming, an LLM judge could assess an agent's strategic ability, navigation skills, or its capacity to solve puzzles in a human-like manner.

“Similarly, in virtual environments or gaming, an LLM judge could assess an agent's strategic ability, navigation skills, or its capacity to solve puzzles in a human-like manner.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Attacks on LLM-as-a-Judge are relatively under-explored compared to attacks on traditional Natural Language Generation models.

“While attacks on traditional Natural Language Generation (NLG) models are well-studied, attacks on LLM-as-a-Judge are relatively under-explored.”

May combine two claims.

Probabilistic outputs of LLM-as-a-Judge can introduce inconsistencies, overconfidence, and biases inherited from training data.

“Ensuring reliability remains a key issue, because probabilistic outputs can introduce inconsistencies, overconfidence, and biases inherited from training data.”

Probably not a substantial claim: may not pass S2 (checkable).

LLM-as-a-Judge is particularly valuable for large-scale, efficient, and adaptable evaluation.

“It is particularly valuable for large-scale, efficient, and adaptable evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

By integrating LLM-as-a-Judge within feedback-rich frameworks, models gain the ability to not only evaluate external content but also to introspect and evolve their own reasoning processes.

“By integrating LLM-as-a-Judge within such feedback-rich frameworks, models gain the ability to not only evaluate external content but also to introspect and evolve their own reasoning processes.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Although current methods have improved the reliability of LLM-as-a-Judge, many challenges, including adaptability and robustness, remain unresolved.

“Although current methods have improved the reliability of LLM-as-a-Judge, many challenges, including adaptability and robustness, remain unresolved.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

This lack of transparency undermines trust in the evaluation process and can be particularly problematic in fields where human oversight and accountability are paramount.

“This lack of transparency undermines trust in the evaluation process and can be particularly problematic in fields where human oversight and accountability are paramount.”

Probably not a substantial claim: may not pass S4 (based).

LLM-as-a-Judge can be used to generate preference labels for RLHF at a scale that is prohibitive for human annotators.

“For instance, it can be used to generate preference labels for RLHF at a scale that is prohibitive for human annotators, thus fueling the development of more sophisticated models.”

May combine two claims.

When an evaluation is deemed unfair or incorrect, there is no clear way to trace the error or hold the model accountable.

“When an evaluation is deemed unfair or incorrect, there is no clear way to trace the error or hold the model accountable.”

Probably not a substantial claim: may not pass S3 (load-bearing).

An LLM judge could evaluate a robot's performance in a complex task like preparing a meal, assessing not only if the final product is correct, but also if the robot's movements were efficient, safe, and followed a logical sequence.

“For instance, an LLM judge could evaluate a robot's performance in a complex task like preparing a meal. The judge would need to assess not only if the final product is correct, but also if the robot's movements were efficient, safe, and followed a logical sequence, such as correctly aligning a cup with a dispenser to avoid spillage.”

May combine two claims.

Recent studies have begun incorporating LLM-as-a-Judge into multi-agent frameworks to guide inter-agent interactions, thereby improving overall decision-making efficiency and quality.

“Recent studies [226] have begun incorporating LLM-as-a-Judge into multi-agent frameworks to guide inter-agent interactions, thereby improving overall decision-making efficiency and quality.”

• cites [226]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Fine-tuned evaluators often exhibit overfitting, meaning their evaluation capabilities may not generalize well beyond their training data.

“While fine-tuned evaluators may seem like a solution, they often exhibit overfitting , meaning their evaluation capabilities may not generalize well beyond their training data.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A model might unfairly penalize a response written in a non-standard English dialect or a writing style associated with a marginalized group.

“For instance, a model might unfairly penalize a response written in a non-standard English dialect or a writing style associated with a marginalized group.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Currently, MLLM-as-a-Judge frameworks are emerging for evaluating models.

“Currently, MLLM-as-a-Judge frameworks [17] are emerging for evaluating models.”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Even for purely text-based tasks, current LLMs have limitations in abstract and causal reasoning.

“Even for purely text-based tasks, current LLMs have limitations in abstract and causal reasoning.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

An attacker could introduce imperceptible perturbations to the text-such as paraphrasing a key sentence or adding a misleading but grammatically correct phrase-to trick the judge model into a different conclusion.

“For example, an attacker could introduce imperceptible perturbations to the text-such as paraphrasing a key sentence or adding a misleading but grammatically correct phrase-to trick the judge model into a different conclusion.”

Probably not a substantial claim: may not pass S4 (based).

When judgment is performed continuously and at a high frequency, an LLM starts to approximate the process of reasoning itself.

“When judgment is performed continuously and at a high frequency-for instance, an LLM evaluating every step of its own thought process-it starts to approximate the process of reasoning itself.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The uncertain and evolving nature of robustness risks underscores the necessity of proactive mitigation strategies.

“Finally, the uncertain and evolving nature of robustness risks underscores the necessity of proactive mitigation strategies.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Traditional human labeling is a major bottleneck, as it is both costly and slow.

“Traditional human labeling is a major bottleneck, as it is both costly and slow.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Evaluation-driven convergence could harm innovation and reduce the richness of the information ecosystem.

“This phenomenon, known as evaluation-driven convergence , could harm innovation and reduce the richness of the information ecosystem.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The paradigm of LLM-as-a-Judge is poised to become a core mechanism for scaling AI development.

“The paradigm of LLM-as-a-Judge is poised to become a core mechanism for scaling AI development, particularly in an era where models are increasingly trained through interactive and iterative feedback loops.”

May combine two claims.

When used as judges, LLMs can amplify and perpetuate societal biases, leading to unfair evaluations.

“When used as judges, these models can amplify and perpetuate these biases, leading to unfair evaluations.”

• AssumptionRests on an assumption: LLMs are trained on vast amounts of internet data that often contains societal biases related to gender, race, and other demographics.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Fine-tuned evaluator models can inherit subtle biases from their training datasets, necessitating careful meta-evaluation to ensure fairness.

“These models can also inherit subtle biases from their training datasets, necessitating careful meta-evaluation to ensure fairness.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

LLM-as-a-Judge is more than just an evaluation tool; it's a mechanism for enhancing a model's reasoning capabilities.

“This is why LLM-as-a-Judge is more than just an evaluation tool; it's a mechanism for enhancing a model's reasoning capabilities.”