LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

While LLMs-as-judges have garnered significant attention for their effectiveness in diverse scenarios, the exploration of various biases that impact their performance remains relatively underdeveloped.

“In summary, while LLMs-as-judges have garnered significant attention for their effectiveness in diverse scenarios, the exploration of various biases that impact their performance remains relatively underdeveloped.”

May combine two claims.

Existing LLM systems often face significant computational costs when performing evaluation tasks; pairwise comparison methods require multiple rounds of comparisons for each candidate, which becomes extremely time-consuming as the number of candidates grows.

“Existing LLM systems often face significant computational costs when performing evaluation tasks.For example, pairwise comparison methods require multiple rounds of comparisons for each candidate, which becomes extremely time-consuming as the number of candidates grows.”

May combine two claims.

Although preliminary studies have highlighted the vulnerability of LLMs-as-judges to adversarial manipulations, this field remains largely underexplored.

“Although preliminary studies [184,196] have highlighted the vulnerability of LLMs-as-judges to adversarial manipulations, this field remains largely underexplored.”

• cites [184]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [196]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

LLMs may be influenced by biases present in their training data, leading to unfair judgments in different social, cultural, or legal contexts.

“LLMs may be influenced by biases present in their training data, leading to unfair judgments in different social, cultural, or legal contexts.”

May combine two claims.

Current LLMs-as-judges systems often treat reasoning and evaluation capabilities as distinct and independent modules, which can hinder effectiveness when addressing complex tasks.

“Current LLMs-as-judges systems often treat reasoning and evaluation capabilities as distinct and independent modules, which can hinder effectiveness when addressing complex tasks.”

May combine two claims.

Current LLM judges often rely on manually predefined evaluation criteria, lacking the ability to adapt dynamically during the assessment process.

“Current LLM judges often rely on manually predefined evaluation criteria, lacking the ability to adapt dynamically during the assessment process.”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [233]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [273]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [280]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [286]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Current LLM-as-Judge systems often operate as black boxes, with their rulings lacking transparency and a clear reasoning process.

“Current LLM-as-Judge systems often operate as black boxes, with their rulings lacking transparency and a clear reasoning process.”

May combine two claims.

LLMs are sensitive to noise, incompleteness, or ambiguity in input instructions, which may lead to errors or instability in evaluation results when handling complex or highly uncertain texts.

“LLMs are sensitive to noise, incompleteness, or ambiguity in input instructions, which may lead to errors or instability in evaluation results when handling complex or highly uncertain texts.”

May combine two claims.

Current LLMs-as-Judge systems typically rely on a single model for evaluation; while straightforward, this approach is prone to biases inherent in individual models, leading to reduced accuracy and stability.

“Current LLMs-as-Judge systems typically rely on a single model for evaluation.While this approach is straightforward, it is prone to biases inherent in individual models, leading to reduced accuracy and stabilitys.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Existing LLM judges often exhibit limited adaptability in practical applications; while they may perform effectively on specific downstream tasks, they frequently struggle in cross-domain or multi-task settings.

“Existing LLM judges often exhibit limited adaptability in practical applications.While these judges may perform effectively on specific downstream tasks, they frequently struggle in cross-domain or multi-task settings, thereby falling short of meeting the diverse and broader demands of real-world applications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The multi-LLM evaluation paradigm, which relies on multiple rounds of interaction, further exacerbates computational demands.

“Similarly, the multi-LLM evaluation paradigm, which relies on multiple rounds of interaction, further exacerbates computational demands.”

Probably not a substantial claim: may not pass S4 (based).

Current LLMs-as-Judge systems often fall short when handling tasks in specialized fields due to insufficient domain knowledge.

“Current LLMs-as-Judge systems often fall short when handling tasks in specialized fields due to insufficient domain knowledge.”

May combine two claims.

Human-AI collaboration systems offer greater reliability and depth but face challenges in coordinating between models and humans, ensuring consistent evaluation standards, and integrating feedback.

“While this system offers greater reliability and depth, it comes with challenges in coordinating between the models and humans, ensuring consistent evaluation standards, and integrating feedback.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Creative Beam Search uses an LLM as a judge to simulate human creative selection, enhancing diversity and creativity of generated results.

“Also, Musolesi et al. [160] proposed Creative Beam Search, with the LLM acting as a judge to simulate the human creative selection process, thereby enhancing the diversity and creativity of the generated results.”

• cites [160]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Pairwise evaluation closely mirrors human decision-making processes by focusing on relative preferences rather than assigning absolute scores.

“Pairwise evaluation closely mirrors human decision-making processes by focusing on relative preferences rather than assigning absolute scores.”

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [84]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [89]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [188]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Listwise evaluation takes into account interactions between multiple candidates, making it well-suited for applications that require holistic analysis.

“Listwise evaluation takes into account the interactions between multiple candidates, making it well-suited for applications that require holistic analysis.”

• cites [87]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [166]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [263]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [302]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Tree of Thoughts (ToT) models problem-solving as a tree structure, allowing the model to explore multiple solution paths and optimize path selection through self-assessment mechanisms.

“Further optimization strategies include the Tree of Thoughts (ToT) [265] method, which models the problem-solving process as a tree structure.This allows the model to explore multiple solution paths and optimize path selection through self-assessment mechanisms.”

• cites [265]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Reference-free evaluation provides greater flexibility for open-ended tasks but has difficulty obtaining satisfactory evaluations in domains where the LLM lacks relevant knowledge.

“The main advantage of this approach is its independence from specific references, providing greater flexibility for open-ended tasks.However, its drawback lies in the difficulty of obtaining satisfactory evaluations in domains where the LLM lacks relevant knowledge.”

• cites [82]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [194]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [292]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

CREAM introduces cross-iteration consistency constraints to regulate training and prevent learning unreliable preference data, significantly enhancing reward consistency and alignment performance.

“To address this issue, CREAM [239] introduces cross-iteration consistency constraints to regulate the training process and prevent the model from learning unreliable preference data.This significantly enhances reward consistency and alignment performance.”

• cites [239]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Multi-LLM evaluation systems can cover a broader range of evaluation criteria and provide a more comprehensive assessment by leveraging the strengths of different models.

“By leveraging the strengths of different models, a multi-model system can cover a broader range of evaluation criteria and provide a more comprehensive assessment.”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [39]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [132]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Linguistic quality is crucial in tasks like text generation, machine translation, and summarization, where clarity and readability are essential.

“Linguistic quality is crucial in tasks like text generation, machine translation, and summarization, where clarity and readability are essential.”

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [59]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [283]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The LLMs-as-judges framework has attracted growing attention from both academia and industry due to its effectiveness, ability to generalize across tasks, and interpretability in natural language.

“This framework has attracted growing attention from both academia and industry due to their excellent effectiveness, ability to generalize across tasks, and interpretability in the form of natural language.”

• AssumptionRests on an assumption: Traditional metrics like BLEU and ROUGE are insufficient for comprehensive evaluation of natural language generation tasks.
• AssumptionRests on an assumption: Human annotations are the ground truth for evaluating model outputs.
• AssumptionRests on an assumption: Collecting human annotations is typically time-consuming and resource-intensive, making it challenging to scale up for large-scale evaluation.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Distinct tasks and domains require specific evaluation criteria, making it difficult for LLMs to adapt their standards dynamically to specific contexts.

“Moreover, distinct tasks and domains require specific evaluation criteria, making it difficult for LLMs to adapt their standards dynamically to specific contexts.”

May combine two claims.

The three evaluation modes are not entirely independent; pointwise scores can be aggregated to create pairwise comparisons or ranked lists, and pairwise preferences can be organized into a complete ranking list for listwise analysis.

“In general, these three evaluation modes are not entirely independent.pointwise scores can be aggregated to create pairwise comparisons or used to construct a ranked list.Similarly, pairwise preferences can be organized into a complete ranking list for listwise analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LLMs exhibit remarkable generalization and adaptability, with strong transfer capabilities across previously unseen tasks and diverse domains.

“LLMs exhibit remarkable generalization and adaptability, showcasing strong transfer capabilities across previously unseen tasks and diverse domains [1,9].”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Performance evaluation is the most fundamental application objective of an LLM judge, serving as the cornerstone for understanding and optimizing their other function.

“Performance evaluation represents the most fundamental application objective of an LLM judges, serving as the cornerstone for understanding and optimizing their other function.”

Classified as probably not a substantial claim.

Current LLMs are still not sufficiently reliable for automated paper review, particularly in scenarios requiring logical reasoning or a deep knowledge base.

“Their findings indicate that current LLMs are still not sufficiently reliable for such tasks, particularly in scenarios requiring logical reasoning or a deep knowledge base.”

• cites [297]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

LLM judges can adjust their evaluation criteria based on the specific task context, rather than relying on a fixed set of metrics, making the evaluation process more flexible and refined.

“First, LLM judges can adjust their evaluation criteria based on the specific task context, rather than relying on a fixed set of metrics, making the evaluation process more flexible and refined.”

• AssumptionRests on an assumption: LLMs have extensive knowledge and deep contextual understanding that enables them to flexibly adapt to various evaluation tasks.

May combine two claims.

Single-LLM evaluation systems have limited flexibility and may struggle with tasks that demand specialized knowledge or reasoning over complex inputs.

“However, its flexibility is limited, as it may struggle with tasks that demand specialized knowledge or reasoning over complex inputs.”

May combine two claims.

The evaluation results of LLMs are often influenced by the prompt template, which can lead to biased or inconsistent assessments.

“For example, the evaluation results of LLMs are often influenced by the prompt template, which can lead to biased or inconsistent assessments [260].”

• cites [260]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Graph of Thoughts (GoT) extends ToT by introducing directed graphs, where non-linear interactions between nodes improve efficiency and precision of multi-step reasoning.

“The Graph of Thoughts (GoT) [15] method extends this concept by introducing directed graphs, where the non-linear interactions between nodes improve the efficiency and precision of multi-step reasoning.”

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Pairwise evaluation is especially effective when the differences between outputs are subtle and difficult to quantify.

“This approach is especially effective when the differences between outputs are subtle and difficult to quantify.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multi-LLM evaluation systems come at a higher computational cost and require more resources, making deployment and maintenance more challenging, particularly for large-scale tasks.

“However, this comes at a higher computational cost and requires more resources, making deployment and maintenance more challenging, particularly for large-scale tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Single-LLM evaluation systems are simple to deploy and scale, making them efficient for tasks that do not require specialized evaluation.

“A single LLM evaluation system relies on a single model to perform the evaluation tasks.It is simple to deploy and scale, making it efficient for tasks that don't require specialized evaluation.”

• cites [141]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [145]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [280]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Single-LLM evaluation systems, if not properly trained, may introduce biases leading to inaccurate evaluations.

“Additionally, if not properly trained, a single model may introduce biases, leading to inaccurate evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

SE-GBS integrates self-assessment into multi-step reasoning decoding, generating scores that reflect logical correctness and ensuring accuracy and consistency of the reasoning chain.

“SE-GBS [250] integrates self-assessment into the multi-step reasoning decoding process, generating scores that reflect logical correctness and further ensuring the accuracy and consistency of the reasoning chain.”

• cites [250]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Step-by-step validation can enhance the performance of LLMs in multi-step reasoning tasks, particularly in mathematics.

“Similarly, Lightman et al. [137] discuss how step-by-step validation can enhance the performance of LLMs in multi-step reasoning tasks, particularly in the domain of mathematics.”

• cites [137]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Using the same LLM for both policy generation and reward modeling can pose challenges in ensuring the accuracy of rewards, potentially leading to accumulated biases and preference data noise that undermine training effectiveness.

“However, using the same LLM for both policy generation and reward modeling can pose challenges in ensuring the accuracy of the rewards.This dual role setup may lead to accumulated biases and preference data noise, which can undermine the training effectiveness.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Content accuracy is particularly crucial in tasks such as code generation and fact-checking.

“Content accuracy is particularly crucial in tasks such as code generation and fact-checking.”

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [101]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [213]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Pointwise evaluation may fail to capture relative quality differences between candidates and can be influenced by biases arising from evaluating items in isolation.

“Although pointwise evaluation is simple and easy to apply, it may fail to capture the relative quality differences between candidates and can be influenced by biases arising from evaluating items in isolation.”

• cites [109]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [229]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [266]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

CGPO groups tasks by category and uses Mixed Judges to assign a specific reward model to each task group, ensuring reward signals align with task objectives and preventing conflicts between goals.

“In addition, CGPO [257] groups tasks by category (such as dialogue, mathematical reasoning, safety, etc.) and uses”

• cites [257]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

LLM judges offer a scalable and reproducible alternative to human evaluation, significantly reducing the costs and time associated with human involvement.

“Finally, LLM judges offer a scalable and reproducible alternative to human evaluation, significantly reducing the costs and time associated with human involvement.”

• AssumptionRests on an assumption: Collecting human annotations is typically time-consuming and resource-intensive, making it challenging to scale up for large-scale evaluation.

May combine two claims.

Human-AI collaboration systems increase both cost and time, making them less scalable than purely model-based systems.

“Additionally, the inclusion of human evaluators increases both the cost and time required for the evaluation process, making it less scalable than purely model-based systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Generating feedback F generally demands a higher level of model capability than generating evaluation result Y, as it requires assessing quality and providing actionable recommendations.

“Moreover, generating feedback F generally demands a higher level of model capability, as it requires not only assessing the quality of the input but also providing concrete, actionable recommendations for improvement.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [155]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

LLM judges can generate interpretive evaluations, offering more comprehensive feedback on model performance and enabling researchers to gain deeper insights into the evaluator's strengths and weaknesses.

“Second, LLM judges can generate interpretive evaluations, offering more comprehensive feedback on model performance and enabling researchers to gain deeper insights into the evaluater's strengths and weaknesses.”

• AssumptionRests on an assumption: LLMs have extensive knowledge and deep contextual understanding that enables them to flexibly adapt to various evaluation tasks.

Probably not a substantial claim: may not pass S3 (load-bearing).

In multi-LLM systems, the methods through which models achieve consensus or resolve differences remain key areas of ongoing exploration.

“Moreover, while cooperation between models often enhances evaluation results, the methods through which these models achieve consensus or resolve differences remain key areas of ongoing exploration.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

FRANK's detailed categorization of errors offers valuable insights into the types of factual inaccuracies common in generated summaries, highlighting areas where LLMs often struggle.

“FRANK's detailed categorization of errors offers valuable insights into the types of factual inaccuracies common in generated summaries, highlighting areas where LLMs often struggle.”

• cites [169]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MANS dataset forms part of the OpenMEVA framework and compiles stories from various natural language generation models using well-known corpora like ROCStories and WritingPrompts.

“Another notable dataset is the MANS [73], which forms part of the OpenMEVA [73] framework.It compiles stories from various natural language generation models using well-known corpora like ROCStories [159] and WritingPrompts [60].”

• cites [73]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [159]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [60]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

GED addresses inconsistencies in LLM preference evaluations by leveraging multiple weak evaluators to construct preference graphs, and then utilizing a DAG structure to ensemble and denoise these graphs for better, non-contradictory evaluation results.

“In addition to that, GED [90] addresses inconsistencies in LLM preference evaluations by leveraging multiple weak evaluators to construct preference graphs, and then utilize DAG structure to ensemble and denoise these graphs for better, non-contradictory evaluation results.”

• cites [90]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Chen et al. found that within retrieval-augmented generation (RAG) frameworks, LLM judges do not exhibit a significant self-preference effect during evaluation.

“Moreover, Chen et al. [33] found that within retrieval-augmented generation (RAG) frameworks, LLM judges do not exhibit a significant self-preference effect during evaluation.”

• cites [33]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Valmeekam et al. raise concerns about the effectiveness of LLMs as self-validation tools in the absence of reliable external validators.

“Valmeekam et al. [224] also raise concerns about the effectiveness of LLMs as self-validation tools in the absence of reliable external validators.”

• cites [224]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

CVALUES benchmark is the first comprehensive framework tailored to assess values alignment in the Chinese language context, focusing on two critical criteria: Safety and Responsibility.

“Moreover, the CVALUES [255] benchmark is a more recent contribution aimed at evaluating human values alignment specifically for Chinese LLMs.It represents the first comprehensive framework tailored to assess values alignment in the Chinese language context, focusing on two critical criteria: Safety and Responsibility.”

• cites [255]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Meta-evaluation is a crucial step in determining the reliability, consistency, and validity of LLM judges.

“Meta-evaluation, the process of assessing the quality of the evaluator itself, is a crucial step in determining the reliability, consistency, and validity of LLM judges.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Xu et al. propose a multi-agent collaboration strategy to enhance the reasoning abilities of LLMs by simulating the academic peer review process.

“Xu et al. [261] propose a multi-agent collaboration strategy to enhance the reasoning abilities of LLMs by simulating the academic peer review process.”

• cites [261]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Human-AI Collaboration Systems bridge the gap between automated LLM judgments and the essential need for human oversight, particularly in high-stakes domains such as law, healthcare, and education.

“Human-AI Collaboration Systems bridge the gap between automated LLM judgments and the essential need for human oversight, particularly in high-stakes domains such as law, healthcare, and education.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Per-DOC includes 7,000 instances of paired stories generated from the same premise.

“Per-DOC includes 7,000 instances of paired stories generated from the same premise.”

• cites [229]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

StoryER dataset offers a distinct approach to evaluating story generation by focusing on preference prediction and aspect-based rating.

“The StoryER [26] dataset offers a distinct approach to evaluating story generation by focusing on preference prediction and aspect-based rating.”

• cites [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Zhou et al. found that while LLMs show certain promise in the domain of automated reviewing, they are not yet sufficient to fully replace human reviewers, particularly in areas with high technical complexity or strong innovation.

“They found that while LLMs show certain promise in the domain of automated reviewing, they are not yet sufficient to fully replace human reviewers, particularly in areas with high technical complexity or strong innovation.”

• cites [297]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The HANNA dataset is tailored for evaluating automatic story generation (ASG), featuring 1,056 stories generated by 10 different systems from 96 prompts.

“The HANNA [34] dataset is tailored for evaluating automatic story generation (ASG), featuring 1,056 stories generated by 10 different systems from 96 prompts.”

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

BiasAlert integrates external human knowledge with the LLM judge's inherent reasoning capabilities to reliably identify and mitigate bias, outperforming GPT4-as-A-Judge across various scenarios.

“For instance, BiasAlert [61], a tool designed to detect social bias in LLM-generated open-text outputs.It integrates external human knowledge with the LLM judge's inherent reasoning capabilities to reliably identify and mitigate bias, outperforming GPT4-as-A-Judge across various scenarios.”

• cites [61]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Due to the convenience and effectiveness of LLM Judges, they have been widely applied as judges across various domains.

“Due to the convenience and effectiveness of LLM Judges, they have been widely applied as judges across various domains.”

May combine two claims.

PRE can automatically evaluate LLMs through a peer-review process, selecting qualified LLMs as reviewers through a qualification exam and aggregating their ratings using weights proportional to their agreement with humans, demonstrating effectiveness and robustness in evaluating text summarization tasks.

“Likewise, PRE [27,39] can automatically evaluate LLMs through a peer-review process.It selects qualified LLMs as reviewers through a qualification exam and aggregates their ratings using weights which is proportional to their agreement of humans, demonstrating effectiveness and robustness in evaluating text summarization tasks.”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [39]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

WMT primarily focuses on high-resource languages, which may limit its applicability to low-resource or underrepresented languages.

“However, WMT primarily focuses on high-resource languages, which may limit its applicability to low-resource or underrepresented languages.”

• cites [66]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Using text-only LLM judges resulted in a closer alignment with human preferences than LVLMs.

“Interestingly, they found that using the text-only LLM judges resulted in a closer alignment with human preferences than LVLMs.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Focusing solely on factual errors may overlook other aspects of summary quality, such as coherence and fluency.

“However, focusing solely on factual errors may overlook other aspects of summary quality, such as coherence and fluency.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Effective evaluation in education requires LLMs to consider not only correctness but also creativity, clarity, and logical coherence.

“Effective evaluation in education requires LLMs to consider not only correctness but also creativity, clarity, and logical coherence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Babaei et al. demonstrated how LLMs can process unstructured text data, such as customer histories, contract terms, and news reports, to enhance the precision of credit assessments.

“For example, Babaei et al. [7] demonstrated how LLMs can process unstructured text data, such as customer histories, contract terms, and news reports, to enhance the precision of credit assessments.”

• cites [7]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In biology, the main evaluation challenges include the complexity and diversity of the data and the need for specialized biological knowledge.

“The main evaluation challenges in biological field include the complexity and diversity of the data and the need for specialized biological knowledge [86,167].”

• cites [86]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [167]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Traditional ranking models often rely on shallow features or direct matching scores, which may not yield optimal results.

“Traditional ranking models often rely on shallow features or direct matching scores, which may not yield optimal results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ReSTEM utilizes a self-training approach based on the expectation-maximization framework to enhance the problem-solving capabilities of large language models, particularly in areas such as solving mathematical problems and generating code.

“ReSTEM [199], on the other hand, utilizes a self-training approach based on the expectation-maximization framework to enhance the problemsolving capabilities of large language models, particularly in areas such as solving mathematical problems and generating code.”

• cites [199]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In-Context Learning allows LLMs to dynamically adapt to evaluation tasks using carefully curated examples or explanations within the prompt.

“In-Context Learning (ICL) is a distinctive capability of LLMs that allows them to dynamically adapt to evaluation tasks using carefully curated examples or explanations within the prompt [53].”

• cites [53]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Yelp dataset contains detailed review data from 11 metropolitan areas, covering approximately 150,000 businesses, nearly 7 million user reviews, and over 200,000 images.

“Another commonly used dataset is the Yelp dataset [4], which contains detailed review data from 11 metropolitan areas, covering approximately 150,000 businesses, nearly 7 million user reviews, and over 200,000 images.”

• cites [4]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

In the legal domain, interpretability and transparency of the evaluation results are essential core requirements, as legal practice highly depends on clear logic and verifiable conclusions.

“At the same time, interpretability and transparency of the evaluation results are essential core requirements, as legal practice highly depends on clear logic and verifiable conclusions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Zhang et al. annotated the MovieLens data to create a sub-dataset featuring user self-explanation texts, which are rated on a five-point Likert scale across four dimensions: Persuasiveness, Transparency, Accuracy, and Satisfaction.

“Zhang et al. [284] further annotated the MovieLens [80] data to create a sub-dataset featuring user self-explanation texts.In this sub-dataset, users write explanatory texts after being presented with a recommended movie.These explanations are then rated on a five-point Likert scale across four dimensions: Persuasiveness, Transparency, Accuracy, and Satisfaction.”

• cites [284]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [80]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

ChatEval is a multi-agent debate framework that utilizes multiple LLMs with diverse role prompts and communication strategies on open-ended questions and traditional NLG tasks, significantly improving evaluation performance compared to single-agent methods.

“ChatEval [22] is another multi-agent debate framework that utilizes multiple LLMs with diverse role prompts and communication strategies on open-ended questions and traditional NLG tasks, significantly improves evaluation performance compared to single-agent methods.”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The LLMAAA framework incorporates an active learning strategy to select high-information samples, mitigating the effects of noisy labels and reducing reliance on costly human annotation.

“Additionally, the LLMAAA [282] framework incorporates an active learning strategy to efficiently select highinformation samples for annotation, thereby mitigating the effects of noisy labels and reducing the reliance on costly human annotation.”

• cites [282]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Latif et al. demonstrated that, with data augmentation, LLM-annotated samples can significantly enhance the performance of speech emotion recognition models.

“Latif et al. [117] explored the application of LLMs in speech emotion annotation, demonstrating that, with data augmentation, LLM-annotated samples can significantly enhance the performance of speech emotion recognition models.”

• cites [117]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Traditional evaluation methods for RAG systems rely on large manually annotated datasets, which are time-consuming and costly.

“Traditional evaluation methods rely on large manually annotated datasets, which are time-consuming and costly.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MovieLens dataset is a widely-used public dataset for movie recommendations, available in multiple versions with varying scales, ranging from thousands of users and ratings to millions.

“The MovieLens [80] dataset is a widely-used public dataset for movie recommendations, available in multiple versions with varying scales, ranging from thousands of users and ratings to millions.”

• cites [80]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

In the multimodal domain, one primary challenge in evaluating multimodal tasks lies in the significant heterogeneity among modalities, including substantial differences in data structures, representation methods, and feature distributions.

“One of the primary challenges in evaluating multimodal tasks lies in the significant heterogeneity among these modalities, including substantial differences in data structures, representation methods, and feature distributions.”

May combine two claims.

Brake et al. leveraged LLMs, such as Llama2, to assess clinical note consistency, with results indicating agreement levels comparable to human annotators.

“Similarly, Brake et al. [17] leveraged LLMs, such as Llama2, to assess clinical note consistency, with results indicating agreement levels comparable to human annotators.”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Data Synthesis enables the generation of diverse data samples, enhancing a model's generalization ability to unseen examples while reducing reliance on sensitive real-world data.

“Data Synthesis enables the generation of diverse data samples, enhancing a model's generalization ability to unseen examples while reducing reliance on sensitive real-world data [2,52,108,157,235,268].”

• cites [2]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [52]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [108]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [157]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [235]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [268]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

For dialogue quality, Fusion-Eval assessed six aspects: coherence, engagingness, naturalness, groundedness, understandability, and overall quality, attaining a turn-level Spearman correlation of 0.744.

“For dialogue quality, it assessed six aspects: coherence, engagingness, naturalness, groundedness, understandability, and overall quality, attaining a turn-level Spearman correlation of 0.744.”

• cites [198]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

By incorporating human insights, Hybrid systems can ensure the final judgment is more reliable and aligned with ethical considerations, and empower continuous model improvement through feedback loops.

“By incorporating human insights, Hybrid systems can ensure the final judgment is more reliable and aligned with ethical considerations, and empower continuous model improvement through feedback loops.”

May combine two claims.

Törnberg et al. found that ChatGPT-4 surpassed human classifiers in accuracy and reliability for classifying Twitter users' political leanings, with bias levels comparable to or lower than those of human classifiers.

“Their findings revealed that ChatGPT-4 not only surpassed human classifiers in accuracy and reliability but also exhibited bias levels that were comparable to or lower than those of human classifiers.”

• cites [217]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

TREC DL datasets are derived from the expanded MS MARCO v2 collection, which contains approximately 138 million passages, with relevance judgments provided by NIST assessors using a 4-point scale (0 to 3).

“These datasets are derived from the expanded MS MARCO v2 collection [11], which contains approximately 138 million passages.Relevance judgments are provided by assessors from the National Institute of Standards and Technology (NIST) using a 4-point scale (0 to 3).”

• cites [11]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LeCaRDv2 is a large-scale dataset tailored for legal case retrieval, incorporating three distinct aspects: characterization, penalty, and procedure.

“One notable example is LeCaRDv2 [129], a large-scale dataset tailored for legal case retrieval.LeCaRDv2 enriches the concept of relevance by incorporating three distinct aspects: characterization, penalty, and procedure.”

• cites [129]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

SELF-DEBUGGING demonstrates self-correction in code generation by identifying and rectifying errors through self-explanation and feedback, significantly enhancing performance across various code generation tasks.

“On the other hand, SELF-DEBUGGING [32] demonstrates a practical application of self-correction in code generation by identifying and rectifying errors through self-explanation and feedback.This approach has significantly enhanced the performance of LLMs across various code generation tasks.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

These systems allow human evaluators to provide real-time adjustments, enhancing the accuracy and trustworthiness of the evaluation process.

“These systems allow human evaluators to provide real-time adjustments, enhancing the accuracy and trustworthiness of the evaluation process.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

With the increasing capabilities of large language models (LLMs), evaluating dialogue generation has become more complex, requiring multi-faceted evaluation frameworks to assess various aspects of conversational quality.

“With the increasing capabilities of large language models (LLMs), evaluating dialogue generation has become more complex, requiring multi-faceted evaluation frameworks to assess various aspects of conversational quality.”

May combine two claims.

FineSurE breaks down evaluation into multiple dimensions such as faithfulness, completeness, and conciseness, and outperforms traditional methods in terms of evaluation accuracy.

“FineSurE [203] is an advanced example of this method, offering a framework for the fine-grained evaluation of text summarization quality.It breaks down the evaluation into multiple dimensions, such as faithfulness, completeness, and conciseness.Through detailed analysis, including fact-checking and key fact alignment, FineSurE [203] outperforms traditional methods in terms of evaluation accuracy.”

• cites [203]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

PoLL demonstrates that using a diverse panel of smaller models as judges through max voting and average pooling is not only an effective method for evaluating LLM performance, but also reduces intra-model bias of a single large model.

“Furthermore, PoLL [225] demonstrates that using a diverse panel of smaller models as judges through max voting and average pooling is not only an effective method for evaluating LLM performance, but also reduces intra-model bias of a single large model.”

• cites [225]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Human oversight after the initial automated evaluation can guarantee fairness and consistency with academic standards.

“This process illustrates how human oversight after the initial automated evaluation can guarantee fairness and consistentcy with academic standards.”

• cites [35]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

PKU-SafeRLHF dataset comprises 83.4K preference entries, focusing on two primary dimensions: harmlessness and usefulness.

“One notable dataset is PKU-SafeRLHF [96], which was specifically curated for studying safe alignment in large language models.The dataset comprises 83.4K preference entries, focusing on two primary dimensions: harmlessness and usefulness.”

• cites [96]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Evaluating story generation systems is inherently complex, as it involves assessing not only linguistic quality but also narrative elements like coherence, engagement, and surprise.

“Evaluating story generation systems is inherently complex, as it involves assessing not only linguistic quality but also narrative elements like coherence, engagement, and surprise.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In software engineering, the challenges that LLMs-as-judges need to overcome include complex code structures and the diversity of evaluation criteria.

“The challenges that LLMs-as-judges need to overcome in the software engineering domain include complex code structures and the diversity of evaluation criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Evol-Instruct enhances LLM performance by automatically generating instruction data, significantly boosting model capabilities.

“Evol-Instruct [254,278] further enhances LLM performance by automatically generating instruction data, significantly boosting model capabilities.”

• cites [254]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [278]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the legal domain, the application of LLMs requires a deep understanding of the legal framework of specific jurisdictions, complex legal language, and rigorous logical reasoning abilities.

“In the legal domain, the application of LLMs requires a deep understanding of the legal framework of specific jurisdictions, complex legal language, and rigorous logical reasoning abilities [128].”

• cites [128]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Communication among LLMs can enable emergent abilities through their interactions, leading to a cohesive decision-making process and better judgment performance.

“Recent research has shown that communication among LLMs can enable emergent abilities through their interactions [261], leading to a cohesive decision-making process and better judgment performance.”

• cites [261]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Each HANNA story is annotated by three human reviewers across six criteria: relevance, coherence, resonance, surprise, engagement, and complexity.

“Each story is annotated by three human reviewers across six criteria: relevance, coherence, resonance, surprise, engagement, and complexity.”

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In information retrieval, evaluating systems presents several challenges, particularly due to the complexity of real-world data, the diversity of user needs, and personalization.

“However, evaluating these systems presents several challenges, particularly due to the the complexity of real-world data, the diversity of user needs, and personalization.”

May combine two claims.

Badshah et al. introduced a reference-guided verdict method for evaluating free-form text using multiple LLMs as judges; combining these LLMs through majority vote significantly improves the reliability and accuracy of evaluations, particularly for complex tasks.

“For example, Badshah et al. [8] introduced a reference-guided verdict method for evaluating free-form text using multiple LLMs as judges.Combining these LLMs through majority vote significantly improves the reliability and accuracy of evaluations, particularly for complex tasks.”

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Cooperation among multi-LLMs can complement each other's strengths to a certain degree, but this method still includes the risk of groupthink, where similar models reinforce each other's biases rather than providing diverse insights.

“Although the cooperation can complement each other's strengths between LLMs to a certain degree, this method still includes the risk of groupthink, where similar models reinforce each other's biases rather than providing diverse insights.”

May combine two claims.

PERSER dataset addresses the subjectivity inherent in open-domain text generation evaluations by restructuring existing datasets and introducing personalized tags, resulting in two sub-datasets: Per-MPST and Per-DOC.

“The PERSER [229] dataset takes a different approach by addressing the subjectivity inherent in open-domain text generation evaluations.PERSER restructures existing datasets and introduces personalized tags, resulting in two sub-datasets: Per-MPST and Per-DOC.”

• cites [229]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

To evaluate LLM-based judges, a common approach is to measure their alignment with human preferences, as human judgments are often considered the gold standard for quality and reliability.

“To evaluate LLM-based judges, a common approach is to measure their alignment with human preferences, as human judgments are often considered the gold standard for quality and reliability.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

In the legal domain, bias and fairness of the model are of significant concern, as any bias in legal evaluations could have a profound impact on judicial fairness.

“Furthermore, the bias and fairness of the model are of significant concern, as any bias in legal evaluations could have a profound impact on judicial fairness.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

HHH dataset is designed to evaluate LLM performance across various human-model interaction scenarios, emphasizing three core human-centered values: honesty, helpfulness, and harmlessness.

“Another influential dataset is the HHH [6] (Honesty, Helpfulness, and Harmlessness) dataset, designed to evaluate LLM performance across various human-model interaction scenarios.The dataset emphasizes three core human-centered values: honesty, helpfulness, and harmlessness.”

• cites [6]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

He et al. found that GPT-4 achieved an accuracy of 83.6% in text annotation, while the highest accuracy achievable by MTurk workers was 81.5%.

“Their comparative study revealed that, even with best practices, the highest accuracy achievable by MTurk workers was 81.5%, whereas GPT-4 achieved an accuracy of 83.6%.”

• cites [85]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

In recommendation systems, explanation evaluation plays a crucial role in helping users understand why a specific product, movie, or piece of content is recommended.

“In recommendation systems, explanation evaluation plays a crucial role in helping users understand why a specific product, movie, or piece of content is recommended.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

StoryER's first component is a 100k Story Ranking Data, which pairs stories from the WritingPrompts dataset, each pair including one story with high user engagement (upvotes ≥ 50) and another with low engagement (upvotes ≤ 0).

“StoryER is divided into two primary components: the first is a 100k Story Ranking Data, which pairs stories from the WritingPrompts dataset.Each pair includes one story with high user engagement (upvotes ≥ 50) and another with low engagement (upvotes ≤ 0).”

• cites [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MANS focuses on manual annotations of narrative elements, serving as a robust testbed for exploring diverse evaluation metrics.

“MANS [73] focuses on manual annotations of narrative elements, serving as a robust testbed for exploring diverse evaluation metrics.”

• cites [73]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

STaR employs a self-guided iterative process to improve model performance on complex reasoning tasks.

“STaR [277] employs a self-guided iterative process to improve model performance on complex reasoning tasks, offering an effective solution for tackling increasingly sophisticated reasoning challenges in the future.”

• cites [277]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

HelpSteer and HelpSteer2 are designed to improve the alignment and usefulness of LLMs, providing multi-attribute data.

“Datasets such as HelpSteer [238] and HelpSteer2 [237] are designed to improve the alignment and usefulness of LLMs.They provide multi-attribute data, enabling the training of models that can generate responses that are factually correct, coherent, and tailored to diverse user preferences.”

• cites [238]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [237]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

PORTIA enables the model to make more consistent and unbiased judgments, focusing on answer quality rather than order.

“By presenting content in a balanced and aligned format, PORTIA enables the model to make more consistent and unbiased judgments, focusing on answer quality rather than order.”

• cites [134]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Adversarial attacks on LLMs can be classified into three main categories based on manipulation level: text-level manipulations, structural and semantic distortions, and optimization-based attacks.

“These attacks can be classified into three main categories based on the manipulation level: text-level manipulations, structural and semantic distortions, and optimization-based attacks.”

May combine two claims.

PRD introduces Peer Rank (PR) and Peer Discussion (PD) mechanisms.

“To address self-enhancement bias, PRD [132] introduces Peer Rank (PR) and Peer Discussion (PD) mechanisms.”

• cites [132]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Sentiment bias in LLMs-as-judges refers to the tendency of the model to favor responses that exhibit certain emotional tones, such as positive or neutral sentiments, over others, regardless of their actual content quality or relevance.

“Sentiment bias [267] in the context of LLMs-as-judges refers to the tendency of the model to favor responses that exhibit certain emotional tones, such as positive or neutral sentiments, over others, regardless of their actual content quality or relevance.”

• cites [267]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

JUDGEBENCH proposes a novel approach to assessing LLM-based judges on challenging response pairs across domains like knowledge, reasoning, mathematics, and coding, introducing preference labels that reflect objective correctness.

“JUDGEBENCH [213] proposes a novel approach to assessing LLM-based judges on challenging response pairs across domains like knowledge, reasoning, mathematics, and coding.It addresses the limitations of existing benchmarks by introducing preference labels that reflect objective correctness, providing a robust platform for evaluating the capabilities of advanced LLM-based judges.”

• cites [213]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Incorporating external feedback mechanisms during judgment introduces an objective and independent judgment mechanism not influenced by the LLM's internal iterations or self-perception.

“One potential solution involves incorporating external feedback mechanisms during judgment, as it introduces an objective and independent judgment mechanism that is not influenced by the LLM's internal iterations or self-perception.”

• cites [259]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The swap-based debiasing method can be divided into two categories: score-based and comparison-based.

“The swap-based debiasing method can be further divided into two categories: score-based and comparison-based.”

May combine two claims.

Discussion-based methods incorporate peer ranking and discussion to improve evaluation accuracy, reducing individual positional bias and enhancing alignment with human judgments.

“Discussion-based methods [107,132] incorporate peer ranking and discussion to improve evaluation accuracy.Instead of relying solely on a single LLM's judgment, these methods prompt multiple LLMs to compare answers and discuss preferences to reach a consensus, thereby reducing individual positional bias and enhancing alignment with human judgments.”

• cites [107]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [132]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Position bias is a prevalent issue not only in the context of LLMs-as-judges but also in human decision-making and across various machine learning domains.

“Position bias is a prevalent issue not only in the context of LLMs-as-judges but also in human decision-making and across various machine learning domains.”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [182]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [197]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [288]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [111]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [234]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Distraction bias highlights the vulnerability of LLMs to attentional diversion caused by inconsequential content.

“This bias highlights the vulnerability of LLMs to attentional diversion caused by inconsequential content.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the comparison-based method, if the LLM's judgments are inconsistent after swapping, the outcome is considered a tie.

“In contrast, the comparison-based method considers the outcome a tie if the LLM's judgments are inconsistent after swapping.”

• cites [151]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Chatbot Arena offers a user-driven evaluation framework where participants interact with anonymous models and vote based on their preferences, using the Bradley-Terry model to rank LLMs and chatbots.

“Chatbot Arena, on the other hand, offers a user-driven evaluation framework where participants interact with anonymous models and vote based on their preferences.The platform has collected over 1,000,000 user votes, using the Bradley-Terry model to rank LLMs and chatbots, providing valuable insights into user preferences and model performance in open-domain dialogue.”

• cites [292]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

WildBench collects challenging examples from real users via the AI2 WildChat project, providing fine-grained annotations, task types, and checklists for response quality evaluation, and employs length-penalized Elo ratings.

“WildBench [138] collects challenging examples from real users via the AI2 WildChat project, providing fine-grained annotations, task types, and checklists for response quality evaluation, and employs length-penalized Elo ratings to ensure unbiased assessments.”

• cites [138]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Social-Related Biases refer to biases in language models that resemble social phenomena.

“Social-Related Biases refer to biases in language models that resemble social phenomena [289].”

• cites [289]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Contextual Bias refers to the tendency of LLMs to produce skewed or biased judgments based on the specific context in which they are applied.

“Contextual Bias refers to the tendency of LLMs to produce skewed or biased judgments based on the specific context in which they are applied.”

• cites [179]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [300]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.