Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

May combine two claims.

Many benchmarks failed to measure the complex, evolving capabilities of LLMs in real-world settings or address the risks of LLMs gaming and overfitting them.

“As a result, many benchmarks failed to measure the complex, evolving capabilities of LLMs in real-world settings or address the risks of LLMs gaming and overfitting them.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Preliminary observations indicated systemic inadequacies in most current LLM benchmarks, where superficial metrics often replaced comprehensive evaluations of LLM functionality and the assurance of benchmark integrity.

“Our preliminary observations indicated systemic inadequacies in most current LLM benchmarks, where superficial metrics often replaced comprehensive evaluations of LLM functionality and the assurance of benchmark integrity.”

May combine two claims.

Many benchmarks predominantly focus on English, with limited representation of other languages, such as Simplified Chinese, and frequently neglect cultural and contextual variations in their questions and answers.

“A key observation is that many benchmarks predominantly focus on English, with limited representation of other languages, such as Simplified Chinese, and frequently neglect cultural and contextual variations in their questions and answers.”

May combine two claims.

Current benchmarks generally adopt task-based formats, such as Multiple Choice Questions (MCQs) and dialogue-based evaluations, which tend to be static and do not capture the evolving nature of human-AI interactions.

“Another critical issue is that current benchmarks generally adopt task-based formats, such as Multiple Choice Questions (MCQs) and dialogue-based evaluations, which tend to be static and do not capture the evolving nature of human-AI interactions.”

May combine two claims.

Benchmarks often assume singular correct answers in culturally nuanced questions, overlooking alternative valid responses from different cultural or religious perspectives, which limits the inclusivity of these evaluations.

“For instance, benchmarks often assume singular correct answers in culturally nuanced questions, overlooking alternative valid responses from different cultural or religious perspectives, which limits the inclusivity of these evaluations.”

May combine two claims.

Variations in prompt formulation can lead to significant differences in model performance, complicating the interpretation of benchmark results.

“Challenges can arise when variations in prompt formulation lead to significant differences in model performance, which can complicate the interpretation of benchmark results”

May combine two claims.

The MMLU benchmark reported performance variations among different LLMs, particularly GPT-3 struggled with tasks requiring detailed procedural knowledge or calculations, indicating potential inadequacy in capturing specialized performance.

“MMLU [39]: The study designed a benchmark that focused on a wide range of tasks across different fields, like Professional Law and College Medicine.However, it reported performance variations among different LLMs, particularly GPT-3, which struggled with tasks requiring detailed procedural knowledge or calculations, like Elementary Mathematics.This indicates the benchmark's potential inadequacy in”

• cites [39] Wei, 2023, Jailbroken: How does LLM safety training fail?, Proc. 37th Conf. Neural Inf. Process. Syst.Not checked yet

  Not checked in the free run.

May combine two claims.

The MMLU benchmark highlighted the challenge of discerning genuine reasoning from technical optimization in LLMs, as evidenced by varied performance across tasks and reliance on pretraining, without a clear mechanism to ensure genuine understanding or reasoning capabilities.

“MMLU [39]: The study highlighted the challenge of discerning genuine reasoning from technical optimization in LLMs, as evidenced by the varied performance across tasks and the reliance on pretraining, without a clear mechanism to ensure genuine understanding or reasoning capabilities.”

• cites [39] Wei, 2023, Jailbroken: How does LLM safety training fail?, Proc. 37th Conf. Neural Inf. Process. Syst.Not checked yet

  Not checked in the free run.

May combine two claims.

The MultiMedQA benchmark applied standardized benchmarks to Flan-PaLM, a model not exclusively trained on medical data, failing to fully capture Flan-PaLM's specialized performance in the medical context, as evidenced by its limited alignment with clinical consensus compared to clinician-generated answers.

“MultiMedQA [10]: The study applied standardized benchmarks (MultiMedQA) to Flan-PaLM, a model not exclusively trained on medical data.This approach failed to fully capture Flan-PaLM's specialized performance in the medical context, as evidenced by its limited alignment with clinical consensus compared to clinician-generated answers.”

• cites [10] Srivastava, 2023, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Trans. on Mach. Learn. Res.Not checked yet

  Not checked in the free run.

May combine two claims.

The HumanEval benchmark relied on automatic unit tests to evaluate functional correctness of code generated by LLMs, which did not directly address whether the model's solutions stemmed from genuine understanding or were merely optimized for the test scenarios.

“HumanEval [8]: The study relied on automatic unit tests to evaluate the functional correctness of code generated by LLMs, which did not directly address whether the model's solutions stemmed from genuine understanding, or were merely optimized for the test scenarios, thus not fully assessing genuine reasoning capabilities.”

• cites [8] Liang, 2022, Holistic evaluation of language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

The PromptBench benchmark demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.

“PromptBench [7]: The study demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.”

• cites [7] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The HELM benchmark included targeted evaluations and multi-metric measurement across various scenarios, but did not specifically address the tension between helpfulness and harmlessness in its criteria or methodologies, indicating an unresolved challenge in balancing these aspects.

“HELM [31]: The HELM evaluation framework included targeted evaluations and multi-metric measurement across various scenarios, but did not specifically address the tension between helpfulness and harmlessness in its criteria or methodologies, indicating an unresolved challenge in balancing these aspects within the”

• cites [31] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The MultiMedQA benchmark introduced a sophisticated approach for evaluating LLMs in medical question answering, incorporating human evaluation frameworks to assess balance between helpfulness and harmlessness; however, it recognized the complexity in achieving this balance due to intricate nature of medical knowledge and societal values, without presenting a resolved methodology to adequately

“MultiMedQA [10]: The study introduced a sophisticated approach for evaluating LLMs in medical question answering, incorporating human evaluation frameworks to assess the balance between helpfulness and harmlessness; however, it recognized the complexity in achieving this balance due to the intricate nature of medical knowledge and societal values, without presenting a resolved methodology to”

• cites [10] Srivastava, 2023, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Trans. on Mach. Learn. Res.Not checked yet

  Not checked in the free run.

The quote only partly states this claim.

Genuine reasoning vs technical optimization is an inadequacy in 22 out of 23 surveyed LLM benchmarks.

“Prevalence: 22/23”

May combine two claims.

The SciBench benchmark demonstrated challenges in evaluating true reasoning capabilities of LLMs due to reliance on technical optimization, as seen when LLMs, prompted with external tools, incorrectly derived equations, highlighting their struggle to understand mathematical relationships beyond mere optimization.

“SciBench [46]: The study demonstrated challenges in evaluating the true reasoning capabilities of LLMs due to reliance on technical optimization, as seen when LLMs, prompted with external tools, incorrectly derived equations, highlighting their struggle to understand mathematical relationships beyond mere optimization.”

• cites [46] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The PromptBench benchmark acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.

“PromptBench [7]: The study acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.”

• cites [7] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The AgentBench benchmark introduced AgentBench to evaluate LLMs as agents across diverse real-world scenarios, but faced the challenge of response variability due to standardized evaluations not fully capturing specialized, context-driven response behavior, as evidenced by significant performance disparity between commercial and open-source LLMs across different tasks.

“AgentBench [49]: The study introduced AgentBench to evaluate LLMs as agents across diverse real-world scenarios, but it inherently faced the challenge of response variability due to standardized evaluations not fully capturing the specialized, context-driven response behavior of LLMs tailored for specific scenarios, as evidenced by the significant performance disparity between commercial and open”

• cites [49] Xu, 2023, On the tool manipulation capability of open-sourced large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

The HumanEval benchmark evaluated Codex's performance mainly through functional correctness using a unique set of hand-written programming problems, which might not fully capture the model's behavior in diverse real-world scenarios.

“HumanEval [8]: The study demonstrated inadequacy in response variability as it evaluated Codex's performance mainly through functional correctness, using a unique set of hand-written programming problems (HumanEval), which might not fully capture the model's behavior in diverse real-world scenarios.”

• cites [8] Liang, 2022, Holistic evaluation of language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

The Chain-of-Thought Hub benchmark used final answer accuracy as a proxy for reasoning capability without considering the correctness of intermediate steps, potentially failing to differentiate genuine reasoning from technical optimization.

“Chain-of-Thought Hub [40]: The study used final answer accuracy as a proxy for reasoning capability without considering the correctness of intermediate steps, potentially failing to differentiate genuine reasoning from technical optimization.”

• cites [40] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The HELM benchmark acknowledged the challenge of standardizing language model evaluation across diverse scenarios, indicating an understanding of response variability but not fully addressing it through its methodology.

“HELM [31]: The study acknowledged the challenge of standardizing language model evaluation across diverse scenarios, indicating an understanding of response variability but not fully addressing it through its methodology.”

• cites [31] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The PromptBench benchmark did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.

“PromptBench [7]: The study did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on the robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.”

• cites [7] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The predominance of English and Simplified Chinese (overlooking Traditional Chinese differences) in LLM benchmarks frequently reflected the default values and beliefs of their creators, overlooking the pluralistic nature of global beliefs and practices.

“The predominance of English and Simplified Chinese (overlooking Traditional Chinese differences) in those benchmarks frequently reflected the default values and beliefs of their creators, overlooking the pluralistic nature of global beliefs and practices.”

May combine two claims.

The HumanEval benchmark did not address the tension between helpfulness and harmlessness in LLM benchmarks; its methodology primarily centered on functional correctness through unit tests and did not incorporate a framework to evaluate or balance helpfulness versus harmlessness of generated code.

“HumanEval [8]: The study focused on evaluating LLMs trained on code, specifically assessing their ability to generate standalone Python functions from docstrings.However, it did not address the tension between helpfulness and harmlessness in LLM benchmarks.The methodology primarily centered on functional correctness through unit tests and did not incorporate a framework to evaluate or balance the”

• cites [8] Liang, 2022, Holistic evaluation of language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

The AgentBench benchmark extensively evaluated LLMs across various tasks and environments, but did not establish a clear methodology or criteria for balancing provision of useful information with prevention of harmful outputs, indicating a potential oversight in considering the complex and subtle balance between those aspects in real-world applications.

“AgentBench [49]: The study extensively evaluated LLMs across various tasks and environments, but did not establish a clear methodology or criteria for balancing the provision of useful information with the prevention of harmful outputs, as seen in their evaluation setup and performance analysis sections.This gap indicated a potential oversight in considering the complex and subtle balance between”

• cites [49] Xu, 2023, On the tool manipulation capability of open-sourced large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

HaluEval focused on evaluating LLMs' ability to recognize hallucinations across different tasks without addressing linguistic diversity or embedded logic specific to non-English languages.

“The study focused on evaluating LLMs' ability to recognize hallucinations across different tasks, without addressing the linguistic diversity or embedded logic specific to non-English languages.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “The benchmark did not acknowledge or attempt to address this inadequacy, relying instead on predominantly English-centric data and evaluation criteria.”

• cites [52] Qin, 2023, ToolLLM: Facilitating large language models to master 16000+ real-world APISNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Xiezhi required considerable effort to adapt and scale for evaluating various LLMs, as evidenced by the detailed description of its complex setup and evaluation processes.

“The study utilized the Xiezhi benchmark, which required considerable effort to adapt and scale for evaluating various LLMs, as evidenced by the detailed description of its complex setup and evaluation processes.”

• cites [42] Fu, 2023, Chain-of-thought hub: a continuous effort to measure large language models’ reasoning performanceNot checked yet

  Not checked in the free run.

May combine two claims.

M3KE required significant engineering efforts for setting up and assessing a wide range of Chinese LLMs, without specifically addressing the complexity and scalability challenges of the benchmark's installation and operational efficiency.

“The study introduced M3KE, requiring significant engineering efforts for setting up and assessing a wide range of Chinese LLMs, without specifically addressing the complexity and scalability challenges of the benchmark's installation and operational efficiency.”

• cites [9] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

KoLA acknowledged the inadequacy by incorporating evolving data sources including non-English content, but did not explicitly address or attempt to resolve the benchmark inadequacy of ignoring linguistic differences and embedded logics.

“The study acknowledged the inadequacy by incorporating evolving data sources, including non-English content like news articles and novels, to evaluate models' understanding and generation abilities in diverse linguistic contexts.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “However, it did not explicitly address or attempt to resolve the benchmark inadequacy of ignoring linguistic differences and embedded logics in different languages.”

• cites [11] Guha, 2024, Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, Proc. Adv. Neural Inf. Process. Syst.Not checked yet

  Not checked in the free run.

May combine two claims.

FLUE focused on the financial domain, utilizing primarily English datasets without addressing or incorporating multilingual contexts or diverse cognitive frameworks.

“The study focused on developing and evaluating language models specifically for the financial domain, utilizing primarily English datasets without addressing or incorporating multilingual contexts or the diverse cognitive frameworks shaped by different languages' unique structures.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “There was no mention of efforts to include or evaluate the models' performance across various languages or to consider linguistic diversity explicitly.”

• cites [44] sourceNot checked yet

  Not checked in the free run.

May combine two claims. The quote only partly states this claim.

Xiezhi constructed a benchmark focusing on domain knowledge evaluation without explicitly addressing linguistic diversity or embedded logics; it predominantly utilized data from Simplified Chinese (erroneously labeling them as 'Chinese' without differentiation with Traditional Chinese) educational systems and academic surveys, with no clear mention of adjustments for linguistic differences beyond

“The study constructed the Xiezhi benchmark focusing on domain knowledge evaluation across a wide array of disciplines without explicitly addressing linguistic diversity or the embedded logics in different languages.”

• cites [42] Fu, 2023, Chain-of-thought hub: a continuous effort to measure large language models’ reasoning performanceNot checked yet

  Not checked in the free run.

May combine two claims.

Chain-of-Thought Hub included bilingual reasoning in English and Simplified Chinese (erroneously labeling them as 'Chinese' without differentiation with Traditional Chinese), but did not address linguistic differences and embedded logics across different languages beyond this.

“The benchmark study did not acknowledge this inadequacy or make attempts to address it.”

• cites [40] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

C-Eval explicitly focused on evaluating LLMs in a Simplified Chinese (erroneously labeling them as 'Chinese' without differentiation with Traditional Chinese) context, without addressing or attempting to include diverse linguistic backgrounds or cognitive frameworks.

“The benchmark did not acknowledge or address this inadequacy.”

• cites [43] Sawada, 2023, ARB: Advanced reasoning benchmark for large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

LegalBench focused on American legal reasoning and did not address linguistic differences or embedded logics in different languages; it utilized tasks primarily in Natural English without considering multilingual contexts.

“The study focused on developing and evaluating benchmarks within the context of American legal reasoning and did not address linguistic differences or embedded logics in different languages.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “The benchmark study did not acknowledge this specific inadequacy or make attempts to address it.”

• cites [14] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

ToolBench focused on integrating LLMs with real-world APIs without explicitly addressing linguistic diversity or embedded logics; its methodology was primarily based on automatic construction using ChatGPT, which follows a predominantly English-centric approach.

“The study focused on integrating LLMs with a vast number of real-world APIs without explicitly addressing the challenge of linguistic diversity or the embedded logics in different languages.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “The methodology was primarily based on automatic construction using ChatGPT, which inherently follows a predominantly English-centric approach due to its training data and design, without adapting to or considering the unique cognitive and logical frameworks that different languages might embody.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “The study did not acknowledge this specific inadequacy or attempt to address it.”

• cites [48] Liu, 2023, M3KE: A massive multi-level multi-subject knowledge evaluation benchmark for Chinese large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

AGIEval utilized a bilingual (English and Simplified Chinese, erroneously labeling them as 'Chinese') approach without specifically addressing or accounting for the unique cognitive and logical frameworks shaped by each language's structure.

“While the study acknowledged the inclusion of tasks in both English and Chinese, it did not discuss efforts to address the unique systems of reasoning fostered by different linguistic structures, suggesting a potential overlook of the linguistic diversity and embedded logic inadequacy.”

• cites [12] Yu, 2023, KOLA: Carefully benchmarking world knowledge of large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

T-Bench encountered substantial difficulty in effectively scaling benchmarks for open-source LLMs, noting significant performance disparities between open-source models and proprietary ones like GPT-4 in tool manipulation tasks.

“The study encountered substantial difficulty in effectively scaling benchmarks for open-source LLMs.For instance, they noted significant performance disparities between open-source models and proprietary ones like GPT-4 in tool manipulation tasks, necessitating enhanced techniques and infrastructure to bridge this gap.”

• cites [45] Huang, 2023, C-Eval: A multi-level multi-discipline Chinese evaluation suite for foundation modelsNot checked yet

  Not checked in the free run.

May combine two claims.

HaluEval did not address the tension between helpfulness and harmlessness; it focused on generating and annotating hallucinated responses without directly tackling the balance between providing useful information and avoiding harmful outputs.

“The study introduced HaluEval, a benchmark for evaluating LLMs' performance in recognizing hallucinations, which inherently did not address the tension between helpfulness and harmlessness.It focused on generating and annotating hallucinated responses to assess LLMs' ability to recognize hallucinations, without directly tackling the balance between providing useful information and avoiding harmful”

• cites [52] Qin, 2023, ToolLLM: Facilitating large language models to master 16000+ real-world APISNot checked yet

  Not checked in the free run.

May combine two claims.

MultiMedQA's benchmarks primarily focused on English-language datasets and did not address the need for multilingual evaluations.

“The study's benchmarks primarily focused on English-language datasets and did not address the need for multilingual evaluations, indicating a disregard for linguistic differences and embedded logics in different languages.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “While the benchmark study, Multi-MedQA, was praised for its diversity and inclusion of various medical exam, research, and consumer sources, the study acknowledged its limitation by not incorporating a broader variety of languages and cultural contexts, thereby not addressing the mentioned benchmark inadequacy.”

• cites [10] Srivastava, 2023, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Trans. on Mach. Learn. Res.Not checked yet

  Not checked in the free run.

May combine two claims.

HELM acknowledged the challenge of accounting for linguistic diversity but did not provide a solution, focusing on English and limited consideration of other languages.

“The study acknowledged the challenge of accounting for linguistic diversity in benchmarks, but did not provide a solution, exemplified by its focus on English and limited consideration of other languages, which indicates an unaddressed and unresolved inadequacy regarding linguistic differences and embedded logics.”

• cites [31] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

T-Bench focused on enhancing open-source LLMs for tool manipulation without addressing the incorporation or evaluation of multilingual capabilities or considering linguistic diversity.

“The study focused on enhancing opensource LLMs for tool manipulation without addressing the incorporation or evaluation of multilingual capabilities or considering linguistic diversity in its benchmarking process.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “The benchmark study, ToolBench, did not acknowledge or attempt to address this particular inadequacy of LLM benchmarks.”

• cites [45] Huang, 2023, C-Eval: A multi-level multi-discipline Chinese evaluation suite for foundation modelsNot checked yet

  Not checked in the free run.

May combine two claims.

APIBank was primarily focused on English and did not address the construction and evaluation of models for other languages.

“The study's benchmark, API-Bank, was primarily focused on English and did not address the construction and evaluation of models for other languages, indicating a disregard for linguistic differences and embedded logics in different languages.”

• Also stated in: § D. Linguistic Variability and Embedded Logic Diversity · plain

  “The benchmark study acknowledged this limitation and mentioned plans to address data construction and model evaluation for other languages as future work.”

• cites [50] Wang, 2023, SciBench: Evaluating college-level scientific problem-solving abilities of large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

ARB introduced a rubric-based evaluation method for advanced reasoning tasks and tested LLMs, requiring human evaluators for complex symbolic answers and proof-like questions, indicating prolonged evaluation time; it did not specifically address measures to significantly reduce test iteration time.

“The study introduced a rubric-based evaluation method for advanced reasoning tasks and tested LLMs, including GPT-4, on a diverse set of problems, requiring human evaluators for complex symbolic answers and proof-like questions, indicating prolonged evaluation time.While it acknowledged the challenges in automating the evaluation of advanced reasoning tasks, it did not specifically address”

• cites [41] Hendrycks, 2020, Measuring massive multitask language understandingNot checked yet

  Not checked in the free run.

May combine two claims.

FLUE introduced the Financial Language Understanding Evaluation (FLUE) and a new financial language model (FLANG), focusing on domain-specific pre-training and evaluation without explicitly addressing or attempting to standardize the implementation process across different teams or laboratories.

“The study introduced the Financial Language Understanding Evaluation (FLUE) and a new financial language model (FLANG), focusing on domain-specific pre-training and evaluation without explicitly addressing or attempting to standardize the implementation process across different teams or laboratories, which could lead to inconsistent benchmark applications and results.”

• Also stated in: § A. Inconsistent Benchmark Implementation · plain

  “The study did not mention any measures to address this benchmark inadequacy directly.”

• cites [44] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

MMLU developed a new benchmark to evaluate LLMs across diverse subjects and tested them in zero-shot and few-shot settings, but did not directly address the uniformity and objectivity in benchmark implementation across different laboratories.

“The study developed a new benchmark to evaluate LLMs across a diverse set of subjects and tested them in zero-shot and few-shot settings, indicating an innovative approach yet not directly addressing the uniformity and objectivity in benchmark implementation across different laboratories.”

• cites [39] Wei, 2023, Jailbroken: How does LLM safety training fail?, Proc. 37th Conf. Neural Inf. Process. Syst.Not checked yet

  Not checked in the free run.

May combine two claims.

BIG-Bench utilized generative AI models to assess other LLMs, inheriting biases and inaccuracies from the models used to generate evaluations, potentially compromising objectivity and reliability.

“The benchmark utilized generative AI models to assess other LLMs, inheriting the biases and inaccuracies from the models used to generate evaluations, thus potentially compromising the objectivity and reliability of these assessments.”

• cites [6] Zhu, 2024, Promptbench: A unified library for evaluation of large language models, J. of Mach. Learn. Res.Not checked yet

  Not checked in the free run.

May combine two claims.

Xiezhi introduced the Xiezhi benchmark aiming to establish a comprehensive and multidisciplinary auto-updating benchmark, but does not explicitly discuss measures taken to standardize benchmark implementation across different research teams or laboratories.

“The study introduced the Xiezhi benchmark, which aimed to establish a comprehensive and multidisciplinary auto-updating benchmark.Despite its efforts to address inconsistencies by incorporating a broad range of disciplines and updating content, the study does not explicitly discuss measures taken to standardize benchmark implementation across different research teams or laboratories to ensure”

• cites [42] Fu, 2023, Chain-of-thought hub: a continuous effort to measure large language models’ reasoning performanceNot checked yet

  Not checked in the free run.

May combine two claims.

Chain-of-Thought Hub integrated diverse benchmarks for evaluating LLM reasoning, but did not address implementation inconsistencies, focusing instead on model performance comparison.

“The study integrated diverse benchmarks (e.g., GSM8k, MATH, MMLU) for evaluating LLM reasoning, but did not address implementation inconsistencies, focusing instead on model performance comparison.There was no mention of efforts to standardize benchmark execution or address potential inconsistencies across different research teams.”

• cites [40] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

HELM acknowledged the challenge of ensuring uniform evaluation methods across all models, highlighting ongoing efforts to refine and adapt benchmarking practices, without explicitly stating that this inadequacy has been fully resolved.

“The study acknowledged the challenge of ensuring uniform evaluation methods across all models, highlighting ongoing efforts to refine and adapt benchmarking practices to maintain relevancy and accuracy in assessments, without explicitly stating that this inadequacy has been fully resolved.”

• cites [31] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

LegalBench introduced LegalBench with diverse legal reasoning tasks and conducted preliminary evaluations, but did not address the inconsistency in benchmark implementation directly.

“The study introduced LegalBench with an emphasis on diverse legal reasoning tasks and conducted preliminary evaluations on different models, yet it did not address the inconsistency in benchmark implementation directly.”

• Also stated in: § A. Inconsistent Benchmark Implementation · hedged

  “Although the study aimed to foster standardized evaluations through its collaborative and open design, it implicitly acknowledged the challenge of ensuring uniformity across implementations by inviting community contributions to refine and expand the benchmark suite.”

• cites [14] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

KoLA established a new benchmark designed to evaluate LLMs across various knowledge-intensive tasks and introduced a contrastive evaluation system aimed at ensuring fairness and applicability, but did not specifically address the inconsistency in benchmark execution across different laboratories.

“The study established a new benchmark, KoLA, designed to evaluate LLMs across various knowledge-intensive tasks using both known and evolving data sources, and introduced a contrastive evaluation system aimed at ensuring fairness and applicability in LLM assessment.However, it did not specifically address the inconsistency in benchmark execution across different laboratories, focusing instead on”

• Also stated in: § A. Inconsistent Benchmark Implementation · plain

  “The benchmark study acknowledged the need for better fairness and applicability in LLM evaluations but did not make explicit attempts to address the inconsistency in execution across different research teams.”

• cites [11] Guha, 2024, Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, Proc. Adv. Neural Inf. Process. Syst.Not checked yet

  Not checked in the free run.

May combine two claims.

HaluEval utilized ChatGPT to automatically generate hallucinated samples, introducing inconsistency in benchmark implementation due to reliance on a single LLM's capacity, potentially leading to variability in quality; the study acknowledged this limitation but did not propose specific measures to address it.

“The study utilized ChatGPT to automatically generate hallucinated samples, which is subject to the model's capacity to follow complex instructions for hallucination sampling.Their approach introduced inconsistency in benchmark implementation due to the reliance on a single LLM's capacity, potentially leading to variability in the quality of generated samples.The study acknowledged this limitation,”

• cites [52] Qin, 2023, ToolLLM: Facilitating large language models to master 16000+ real-world APISNot checked yet

  Not checked in the free run.

May combine two claims.

APIBank utilized generative AI models to automatically mass-produce training data, which likely inherited biases from the models used, compromising the objectivity and reliability of evaluations.

“The study utilized generative AI models to automatically mass-produce training data, which likely inherited biases from the models used, compromising the objectivity and reliability of evaluations.”

• cites [50] Wang, 2023, SciBench: Evaluating college-level scientific problem-solving abilities of large language modelsNot checked yet

  Not checked in the free run.

The quote only partly states this claim.

18 out of 23 benchmarks exhibit the slow test iteration time inadequacy.

“Prevalence: 18/23”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ARB introduced a novel benchmark to evaluate LLMs using advanced reasoning problems, yet acknowledged the challenge of ensuring consistent and reliable evaluation methods, particularly for symbolic and proof-like problems, without directly addressing the inconsistency in benchmark implementation across different labs.

“The study introduced a novel benchmark, ARB, to evaluate LLMs using advanced reasoning problems, yet acknowledged the challenge of ensuring consistent and reliable evaluation methods, particularly for symbolic and proof-like problems, without directly addressing the inconsistency in benchmark implementation across different labs.”

• cites [41] Hendrycks, 2020, Measuring massive multitask language understandingNot checked yet

  Not checked in the free run.

May combine two claims.

T-Bench utilized generative AI models to enhance open-source LLMs for tool manipulation, explicitly involving model-generated evaluations through programmatic data generation and in-context demonstration retrievers, which could inherit biases from the models used to generate them.

“The study utilized generative AI models to enhance open-source LLMs for tool manipulation, explicitly involving model-generated evaluations through programmatic data generation and in-context demonstration retrievers, which could inherit biases from the models used to generate them.”

• cites [45] Huang, 2023, C-Eval: A multi-level multi-discipline Chinese evaluation suite for foundation modelsNot checked yet

  Not checked in the free run.

May combine two claims.

MultiMedQA utilized a benchmark combining multiple datasets and evaluated models with human and automated methods, which inherently extends the evaluation timeframe, but did not explicitly mention efforts to address or mitigate slow iteration time.

“The study utilized a benchmark combining multiple datasets and evaluated models with human and automated methods, which inherently extends the evaluation timeframe, but did not explicitly mention efforts to address or mitigate slow iteration time.”

• cites [10] Srivastava, 2023, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Trans. on Mach. Learn. Res.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

MultiMedQA aimed to overcome limitations of existing benchmarks by combining six medical question answering datasets and a new dataset, HealthSearchQA, but recognized the challenge of ensuring consistent benchmark application across diverse medical domains without directly addressing a solution.

“The study introduced MultiMedQA, aiming to overcome limitations of existing benchmarks by combining six medical question answering datasets and a new dataset, HealthSearchQA, for comprehensive LLM evaluation.However, it recognized the challenge of ensuring consistent benchmark application across diverse medical domains, acknowledging the difficulty in uniform implementation without directly”

• cites [10] Srivastava, 2023, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Trans. on Mach. Learn. Res.Not checked yet

  Not checked in the free run.

May combine two claims.

SciBench introduced a benchmark requiring detailed solutions and complex reasoning, involving manual LaTeX formatting from PDFs; despite efforts to mitigate slow iteration time, extensive manual data processing indicated potential slow test iteration time.

“The study introduced SciBench, a benchmark requiring detailed solutions and complex reasoning, involving manual LaTeX formatting from PDFs to ensure data integrity and minimize training data leakage.Despite efforts to mitigate slow iteration time through detailed problem and solution documentation, the study's extensive and manual data processing indicated potential slow test iteration time for”

• cites [46] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Chain-of-Thought Hub aimed to evaluate LLMs' reasoning capabilities but did not specifically address slow test iteration time; it required evaluating new models across multiple complex reasoning tasks, likely extending the evaluation period significantly.

“The study introduced the Chain-of-Thought Hub, which aimed to evaluate LLMs' reasoning capabilities, but did not specifically address the slow test iteration time.This benchmark required evaluating new models across multiple complex reasoning tasks, which likely extended the evaluation period significantly, similar to the slow iteration time observed in benchmarks like BIG-bench and HELM.However,”

• cites [40] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

ToolAlpaca utilized generative AI models to automatically generate structured documentation for APIs, which were then used to simulate tool-use instances for training LLMs, inherently incorporating biases from the generative models used.

“The study utilized generative AI models to automatically generate structured documentation for APIs, which were then used to simulate tool-use instances for training LLMs, inherently incorporating biases from the generative models used.”

• cites [47] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

AgentBench utilized generative AI models to create novel tasks and evaluate LLMs, inheriting biases from the models used for generation, as evidenced by the use of model-generated tasks in various environments without addressing the potential for inherited biases.

“The study utilized generative AI models to create novel tasks and evaluate LLMs, inheriting biases from the models used for generation, as evidenced by the use of model-generated tasks in various environments without addressing the potential for inherited biases.”

• cites [49] Xu, 2023, On the tool manipulation capability of open-sourced large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

PromptBench did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and did not attempt to standardize or ensure uniform implementation methodologies.

“The study did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and it did not attempt to standardize or ensure uniform implementation methodologies for these prompts.”

• cites [7] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

AGIEval employed a novel benchmark designed to evaluate foundation models on human-centric tasks, but did not address the issue of implementation inconsistency across different laboratories; it did not explicitly mention measures to ensure uniform implementation.

“The study employed a novel benchmark, AGIEval, which was specifically designed to evaluate foundation models on human-centric tasks derived from high-standard admission and qualification exams.The benchmarks aimed to assess models' general abilities in tasks related to human cognition and problem-solving.However, the study did not address the issue of implementation inconsistency across different”

• cites [12] Yu, 2023, KOLA: Carefully benchmarking world knowledge of large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

HumanEval acknowledged the potential for diverse outcomes from varying implementation methods, but did not specifically address or propose solutions to standardize benchmark execution across different research teams.

“The study acknowledged the potential for diverse outcomes from varying implementation methods, but did not specifically address or propose solutions to standardize benchmark execution across different research teams, focusing instead on the development and evaluation of models using their own benchmarks and methodologies.”

• cites [8] Liang, 2022, Holistic evaluation of language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

SciBench introduced SciBench to systematically examine complex scientific problem-solving capabilities of LLMs, but encountered inconsistencies in benchmark implementation; it acknowledged the challenge of ensuring uniform assessment across different LLM configurations without detailing explicit measures to address these inconsistencies directly.

“The study introduced SciBench to systematically examine complex scientific problem-solving capabilities of LLMs, but it encountered inconsistencies in benchmark implementation.The benchmarks, derived from collegiate-level textbooks and exams, required advanced computations like integration and differentiation, which varied in execution complexity.Despite efforts to minimize data leakage and use”

• cites [46] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

KoLA introduced a benchmark that requires manual annotation for evolving tasks, which was time-intensive and cannot be automated, leading to potential delays in evaluating LLMs; it did not address the need for quicker iteration time.

“The study introduced a benchmark that requires manual annotation for evolving tasks, which was time-intensive and cannot be automated, leading to potential delays in evaluating LLMs akin to those seen in frameworks like BIG-bench and HELM.The benchmark did not address the need for quicker iteration time, despite mentioning the evolving nature of data and tasks.”

• cites [11] Guha, 2024, Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, Proc. Adv. Neural Inf. Process. Syst.Not checked yet

  Not checked in the free run.

May combine two claims.

ToolAlpaca developed a novel framework to automatically generate a diverse tool-use corpus and enhance compact language models' generalized tool-use abilities, without specifically addressing or resolving the inconsistency in benchmark implementation across different laboratories.

“The study developed a novel framework, ToolAlpaca, to automatically generate a diverse tool-use corpus and enhance compact language models' generalized tool-use abilities, without specifically addressing or resolving the inconsistency in benchmark implementation across different laboratories.The benchmark study did not acknowledge this inadequacy of LLM benchmarks nor made attempts to address it.”

• cites [47] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

ToolBench introduced innovative methods like DFSDT and developed an automatic evaluator, ToolEval, but primarily focused on enhancing model performance and generalization rather than standardizing benchmark implementation procedures.

“While the study introduced innovative methods like DFSDT to improve LLM's planning and reasoning abilities and developed an automatic evaluator, ToolEval, to assess LLM's tool-use capabilities, it primarily focused on enhancing model performance and generalization rather than standardizing benchmark implementation procedures.The benchmark study acknowledged the complexity of evaluating LLMs in”

• cites [48] Liu, 2023, M3KE: A massive multi-level multi-subject knowledge evaluation benchmark for Chinese large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

AgentBench introduced a new benchmark designed to evaluate LLMs across various environments and extensively evaluated 27 LLMs, but did not directly address the issue of inconsistency in benchmark implementation across different laboratories.

“The study introduced AgentBench, a new benchmark designed to evaluate LLMs across a variety of environments, and extensively evaluated 27 LLMs, including both API-based commercial models and open-sourced LLMs.However, the study did not directly address the issue of inconsistency in benchmark implementation across different laboratories.Instead, it focused on establishing a new benchmark and”

• cites [49] Xu, 2023, On the tool manipulation capability of open-sourced large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

FLUE introduced benchmarks for evaluating LLMs in the financial domain, necessitating extensive testing and iterations across various scenarios, likely extending the time frame for benchmark completion.

“The study introduced benchmarks for evaluating Large Language Models in the financial domain, necessitating extensive testing and iterations across various scenarios, likely extending the time frame for benchmark completion.The study did not explicitly acknowledge this inadequacy or mention attempts to address it.”

• cites [44] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

T-Bench demonstrated that enhancing open-source LLMs with techniques like model alignment, system prompts, and in-context demonstration retrievers required a practical level of human supervision, indicating that evaluation using the ToolBench benchmark could not be fully automated and might be subject to prolonged iteration time.

“The study demonstrated that the process of enhancing open-source Large Language Models (LLMs) with techniques like model alignment, system prompts, and in-context demonstration retrievers required a practical level of human supervision, indicating that the evaluation of LLMs using the ToolBench benchmark could not be fully automated and might be subject to prolonged iteration time.The study”

• cites [45] Huang, 2023, C-Eval: A multi-level multi-discipline Chinese evaluation suite for foundation modelsNot checked yet

  Not checked in the free run.

May combine two claims.

BOLAA was designed to orchestrate multiple LLM-augmented autonomous agents and required extensive testing across various scenarios, inherently extending the timeframe for completion; it did not explicitly address or attempt to reduce slow iteration time.

“The study's benchmark, BOLAA, was designed to orchestrate multiple LLM-augmented autonomous agents, and required extensive testing across various scenarios, inherently extending the timeframe for completion.The benchmark did not explicitly address or attempt to reduce the slow iteration time associated with evaluating new models, making it susceptible to the described inadequacy.”

• cites [51] Tang, 2023, ToolAlpaca: Generalized tool learning for language models with 3000 simulated casesNot checked yet

  Not checked in the free run.

May combine two claims.

HELM implemented a comprehensive and systematic evaluation approach incorporating extensive metrics and scenarios, which inherently necessitated significant computational resources and time, likely extending iteration time; it did not specifically address or propose solutions to reduce slow test iteration time.

“The study implemented a comprehensive and systematic evaluation approach to benchmarking LLMs, incorporating extensive metrics and scenarios, which inherently necessitated significant computational resources and time for thorough evaluation, thus likely extending iteration time for model testing.The study, while acknowledging the broad and intricate scope of its benchmark, did not specifically”

• cites [31] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

LegalBench outlined challenges in prompt engineering, notably in crafting prompts that accurately assess LLMs' legal reasoning capabilities without introducing bias, thus impacting benchmark integrity; the study acknowledged the complexity of legal language and ongoing effort to refine evaluation techniques, indicating an attempt to address benchmark inadequacies but also highlighting the

“The study outlined challenges in prompt engineering, notably in crafting prompts that ac-curately assess LLMs' legal reasoning capabilities without introducing bias, thus impacting benchmark integrity.”

• Also stated in: § C. Challenge of Proper Prompt Engineering · hedged

  “The study acknowledged the complexity of legal language and the ongoing effort to refine evaluation techniques, indicating an attempt to address benchmark inadequacies but also highlighting the unresolved nature of prompt engineering challenges.”

• cites [14] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

HaluEval utilized a two-step process for generating hallucinated samples, involving a time-consuming manual annotation component; the study acknowledged the limitation of slow iteration time but did not present a solution to expedite the evaluation process significantly.

“The study utilized a two-step process, sampling-then-filtering, for generating hallucinated samples, which inherently involved a time-consuming, manual annotation component by human labelers.The study acknowledged the limitation of slow iteration time, due to the reliance on manual human annotation and complex instruction following by LLMs for quality control, but it did not present a solution to”

• cites [52] Qin, 2023, ToolLLM: Facilitating large language models to master 16000+ real-world APISNot checked yet

  Not checked in the free run.

May combine two claims.

AgentBench developed a comprehensive and systematic benchmark to evaluate LLMs as agents across a wide array of real-world challenges, requiring multi-turn interactions with detailed and complex environments, which inherently suggested a potentially time-consuming evaluation process; it did not specifically address the speed of iteration time as a limitation.

“The study developed a comprehensive and systematic benchmark, AgentBench, to evaluate LLMs as agents across a wide array of real-world challenges, requiring multi-turn interactions with detailed and complex environments.This inherently suggested a potentially time-consuming evaluation process, given the variety and depth of the tasks involved.Although the study provided an integrated toolkit to”

• cites [49] Xu, 2023, On the tool manipulation capability of open-sourced large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

FLUE benchmark development focused on creating assessments across various NLP tasks in the financial domain, but did not explicitly mention efforts to mitigate or acknowledge the intricacies involved in crafting unbiased and effective prompts to accurately gauge LLM capabilities.

“The benchmark development focused on creating assessments across various NLP tasks in the financial domain, but did not explicitly mention efforts to mitigate or acknowledge the intricacies involved in crafting unbiased and effective prompts to accurately gauge LLM capabilities.”

• cites [44] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

M3KE utilized a unified prompt for all models across different settings without detailing efforts to mitigate biases or inaccuracies in prompt formulation, impacting evaluation accuracy and potentially overlooking the complexity of assessing LLMs' true capabilities.

“The study utilized a unified prompt for all models across different settings without detailing efforts to mitigate biases or inaccuracies in prompt formulation, impacting the models' evaluation accuracy and potentially overlooking the complexity of assessing LLMs' true capabilities.The study did not explicitly acknowledge or address this benchmark inadequacy.”

• cites [9] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

MultiMedQA acknowledged the difficulty in creating prompts that accurately assess LLMs without introducing biases or misinterpretations, affecting both functionality and cybersecurity domains; it attempted to address this through instruction prompt tuning, but the challenge remains complex and unresolved.

“The study acknowledged the difficulty in creating prompts that accurately assess LLMs without introducing biases or misinterpretations, affecting both functionality and cybersecurity domains.It attempted to address this through instruction prompt tuning, aiming to align LLMs more closely with medical domain requirements, but the challenge remains complex and unresolved, indicating ongoing issues”

• cites [10] Srivastava, 2023, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Trans. on Mach. Learn. Res.Not checked yet

  Not checked in the free run.

May combine two claims.

PromptBench required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process; it did not specifically address or propose solutions to reduce iteration time.

“The study required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process, which aligned with the mentioned inadequacy of slow iteration time in benchmarks.The study did not specifically address or propose solutions to reduce these iteration time, suggesting that the challenge of aligning rapid AI development”

• cites [7] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

T-Bench developed ToolBench to evaluate open-source LLMs for tool manipulation tasks, employing techniques like model alignment, demonstration retrieval, and generation regulation with system prompts; however, it did not explicitly address the intricacies of crafting unbiased and effective prompts to accurately measure a model's capabilities.

“The study developed ToolBench, a benchmark to evaluate open-source LLMs for tool manipulation tasks, employing techniques like model alignment, demonstration retrieval, and generation regulation with system prompts.However, it did not explicitly address the intricacies of crafting unbiased and effective prompts to accurately measure a model's capabilities.The study acknowledged the importance of”

• cites [45] Huang, 2023, C-Eval: A multi-level multi-discipline Chinese evaluation suite for foundation modelsNot checked yet

  Not checked in the free run.

May combine two claims.

ToolAlpaca utilized a multi-agent simulation to generate a corpus for evaluating LLMs, a process that cannot be automated and likely extends over weeks or months due to manual intervention and complexity, directly mirroring the inadequacy of slow test iteration time.

“The study utilized a multi-agent simulation to generate a corpus for evaluating LLMs, a process that cannot be automated and likely extends over weeks or months due to manual intervention and complexity, directly mirroring the inadequacy of slow test iteration time.The study did not mention addressing this particular benchmark inadequacy.”

• cites [47] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

APIBank introduced a benchmark requiring manual annotation and extensive API integration, which inherently extended the evaluation time frame; although they employed a novel method to reduce annotation costs, the benchmark's design necessitated detailed and manual interventions, likely prolonging the overall process.

“The study introduced API-Bank, a benchmark that requires manual annotation and extensive API integration, which inherently extended the evaluation time frame, echoing the inadequacy of slow test iteration time.Although they employed a novel method to reduce annotation costs, the benchmark's design necessitated detailed and manual interventions, which likely prolong the overall process.The study”

• cites [50] Wang, 2023, SciBench: Evaluating college-level scientific problem-solving abilities of large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

C-Eval outlined the creation and evaluation of the C-EVAL suite, emphasizing rapid understanding and improvement of LLM capabilities, but did not specifically address or propose solutions for slow test iteration time inherent in comprehensive evaluations.

“The study outlined the creation and evaluation of the C-EVAL suite, emphasizing rapid understanding and improvement of LLM capabilities, but did not specifically address or propose solutions for slow test iteration time inherent in comprehensive evaluations like those mentioned for BIG-bench and HELM.This indicated that the process of using C-EVAL for LLM evaluation could indeed be lengthy and”

• cites [43] Sawada, 2023, ARB: Advanced reasoning benchmark for large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

AGIEval utilized a human-centric benchmark focusing on tasks derived from official public and high-standard exams; it did not explicitly address the potential for slow test iteration time due to comprehensive and manual evaluation, indicating the process could be time-consuming.

“The study utilized a human-centric benchmark, AGIEval, specifically designed for evaluating LLMs, focusing on tasks derived from official public and high-standard exams.Despite its innovative approach, the study did not explicitly address the potential for slow test iteration time due to the comprehensive and manual nature of its evaluation process, involving extensive human comparison and”

• cites [12] Yu, 2023, KOLA: Carefully benchmarking world knowledge of large language modelsNot checked yet

  Not checked in the free run.

May combine two claims.

MMLU designed the benchmark to assess models in zero-shot and few-shot settings across a wide range of subjects, focusing on evaluating knowledge acquired during pretraining without directly addressing the intricacies of prompt engineering; it mentioned the use of various prompts but did not scrutinize optimization of such prompts to avoid biases or misinterpretations.

“The study designed the benchmark to assess models in zero-shot and few-shot settings across a wide range of subjects, focusing on evaluating knowledge acquired during pretraining without directly addressing the intricacies of prompt engineering.It mentioned the use of various prompts, but did not scrutinize the optimization of such prompts to avoid biases or misinterpretations that could skew”

• cites [39] Wei, 2023, Jailbroken: How does LLM safety training fail?, Proc. 37th Conf. Neural Inf. Process. Syst.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Evaluating LLMs and generative AI requires a method that assesses not only standardized task performance but also real-world applicability and safety.

“Evaluating LLMs and generative AI requires a method that assesses not only standardized task performance but also real-world applicability and safety, as a lack of such holistic evaluations can lead to LLMs that perform well in controlled environments but fail in critical real-world applications, posing risks such as perpetuating bias, making unsafe decisions, or being vulnerable to manipulation [”

• cites [2] sourceNot checked yet

  Not checked in the free run.

• cites [3] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The study proposed a unified evaluation framework for LLM benchmarks based on the domains of people, process, and technology.

“We proposed a unified evaluation framework for LLM benchmarks, based on the domains of people, process, and technology, as a foundation to comprehensively and holistically assess both the functionality and integrity of LLMs in real-world applications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Benchmarking is a critical process in computer science, serving as a standardized method to evaluate and compare the performance of hardware and software systems.

“Benchmarking is a critical process in computer science, serving as a standardized method to evaluate and compare the performance of hardware and software systems [15].”

• cites [15] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

LLMs may overfit to benchmark datasets, memorizing answers rather than demonstrating true understanding, which undermines the validity of the evaluation.

“Additionally, LLMs may overfit to benchmark datasets, memorizing answers rather than demonstrating true understanding, which undermines the validity of the evaluation [33].”

• cites [33] Deng, 2023, Multilingual jailbreak challenges in large language modelsNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Leveraging the PPT framework ensures our evaluation encompasses all necessary facets to produce reliable and meaningful insights into LLM performance and benchmarking practices.

“Leveraging the PPT framework ensures our evaluation encompasses all necessary facets to produce reliable and meaningful insights into LLM performance and benchmarking practices.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Technical limitations can impede the benchmark's ability to provide accurate assessments, while ensuring the infrastructure's scalability is critical as LLMs evolve and grow more complex.

“Technical limitations can impede the benchmark's ability to provide accurate assessments, while ensuring the infrastructure's scalability is critical as LLMs evolve and grow more complex.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The success of an LLM benchmark is not only in its ability to test specific tasks but also in how well it adapts to new challenges, handles real-world use cases, and resists manipulation or overfitting.

“The success of an LLM benchmark is not only in its ability to test specific tasks but also in how well it adapts to new challenges, handles real-world use cases, and resists manipulation or overfitting.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Without well-defined processes, benchmarks may become outdated, inconsistent, or vulnerable to LLMs being optimized solely for specific test sets rather than demonstrating true capability.

“Without well-defined processes, benchmarks may become outdated, inconsistent, or vulnerable to LLMs being optimized solely for specific test sets rather than demonstrating true capability.”

May combine two claims.

Our previous survey on ransomware revealed concerns about the self-benchmarking practices in many anti-ransomware studies, which often used diverse ransomware samples and methodologies and based their claims of superiority merely on higher detection rates.

“Moreover, within the context of cybersecurity, our previous survey on ransomware revealed concerns about the self-benchmarking practices in many anti-ransomware studies, which often used diverse ransomware samples and methodologies and based their claims of superiority merely on higher detection rates [24].”

• cites [24] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Evaluating both functionality and integrity is essential to ensure that benchmarks provide a comprehensive and authentic assessment of LLMs, free from bias or manipulation.

“Evaluating both functionality and integrity is essential to ensure that benchmarks provide a comprehensive and authentic assessment of LLMs, free from bias or manipulation.”

Probably not a substantial claim: may not pass S2 (checkable).

The PPT framework aligns with the core challenges of LLM benchmarking.

“The PPT framework aligns with the core challenges of LLM benchmarking.”

Probably not a substantial claim: may not pass S4 (based).

A benchmark lacking integrity might be susceptible to LLMs that achieve high scores through memorization of training data or by exploiting specific test set characteristics.

“For example, a benchmark lacking integrity might be susceptible to LLMs that achieve high scores through memorization of training data or by exploiting specific test set characteristics.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Ensuring integrity means the benchmark cannot be easily manipulated and that it provides a true reflection of an LLM's performance, independent of prior exposure to similar tests or over-fitting strategies.

“Ensuring integrity means the benchmark cannot be easily manipulated and that it provides a true reflection of an LLM's performance, independent of prior exposure to similar tests or over-fitting strategies.”

May combine two claims.

The open-ended nature of language means that there can be multiple valid responses to a given prompt, complicating the assessment of correctness.

“However, the open-ended nature of language means that there can be multiple valid responses to a given prompt, complicating the assessment of correctness [1].”

• cites [1] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The development of benchmarks requires careful consideration of workload representativeness to ensure that the tests reflect real-world usage scenarios.

“The development of benchmarks requires careful consideration of workload representativeness to ensure that the tests reflect realworld usage scenarios [19].”

• cites [19] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The search encompassed peer-reviewed publications, prominent technical preprints, and widely cited benchmarking platforms up to October 2023.

“The search encompassed peer-reviewed publications, prominent technical preprints, and widely cited benchmarking platforms up to October 2023.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Standardization is essential in benchmarking to ensure consistency across evaluations, making the results reliable and widely accepted within the community.

“Standardization is essential in benchmarking to ensure consistency across evaluations, making the results reliable and widely accepted within the community [17].”

• cites [17] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

A lack of diversity in evaluators could bias results, making benchmarks less reflective of LLMs' real-world performance across varied scenarios.

“A lack of diversity in evaluators could bias results, making benchmarks less reflective of LLMs' real-world performance across varied scenarios.”

Classified as probably not a substantial claim.

The framework was designed not only to assess technical performance but also to evaluate the broader applicability, robustness, and integrity of LLM benchmarks.

“This framework was designed not only to assess technical performance but also to evaluate the broader applicability, robustness, and integrity of LLM benchmarks.”

Classified as probably not a substantial claim.

The framework was based on a thorough review of existing literature and best practices in the field, tailored to address the unique capabilities and challenges presented by generative AI especially LLMs.

“We based the framework on a thorough review of existing literature and best practices in the field, tailoring it to address the unique capabilities and challenges presented by generative AI especially LLMs.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Functionality considers the alignment of the benchmark with real-world applications, ensuring it tests skills relevant to practical use rather than artificial metrics.

“Functionality also considers the alignment of the benchmark with real-world applications, ensuring it tests skills relevant to practical use rather than artificial metrics.”

Classified as probably not a substantial claim.

The study developed a comprehensive framework to benchmark LLMs, grounded in principles of cybersecurity risk assessment.

“In this section, we developed a comprehensive framework to benchmark LLMs, grounded in principles of cybersecurity risk assessment.”

Classified as probably not a substantial claim.

The study applied thematic analysis to systematically evaluate the selected benchmarks, focusing on identifying patterns of inadequacies in functionality and integrity.

“We applied thematic analysis to systematically evaluate the selected benchmarks, focusing on identifying patterns of inadequacies in functionality and integrity.”

May combine two claims.

LLMs can inadvertently learn and propagate biases present in training data, requiring benchmarks that can detect and mitigate such issues.

“LLMs can inadvertently learn and propagate biases present in training data, requiring benchmarks that can detect and mitigate such issues [32].”

• cites [32] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The study conducted a Structured Literature Review (SLR) to systematically identify relevant LLM benchmarks.

“We conducted a Structured Literature Review (SLR) to systematically identify relevant LLM benchmarks.”

May combine two claims.

LLMs operate as black boxes with opaque internal mechanisms, making it difficult to interpret their decision-making processes and identify potential biases or ethical issues.

“Furthermore, LLMs operate as black boxes with opaque internal mechanisms, making it difficult to interpret their decision-making processes and identify potential biases or ethical issues [4], [27].”

• cites [4] sourceNot checked yet

  Not checked in the free run.

• cites [27] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

A benchmark relying on superficial elements like keyword matching can lead to overestimating an LLM's language understanding when it simply exploits patterns without grasping the content.

“A counterexample is using benchmarks that rely on superficial elements like keyword matching, which can lead to overestimating an LLM's language understanding when it simply exploits patterns without grasping the content.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LLMs like Llama and GPT-4 are capable of understanding and generating human-like text, performing tasks ranging from translation to creative writing.

“LLMs like Llama and GPT-4 are capable of understanding and generating human-like text, performing tasks ranging from translation to creative writing [1], [4].”

• cites [1] sourceNot checked yet

  Not checked in the free run.

• cites [4] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

When evaluating LLM benchmarks, two core aspects must be considered: functionality and integrity.

“When evaluating LLM benchmarks, two core aspects must be considered: functionality and integrity.”

• AssumptionRests on an assumption: Functionality and integrity are the two core aspects for evaluating LLM benchmarks.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

An integrity breach occurs when LLMs appear to excel in benchmarks through surface-level optimizations without demonstrating genuine capability.

“An integrity breach occurs when LLMs appear to excel in benchmarks through these surface-level optimizations without demonstrating genuine capability.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

By integrating people, process, and technology, this framework allows for a more nuanced assessment compared to models that may overlook key aspects of LLM benchmarking.

“By integrating people, process, and technology, this framework allows for a more nuanced assessment compared to models that may overlook key aspects of LLM benchmarking.”

Probably not a substantial claim: may not pass S2 (checkable).

Evaluating such models requires benchmarks that can assess not just quantitative performance, but also qualitative aspects like coherence, relevance, and ethical considerations.

“Evaluating such models requires benchmarks that can assess not just quantitative performance, but also qualitative aspects like coherence, relevance, and ethical considerations [3].”

• cites [3] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The study selected 23 LLM benchmarks that represented a diverse set of tasks and domains, ensuring comprehensive coverage of the LLM evaluation landscape.

“Applying such criteria of our Unified Evaluation Framework, we selected 23 LLM benchmarks that represented a diverse set of tasks and domains, ensuring comprehensive coverage of the LLM evaluation landscape.”

• AssumptionRests on an assumption: The 23 selected LLM benchmarks are representative of state-of-the-art LLM benchmarks.

Probably not a substantial claim: may not pass S2 (checkable).

The global deployment of LLMs requires benchmarks that consider linguistic diversity and cultural nuances, moving beyond English-centric evaluations to include multiple languages and dialects.

“Moreover, the global deployment of LLMs requires benchmarks that consider linguistic diversity and cultural nuances, moving beyond English-centric evaluations to include multiple languages and dialects, which contrasts with traditional benchmarks that are largely language-agnostic or focused on English [35].”

• cites [35] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks.

“The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks [3], [13], [30].”

• cites [3] sourceNot checked yet

  Not checked in the free run.

• cites [13] sourceNot checked yet

  Not checked in the free run.

• cites [30] sourceNot checked yet

  Not checked in the free run.