AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“These benchmarks comprise CommonSenseQA (Talmor et al., 2019), PIQA (Bisk et al., 2020), XSum (Narayan et al., 2018), MMLU (Hendrycks et al., 2021), AGIEval (Zhong et al., 2023a), and AlpacaEval (Li et al., 2023).”

“Although LM “agents” represent a significant paradigm shift toward dynamic applications, our approach to evaluation has remained predominantly static (Liang et al., 2023; Srivastava et al., 2023; Zhong et al., 2023).”

“However, existing benchmarks (Hendrycks et al., 2020; Zhong et al., 2023; Zheng et al., 2023a) for evaluating LLMs mainly focus on NLP tasks, such as common sense reasoning, academic examination, and authenticity verification.”

“With the development of large language models [73]–[82], many researchers have introduced LLMs into code completion [83]–[89].”

“Various benchmarks have been introduced to assess LLMs (Hendrycks et al., 2021; Li et al., 2023a; Zhong et al., 2023; Hugging-Face, 2023; Chang et al., 2023), including: (1) Human-centric evaluation, typified by AdaVision (Gao et al., 2022) and AdaTest (Ribeiro & Lundberg, 2022) that emphasize…”

“There are also several benchmarks attempting in evaluating the general knowledge in Chinese LLMs. C-MMLU (Li et al., 2023c), C-Eval (Huang et al., 2023), M3KE (Liu et al., 2023a) and AGIEval (Zhong et al., 2023) are all Chinese counterparts of MMLU that include diverse sets of questions from multiple subjects with dif…”

“We use common evaluation settings for a single task: 5-shot accuracy for AGIEval, 4-shot accuracy for GSM8K and MATH, 3-shot accuracy for MBPP, and zero-shot accuracy for HumanEval and WinoGrande.”

“Some sets (Ling et al., 2017; Zhong et al., 2023) focus on pure math prowess, while others (Mishra et al., 2022; Suzgun et al., 2022) combine math with other abilities.”

“AQuA [68], SAT-Math [69], MMLU-STEM [70] are composed of multiple-choice questions for human exams across math and other STEM disciplines.”

“Despite LLMs not yet fully replacing scientific researchers in generating creative discoveries, they have demonstrated substantial potential in enhancing researchers’ efficiency in scientific literature analysis (AI4Science and Quantum, 2023).”

“Many popular benchmarks have become inadequate for evaluating frontier LLMs. HLE is designed to be significantly harder, targeting the frontier of human knowledge to better assess advanced model capabilities.”

“Representative benchmarks include AGIEval (Zhong et al., 2024), MMLU-Pro (Wang et al., 2024d), and Gaokao (Guan et al., 2025).”

“QA tasks form a core part of NLP, providing a standard test to measure a model’s ability to understand information and apply reasoning [36, 37, 38, 39].”

“…ecosystem for LMM evaluation has been supported by the rapid emergence of community-driven leaderboards and open-source evaluation tools [10,12–14], which have evolved into central platforms for aggregating evaluation results, tracking performance across models, and promoting reproducibility.”

“(4) Other Popular Benchmarks : We report the average accuracies on GSM8K (8-shot) [15], MMLU (5-shot) [28], Big Bench Hard (BBH) (3-shot) [68], and AGI-Eval (0-shot) [92].”

“For AGIEval, the results were mixed.”

“…such as multi-choice QA to measure the performance of LLMs via simple objective metrics (such as accuracy and F1 score) (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023b), which may deviate from the common usage of LLMs and may not exactly reflect the ability of LLMs in…”

“We focus on unseen tasks, i.e. , the examination datasets of AGIEval [120], as well as language datasets AFQMC [108], WSC [52], knowledge datasets BoolQ [20], NaturalQues-tions [50], understanding datasets C3 [87], XSum [64], and reasoning datasets RTE [24], AX-b and AX-g [101] as the target tasks.”

“NaturalQues-tions (Kwiatkowski et al., 2019) and Trivi-aQA (Joshi et al., 2017) focus on retrieving real-world information, while multi-domain tasks are covered by MMLU (Hendrycks et al., 2020), BBH (Suzgun et al., 2023), and AGI Eval (Zhong et al., 2024).”

“The GPT-series model released by OpenAI is one of the most representative and widely used proprietary LLMs, including the newly released GPT-4o as well as ChatGPT and GPT-4, which have achieved state-of-the-art performance on numerous professional benchmarks without fine-tuning [18–20, 39”

“By addressing these aspects, this survey seeks to provide a thorough understanding of the unique challenges posed by LLM agents and contribute to the development of safer and more reliable Artiicial General Intelligence (AGI) systems [126].”

“Existing physics benchmarks (Zhong et al., 2023; Zhang et al., 2023; Arora et al., 2023; Ma et al., 2024; Ding et al., 2023; Jaiswal et al., 2024b) primarily target middle or high school physics and typically are multiple-choice questions, which are relatively easy for current LLMs to handle.”

“11 The ensemble exhibits two notable characteristics: (i) on tasks where all constituents perform at a similar level, the ensemble improves over the performance of each individual model (e.g., PiQA, BoolQ, MMLU) and (ii) on tasks were either one of the constituents performs exceptionally well, the ensemble moves the n…”

“C L ] 9 S e p 2024 PREPRINT perform like an expert adult, as argued in (Yue et al., 2023; Hendrycks et al., 2020; Zhong et al., 2023; Zhang et al., 2023).”

“AGIEval (Zhong et al., 2023) uses Chinese standardized exams like college entrance exams, math competitions, and lawyer qualification tests to build a test set for Chinese LLMs.”

“AGIEval (Zhong et al., 2023) is a well-known benchmark that quantifies the reasoning capability of foundation models in the context of human-centric standardized exams, including college entrance exams, math competitions, lawyer qualification tests, etc.”

“To address the need for more difficult benchmarks, recent expert-level benchmarks have been devised for LLMs [28; 84] and multimodal models [81; 51].”

“Our evaluation contains a comprehensive analysis over standard benchmarks below 8k context length such as MMLU [30], HellaSwag [79], ARC [15], SAT [82], GSM8k [16], Winograde [58] and MBPP [3], as well as popular long context benchmarks with needles-in-a-haystack [37] and the retrieval and QA…”

“For example, the performance on diverse datasets such as MMLU [21] and AGIEval [74] shows a steady growth as scales expand for Yi series models [72].”

“…we aggregate performance across 7 math benchmarks—SAT-Math, MMLU Elementary, High School, College Math, GSM-IC, Aqua-RAT and Minerva-Math [71, 14, 52, 27, 22]—where the reference model M ref achieves meaningful accuracy above 10%—and 7 NLI benchmarks—HANS, WNLI, ANLI, QNLI, NLI…”

“…BigBench Hard (BBH) (Suzgun et al., 2022), MMLU (Hendrycks et al., 2020), ARC (Clark et al., 2018), TriviaQA (Joshi et al., 2017), NaturalQuestions (Kwiatkowski et al., 2019), AGIEval (Zhong et al., 2023) ， CLUEWSC (Xu et al., 2020), C-Eval (Huang et al., 2023), and CMMLU (Li et al., 2023a).”

“In the era of LLMs, more comprehensive benchmark datasets have been released, such as MMLU [40], BIG-Bench [130], and AGI Eval [193].”

“• Knowledge: GPQA-Diamond (Rein et al., 2023), MMLU-Pro (Wang et al., 2024), C-Eval (Huang et al., 2023), Phybench (Qiu et al., 2025), AGIEval (Zhong et al., 2023), TriviaQA (Joshi et al., 2017), CMMLU (Li et al., 2023).”

“For DPO we choose the LSAT (Wang et al., 2022), LogiQA (Liu et al., 2021) splits from AGIEval (Zhong et al., 2024), Math split of MMLU Pro (Wang et al., 2024b).”

“These benchmarks provide limited guidance for highly specialized fields such as the legal domain[53, 22, 5].”

“…solution-level scores generated by both ORM and PRM. Conversely, if both conditions are not satisfied, the instance is marked as hard and delegated to the fine-grained method (as described in Section 2.2), addressing Issue 1 (excessive refinement) by only refining solutions for the hard problems.”

“With the increasing use of LLMs for various purposes such as math problem solving (Hendrycks et al., 2021), reading comprehension (Zhong et al., 2023 LLM-as-a-Judge (Zheng et al., 2023), FairEval (Wang et al., 2023b), ChatEval (Chan et al., 2023), and our own work, S CALE E VAL .”

“In addition, there is a strong move (Hendrycks et al., 2021b; Zhong et al., 2023; Huang et al., 2023) towards assessing a model’s professional knowledge and expertise through exam tasks.”

“These studies underscored the complexity of achieving task-specific accuracy and functionality with LLMs. Lastly, AGIEval [62] assessed LLMs’ general abilities but noted struggles in complex reasoning tasks.”

“5 (Liu et al., 2024)), Gemma9B (Gemma-2-9B-it (Zhong et al., 2023)), Gemma27B (Gemma-27B-it (Zhong et al., 2023)), and Ministral8B (Ministral-8B-Instruct-2410).”

“Model Evaluation Several benchmarks have been proposed to evaluate Large Language Models (LLMs) on human exams like college entrance and law school admission tests (Suzgun et al., 2022; Sri-vastava et al., 2023; Choudhary and Reddy, 2023; Zhong et al., 2023).”

“Despite the emergence of various benchmarks, no benchmark comprehensively evaluates LLMs as human-like dialogue systems.”

“Furthermore, general data sets designed to evaluate multiple model capabilities simultaneously, such as those for Artificial General Intelligence (AGI; [28]), tend to lack a specific focus on a single foundational ability.”

“First, we use MMLU (Hendrycks et al., 2020) and AGIEval (Zhong et al., 2023), which are based on asking LLMs multiple-choice questions.”

“These tasks are grouped into several categories, such as: (a) General Knowledge/Reasoning ( e.g., ARC (Bhakthavatsalam et al., 2021), AGIEval (Zhong et al., 2024), OpenBookQA (Mihaylov et al., 2018), BBH (Suzgun et al., 2023), ProntoQA (Saparov and He, 2023), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019),…”

“More comprehensive science QA benchmarks like ScienceQA (Lu et al., 2022), C-EVAL (Huang et al., 2023), AGIEVAL (Zhong et al., 2023), MMMU (Yue et al., 2023a), and SciBench (Wang et al., 2023) have recently been introduced, providing a broader scope of assessment.”

“Unless otherwise stated, we follow prior benchmark literature (Suzgun et al., 2022; Zhong et al., 2024) by adopting greedy decoding 7 to minimize the noise for open-ended text generation.”

“NaturalQues-tions (Kwiatkowski et al., 2019) and Trivi-aQA (Joshi et al., 2017) focus on retrieving real-world information, while multi-domain tasks are covered by MMLU (Hendrycks et al., 2020), BBH (Suzgun et al., 2023), and AGI Eval (Zhong et al., 2024).”

“For closed tasks , we utilized five established benchmarks: GPQA (Rein et al., 2023), AGIEval-MATH (Zhong et al., 2024), LIAR (Wang, 2017), WSC (Levesque et al., 2012), and BBH-navigate (Suzgun et al., 2023).”

“0 [77]), generic reasoning (BigBenchHard [92], PIQA [7], AGIEval [118]) and mathematical reasoning (Arithmetic [9], MathQA [3], GSM8k [14], ASDiv [71]).”

“The generic benchmark dataset, such as AGIEval [71] and C-Eval [16], is intended to general models and is not suitable to test a specific model’s output with a standardized structure.”

“We use AGIEval [31] as benchmarks for standardized tests, including Chinese and English tests.”

“These examples were created by first collecting a set of 59K candidate question/answer pairs from existing datasets [LI et al., 2024, of America, 2024, Gao et al., 2024, Zhong et al., 2023] and generating reasoning traces and responses for the respective questions using Gemini Flash [Google, 2024].”

“…(LLMs) (Vaswani et al., 2023; Devlin et al., 2019) such as GPT-4 (OpenAI, 2023), Llama-2 (Touvron et al., 2023), and Mistral (Jiang et al., 2023) have showcased their ability to understand and generate text akin to human interaction (Zhong et al., 2023; Pu et al., 2023; Dasgupta et al., 2023).”

“…benchmarks have been proposed to evaluate the ability of the model to solve more complex tasks such as college entrance exams, law * These authors contributed equally to this work. school admission tests, math competitions and so on (Hendrycks et al., 2021; Guo et al., 2023; Zhong et al., 2023).”

“…on open-source benchmarks in both few-shot and zero-shot settings, including: ARC-Challenge [6], BBH [40], DROP [11], WinoGrande [35], HellaSwag [55], MMLU [15], C-Eval [17], TriviaQA [22], Ape210K [56], GSM8K [7], MATH [56], MBPP [3], HumanEval [5], AGIEval [57], GPQA [34], and MMLU-Pro [42].”

“Evaluating LLMs. Benchmarks are frequently used to evaluate LLMs’ scientific knowledge and mathematical reasoning [10, 23, 24, 35, 50, 56, 58, 69].”

“We perform evaluations on 4 comprehensive benchmarks, including MMLU [31], CMMLU [42], AGIEval [107] and C-Eval [33].”

“CEval-Sci [28], MMLU-Sci [29], SciEval [6], SciBench [7], GPGQ [8], GSM8K [30], MATH [31], Mathematics [32], SAT-Math [33], MMLU-Math [29] from MathInstruction, and CEval-MATH [28].”

“…the min-max normalized average of scores from 0-shot ARC-Easy/Challenging [3], BoolQ [31], HellaSwag [35], OpenBookQA [20], PIQA [2], WinoGrande [23] (known together as the “GPT4All” benchmarks) and AGIEval [37] as calculated by lm-evaluation-harness [5], as well as IFEval [38] 3 and MT-Bench [36].”

“The results indicate that while the best-performing models achieve high accuracy in basic knowledge tasks (>80%), they face significant challenges in professional-level reasoning tasks, with accuracy rates dropping to 50-60% (Huang et al. 2024; Zhong et al. 2024).”

“To support multi-model assessments, we created a connector framework that standardizes interactions with various LLM APIs, allowing for efficient management of the entire assessment process across different models [100].”

“Various benchmarks such as MMLU (Hendrycks et al., 2020), AGI-Eval (Zhong et al., 2023), MedMCQA (Pal et al., 2022), and SocialIQA (Sap et al., 2019) have been designed to assess MCQA proficiency across different domains.”

“…Models (e.g. ChatGPT and GPT-4 (OpenAI, 2023)) focus on their zero-shot/few-shot abilities on basic natural language tasks (Jiao et al., 2023; Zhong et al., 2023; Wang et al., 2023b; Qin et al., 2023a) and their tool-use ability to generate APIs for solving user instructions, such as basic…”

“While previous evaluation studies have assessed models on benchmarks such as exams across diverse topics (Hendrycks et al., 2021; Zhong et al., 2023), they do not consider holistic aspects such as general writing ability and alignment with human values.”

“The model is then evaluated on 4 in-domain benchmarks [12, 19, 31, 40] and 2 out-of-domain benchmarks [70, 18] using zero-shot prompt.”

“…on maths cloze questions and 47.0% on maths choice questions retrieved from authentic Chinese entrance exams to higher education, far away from the average human performance of 73% (Zhong et al., 2023), and achieved a modest accuracy rate of 53.9% on a special MATH dataset (Zheng et al., 2023).”

“To name some of them: AGIEval [69], AlpacaFarm [12], BBQ [46], BoolQ [8], Codex HumanEval [7], GSM8K [9], HELM [32], MMLU [22], MultiPL-E [5],ToxiGen [18], WinoGrande [51], and others.”

“We then populate it with sub-categories and corresponding benchmarks: 1) Multi-subject Test, we use the Chinese partition of AGIEval (Zhong et al., 2023) denoted as AGIEval-ZH, and C-Eval (Huang et al., 2023) which is a comprehensive multi-discipline exam benchmark in Chinese; 2) Mathematical Reasoning, we adopt MGSM…”

“Zhong et al. (2023) and Beeching et al. (2023) evaluate the models on a series of benchmark tasks, but they only evaluate a limited number of models: The former one only evaluate the GPT series from OpenAI, and the latter one only evaluate open-sourced LLM models.”

“To demonstrate the effectiveness of our Language-model-as-an-examiner framework, we first employ GPT-4 [18] as the examiner for a centralized evaluation, since it exhibits a broad understanding of knowledge [9, 48, 49] and a precise judgmental ability [15, 16].”

“…large-scale Chinese benchmark that covers nive tasks such as single-sentence/sentence-pair classification, reading comprehension, etc. AIGEval (Zhong et al., 2023) aims to assess the capabilities of LMs in human-centric standardized exams, including college entrance exams, law school…”

“In particular, unlike recent benchmarks MMCU (Zeng, 2023) and AGIEval (Zhong et al., 2023), M3KE covers all major levels of Chinese education system, ranging from primary school to college, as well as a wide variety of subjects, including humanities, history, politics, law, education, psychology, science, technology,…”

“We evaluate reasoning performance on diverse math and general-purpose benchmarks: MATH -500 (Hendrycks et al., 2021b), AMC 23, test set of MMLU (Hendrycks et al., 2021a), MMLU - PRO (Wang et al., 2024), AGIEVAL (Zhong et al., 2023), GPQA - DIAMOND (Rein et al., 2024) and SUPERGPQA (Team et al., 2025).”

“Evaluation We employ diverse academic evaluation datasets to evaluate different capabilities of LLMs. Con-cretely, we use the following benchmarks: • Pretraining (PT) benchmark: Boolq (Clark et al., 2019), SIQA (Sap et al., 2019), PIQA (Bisk et al., 2020), ARC-c&ARC-e (Clark et al., 2018), MMLU (Hendrycks et al., 2021…”

“Model evaluation results averaged across seven math benchmarks (AIME25, AIME24, AMC23, GaoKao23English, Olympiad-Bench, GSM8K, and MATH500).”

“CEval-Sci [28], MMLU-Sci [29], SciEval [6], SciBench [7], GPGQ [8], GSM8K [30], MATH [31], Mathematics [32], SAT-Math [33], MMLU-Math [29] from MathInstruction, and CEval-MATH [28].”

“Agieval [57], SciBench [48], MiniF2F [55], and OlympiadBench [21] generally aim towards challenging, math competition-style problems.”

“Previous studies (Li et al., 2023; Zhong et al., 2023; Zhang et al., 2023) shows that using few shots on instruction-tuned models does not enhance accuracy and may even may decrease it.”