AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“We report the average accuracy of the model on the MMLU (Hendrycks et al., 2020) and AGIEval (Zhong et al., 2023) benchmarks.”

“Clean and robust benchmarks are the key to guide further progress of various models in NLP. Popular benchmarks used to evaluate large language models include: Comprehensive : MMLU, Big Bench (Srivastava and et al., 2023), AGI Eval (Zhong et al., 2023”

“Standardized tests, which are originally designed for humans, have become common LLM benchmarks in domains such as mathematics [30, 22, 50, 16], medicine [48, 15, 40], and multi-task collections [29, 65, 74, 89].”

“A.1 L IMITATION The major limitation of KoLA is that our coverage is not as extensive as some other recent works (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023).”

“Current LLM-centric scenarios have widely utilized the task format of MCQ, for instance, within benchmarks targeted at assessing LLMs (Hendrycks et al., 2020; Zhong et al., 2023; Huang et al., 2023) and in LLM-based automatic evaluation frameworks (Chiang et al., 2023; Zheng et al., 2023b).”

“[48], MMLU-Pro (M. Pro) [40], BBH [34], DROP [12], MBPP [1], HumanEval (H.Eval) [3], MATH [18], and GSM8k [9].”

“…Math-500, Bird-SQL, LiveCodeBench, Humanities Last Exam, MMLU-Pro, AGIEval English, CRUXEval-I/O, SimpleQA, Chatbot Arena, Webdev Arena [80, 20, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 19, 94, 95, 96, 97, 98] Each metric was then mapped to the AI capability it most directly tests.”

“In finetasks, we choose the 12 tasks belonging to 3 dimensions: • General Knowledge: AGI-Eval (Zhong et al., 2024), C-EVAL (Huang et al., 2023), CMMLU (Li et al., 2024a), M3Exams (Zhang et al., 2023).”

“These include grade-school math questions from GSM8K (Cobbe et al., 2021), algebraic word problems from AQuA (Ling et al., 2017), and SAT math problems from AGIEval (Zhong et al., 2023).”

“…pass@1 scores on HumanEval (0-shot) (Chen et al., 2021) and MBPP (3-shot) (Austin et al., 2021), the average accuracies on GSM8K (8-shot) (Cobbe et al., 2021), MMLU (5-shot) (Hendrycks et al., 2020), Big Bench Hard (BBH) (3-shot) (Suzgun et al., 2022), and AGI-Eval (0-shot) (Zhong et al., 2023).”

“These test sets typically encompass a range of tasks, assessing the LLM’s performance by comparing its task outputs to standard answers (Liang et al., 2022; Yu et al., 2024; Zhong et al., 2023).”

“…[102] Open-formed Reference responses Instruction Following MT-bench [172] Open-formed Reference responses Instruction Following MMLU [173] Automatic Evaluations Golden answers General Ability Tests, Exams AGIEval [174] Automatic Evaluations Golden answers General Ability Tests, Exams S UP -”

“Most existing benchmarks can be categorized as follows: 1) Core-knowledge benchmarks, e.g., MMLU (Hendrycks et al., 2020), C-Eval (Huang et al., 2023) and AGIEval (Zhong et al., 2023); 2) Instruction-following benchmarks, like LLMBAR (Zeng et al., 2023), Flan (Longpre et al., 2023) and NaturalInstructions (Mishra et a…”

“These benchmarks fall short as they fail to discern whether success in resolving vulnerabilities stems from leveraging pre-trained parameterized knowledge [23], [24], or from a logical reasoning between code and vulnerabilities.”

“Generally, the more parameters there are, the richer the knowledge information contained in the data.”

“AGIEval (Zhong et al., 2024) assesses tasks at a human level, which are also challenging.”

“Early efforts to gauge LLM agent self-reflection were often indirect, repurposing existing reasoning or planning tasks, such as AGIEval (Zhong et al., 2023), MedMCQA (Pal et al., 2022), ALFWorld (Shridhar et al., 2021), MiniWoB++ (Liu et al., 2018), etc., into multi-turn feedback loops, to see if models could recogniz…”

“…integrating visual and textual inputs to perform tasks such as image captioning (Chen et al., 2015; Agrawal et al., 2019), visual question answering (Goyal et al., 2017; Marino et al., 2019; Hudson & Manning, 2019), and open-ended instruction following (Hendrycks et al., 2021; Zhong et al., 2023).”

“Beyond general intelligence benchmarks (e.g., GPQA Diamond [109], AGIEval [149], HLE [8]), there are specialized benchmarks targeting mathematical reasoning (e.g., AIME [5], MathVista [12], IMO [9]), coding (e.g., SWE-bench [13], CodeForce [7], LiveCodeBench [11], IOI [10]), and multi-modal…”

“Automatic evaluation is also done using standard benchmarks such as MMLU [59], HELM [60], C-Eval [61], AGIEval [62], AlpacaFarm [63], Chatbot Arena [64], etc.”

“Brookins & DeBacker (2023); Kasberger et al. (2023); Guo (2023); Phelps & Russell (2023); Akata et al. (2023); Gandhi et al. (2023); Horton (2023) explored the potential of using LLMs as players in strategic games, such as the Prisoner’s Dilemma and the Dictator game.”

“Current evaluation benchmarks (Liang et al., 2022; Srivastava et al., 2022; Gao et al., 2021; Zhong et al., 2023; Li et al., 2023a) are mostly a mixture of the former three paradigms.”

“Recently, a surge of high-quality evaluation benchmarks (Hendrycks et al., 2021; Huang et al., 2023) have been proposed to provide a comprehensive capability evaluation of LLMs. Typical benchmarks include MMLU (Hendrycks et al., 2021) (for measuring multitask language understanding ability), Big-Bench (Srivastava et a…”

“This training-free nature of ICL has led to its rapid and broad application across a wide range of scenarios and applications, as illustrated by benchmarks such as Hendrycks et al. (2021); Zhong et al. (2024); Srivastava et al. (2023); Cobbe et al. (2021).”

“We additionally include out-of-distribution test sets from MAWPS (Koncel-Kedziorski et al., 2016), Mathe-matics (Davies et al., 2021), SVAMP (Patel et al., 2021), plus the math subsets of MMLU (Hendrycks et al., 2021a) and SAT (Zhong et al., 2023).”

“This section outlines our data processing pipeline, model selection criteria, and implementation details.”

“We employ GPT-4 to filter through various existing evaluation datasets, including BIGBench (Sri-vastava et al., 2022), AGIEval (Zhong et al., 2023), MMLU (Hendrycks et al., 2020), NYCCHessel et al. (2023) to isolate questions suitable for CoI tasks.”

“To further explore the capability of LLMs, some benchmarks (AlpacaEval (Li et al., 2023b), AGIEval (Zhong et al., 2023), GSM8K (Cobbe et al., 2021), MMLU (Hendrycks et al., 2021) and etc) are introduced to explore the capability of LLMs in solving problems, but they can only manifest the language skills and are inadeq…”

“However, the naive GRPO method [37] in DeepSeek-R1-Zero, designed for tasks like Mathematical Problem Solving (MPS) [12, 20, 19, 60, 50], uses sparse reward based on the binary verification signal.”

“MMLU (Hendrycks et al., 2020) and Big-bench (Srivastava et al., 2022) are designed as multiple-choice questions to measure the knowledge and reasoning ability of LLMs. AGIEval (Zhong et al., 2023) constructs human-centric evaluation for LLMs from standardized exams.”

“…mainstream English mathematical benchmarks, which cover difficulty levels ranging from elementary school to competition level: GSM8K (Cobbe et al., 2021), ASDiv (Miao et al., 2021), SAT_MATH (Zhong et al., 2023), MATH (Hendrycks et al., 2021), AIME2024 (AI-MO, 2024a), and AMC2023 (AI-MO, 2024b).”

“Specifically, our experiments aim to address the following research questions: (i) Can our method prevent model collapse on complex reasoning tasks during iterative self-improvement?”

“Large Language Models (LLMs) have demonstrated remarkable performance in solving complex reasoning tasks (Rae et al., 2021; Lewkowycz et al., 2022; Zhong et al., 2023), such as math reasoning (Imani et al., 2023; Ahn et al., 2024), symbolic reasoning (Kojima et al., 2022), and common-sense…”

“To better understand the capability of LLMs, much effort has been dedicated to evaluating LLMs on multiple benchmarks (Fabbri et al., 2021; Gao & Wan, 2022; Huang et al., 2023; Bang et al., 2023; Zhong et al., 2023).”

“In this section, we introduce the details of experi-398 ments for evaluating our approach. and GaoKao (Zhong et al., 2023) for Chinese phys-422 ical, chemical, and mathematical reasoning.”

“Popular question-answer benchmarks include a wide range of tasks, from common sense reasoning to code generation, with exam-based assessments gaining prominence (e.g., MMLU (Hendrycks et al., 2021a), GSM-8k (Cobbe et al., 2021), AGIEval (Zhong et al., 2023), and GPQA (Rein et al., 2023)).”

“Additionally, we include GaoKaoMath (Zhong et al., 2023), to assess the generality of our method on questions originating from the Chinese college entrance exam.”

“Recognizing the importance of out-of-distribution generalization, we also conducted experiments on LSAT (Zhong et al., 2024) and show results in Table 2.”

“Integrating such knowledge is crucial for handling edge cases and enhancing LLM reliability in specialized tasks, where domain knowledge can significantly improve performance.”

“Commonly used benchmarks include MATH [94], GSM8K [142], and AGIEval [192], which test models across different difficulty levels and problem types.”

“Firstly, these benchmarks are usually built on environments for static or single-objective tasks [Xie et al. , 2024; Zhong et al. , 2023], failing to incorporate real-world dynamic situations or constantly changing goals.”

“…the fine-tuned model, we conducted few-shot evaluations on GSM8K (Cobbe et al., 2021), Math (Hendrycks et al., 2021b), AQuA (Ling et al., 2017), simuleq (Koncel-Kedziorski et al., 2016), numglue (Mishra et al., 2022), MMLU STEM (Hendrycks et al., 2021a), and SAT math (Zhong et al., 2024) datasets.”

“Research has shown that the performance of GPT3 improves significantly when using few-shot learning prompts, particularly in domain-specific tasks [19].”

“…MATH (Hendrycks et al., 2021b), AQuA-RAT (Ling et al., 2017), NumGLUE (Mishra et al., 2022b)—and out-of-domain (OOD) test sets—SVAMP (Patel et al., 2021), SAT (Zhong et al., 2023), MMLU-Math (Hendrycks et al., 2021a), Mathematics (Davies et al., 2021), and SimulEq (Koncel-Kedziorski et al., 2016).”

“Compared to the MoE baseline, both DSE variants exhibit systematically smaller KL divergence, with the most pronounced gap appearing in the early blocks.”

“…& Lundberg, 2022; Kiela et al., 2021; Ma et al., 2021; Lambert et al., 2024; Li et al., 2024a; 2023; Lei et al., 2023; Liang et al., 2022; Wang et al., 2024b; Yu et al., 2023; 2024; Zhong et al., 2023; Huang et al., 2024; Hendrycks et al., 2020; 2021; Starace et al., 2025; Bogin et al., 2024).”

“However, general benchmarks[25, 60] often overlook detailed assessment of chemical knowledge.”

“Second, in terms of data contamination [14], knowledge capacity benchmarks usually publish their test data online to ensure transparency and reproducibility ( e.g., [6, 7]).”

“Early studies leverage open-ended question answering datasets and math word problems as the evaluation benchmarks (Touvron et al., 2023; et al., 2023b; Chen et al., 2024) to evaluate the commonsense knowledge and reasoning capabilities of LLMs. Subsequently, more benchmark datasets like MMLU (Hendrycks et al., 2021),…”

“Reading comprehension is based on the AGIEval dataset (Zhong et al., 2023).”

“Stem and code: MMLU-Pro (Wang et al., 2024), AGIEval (Zhong et al., 2023), MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), GPQA (Rein et al., 2023), MBPP (Austin et al., 2021), and HumanEval (Chen et al., 2021).”

“…achieved significant progress in reasoning through techniques such as chain-of-thought prompting [52, 27] and large-scale reinforcement learning [12, 33], recent models excel in benchmarks focused on mathematics, coding, and tool use (e.g., MMLU [14], AGIEval [57], LogicBench [38], MathVista [30]).”

“We use AGIEval ([22]), an aggregated benchmark aimed at assessing LLMs within the context of human-centric standardized exams to gauge their general abilities in human cognition and problem-solving.”

“We focused on an updated version of the Nous research benchmark consisting of the following metrics: • BIG-bench (Srivastava et al., 2023) • AGIEval (Zhong et al., 2023) • GPT4all: A Combination of HellaSwag, OpenBookQA, Winogrande, ARC Easy, ARC Challenge, BoolQ, PIQA) • The drop in performance is…”

“The ability of LLMs to understand, generate and sometimes even reason with human language has led to transfor-mative applications across numerous fields (Huang et al., 2023; Zhong et al., 2023).”

“Zero-shot methods vs. few-shot methods As Large Language Models develop in size and human-alignment, the gap between zero-shot and few-shot prompting methods seems to narrow [Zhong et al., 2023].”

“Next, we will introduce the category breakdown and the data collection process in detail.”

“Clean and robust benchmarks are the key to guide further progress of various models in NLP. Popular benchmarks used to evaluate large language models include: Comprehensive : MMLU, Big Bench (Srivastava and et al., 2023), AGI Eval (Zhong et al., 2023”

“…G EN 5 , five generative tasks including CoQA [92], SQuAD [93], Natural Questions [94], TriviaQA [95], and DROP [96], as well as (3) MMLU [97], (4) MMLU-Pro [98], (5) AGIEval [99] consisting of 20 tasks from college admission tasks, and (6) BBH [100] consisting of 23 challenging BIG-Bench tasks.”

“To evaluate the models’ knowledge and examination capabilities, we employ a range of benchmarks, including AGIEVAL [267] and MMLU [79].”

“Static benchmarks have been developed to assess the performance of LLMs across various tasks, such as language understanding [12], world knowledge [15, 53], reasoning [49], coding [7], and math [13].”

“At the same time, the corresponding evaluation works for LLMs are flourishing and a series of evaluation benchmarks have been proposed like MMLU [17], C-Eval [19], Xiezhi [15] and AGIEval [63], etc .”

“AGIEval.”

“Recently, AGIEval (Zhong et al., 2023) is proposed to evaluate LLMs using challenging human exams.”

“3, across almost all benchmarks – including professional exams (Hendrycks et al., 2020; Zhong et al., 2023; Clark et al., 2018), math (Cobbe et al., 2021), and reading comprehension (Mihaylov et al., 2018) – models trained with DPO on pragmatic data consistently outperform their SFT counterparts,…”

“…in evaluating LLMs encompass a diverse range including multitask language comprehension (e.g., MMLU [13]), benchmark assessments for LLMs (e.g., Big Bench [34]), commonsense reasoning (e.g., HellaSwag [43], ARC [9], PIQA [4], WinoGrande [31]), and human-centric evaluations (e.g., AGI-eval [46]).”

“Similarly, Mistral [30] report highlights performance across tasks like commonsense reasoning, world knowledge, reading comprehension, Math, code, and popular aggregated results, like MMLU [26], BBH [57], and AGI Eval [73].”

“• General Capabilities: To assess broad language understanding, instruction following, and factual knowledge, we use a diverse set of benchmarks including DROP [15], IFEval [68], AGIEval [66], and MMLU-PRO [56].”

“Thus, the MATH (Hendrycks et al., 2021), AGIEval (Zhong et al., 2023), OlympiadBench (He et al., 2024), and ARB (Sawada et al., 2023) benchmarks (which are all sourced from problems on the Internet) could potentially be contaminated.”

“Existing science benchmarks, such as SciQ (Welbl et al., 2017), Sci-Code (Tian et al., 2024), ScienceQA (Lu et al., 2022), AGIEval (Zhong et al., 2023), SciEval (Sun et al., 2024), SciBench (Wang et al., 2023), and VisScience(Jiang et al., 2024), typically cover a wide range of scientific disciplines, including biolog…”

“Prominent examples include C-Eval [30], AGI-Eval [84], MMLU [29], and CMMLU [36], which assess the proficiency of LLMs in various academic and practical disciplines.”

“Results in Table 1 show that while Std-CoT benefits from additional data, it underperforms compared to EDIT across most tasks.”

“Existing benchmarks to evaluate LLMs include core-knowledge benchmarks [39]–[45] that typically require LLMs to generate a short, specific answer to benchmark questions that can be automatically validated.”

“…AQuA-RAT (Ling et al., 2017), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), NumGLUE (Mishra et al., 2022), Mathe-matics (Davies et al., 2021), MMLU-Math (Hendrycks et al., 2021a), SAT-Math (Zhong et al., 2023), SimulEq (Koncel-Kedziorski et al., 2016) and SVAMP (Patel et al., 2021).”

“To evaluate this skill, numerous benchmarks have been developed, including human examination datasets (e.g., e.g. MMLU (Hendrycks et al., 2020), AGIEval (Zhong et al., 2023), C-Eval (Huang et al., 2023), and RACE (Lai et al., 2017)), datasets for assessing commonsense reasoning (e.g., ARC (Clark et al., 2018) and Comm…”

“…2, Vicuna 13 B, DeepSeek, Mistral-base, Mistral-Instruct, and LLaMa 30 B across various general benchmarks, such as MMLU (Hendrycks et al., 2021), AGIEval (Zhong et al., 2023), BBH (Sri-vastava et al., 2022), ARC (Xu et al., 2023), Hel-laSWAG (Zellers et al., 2019), RACE (Lai et al., 2017), etc.”

“For example, a bilingual benchmark including both English and Chinese has been released by Zhong et al. (2023) which focuses on standardized tests.”

“For general English capabilities on MMLU [Hendrycks et al., 2021a] and AGIEval [Zhong et al., 2023], MindLLM-1.”

“• General: MMLU-Pro (Wang et al., 2024d), MMCU (Zeng, 2023), AGIEval (Zhong et al., 2024), MMLU (Hendrycks et al., 2020), MMLU-Redux (Gema et al., 2025), C-Eval (Huang et al., 2023), CMMLU (Li et al., 2024), BBH (Suzgun et al., 2023), WinoGrande (Sakaguchi et al., 2021), Hu-manity’s Last Exam (HLE) (Phan et al., 2025).”