AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“We discover that on general tasks and datasets comprised primarily of words like MMLU, GPQA, and AGIEval, BlockBPE maintains the same generation quality despite using byte-level pre-tokenization.”

“For other languages, initiatives such as PersianMMLU [26], M3Exam [27] and AGIEval [28] have been proposed.”

“Common LLM benchmarks can be categorized into four types: (1) Knowledge Evaluation benchmarks test the LLM’s mastery of subject knowledge through objective multiple-choice and open-ended questions, including MMLU [6], CMMLU [25], CEval [26], AGIEval [7].”

“…major categories under the few-shot setting to holistically compare our model with the baseline: 1) Aggregate Benchmarks , including AGIEval-en [54], MMLU [15], MMLU-Pro [44], GPQA [37], and C-Eval [19]; 2) Mathematics, Code, and Reasoning , comprising GSM8K [9], MATH [16], HumanEval [7], ARC…”

“Under review as a conference paper at ICLR 2025”

“In this context, multiple-choice questions have emerged as one of the preferred methods for evaluating new generative models with datasets such as RACE (Lai et al., 2017), MMLU (Hendrycks et al., 2021a) or AGIEval (Zhong et al., 2023), which includes university entrance exams, law school entrance exams, mathematics co…”

“Gaokao (Chinese College Entrance Exam) is a subset in the AGIEval (Zhong et al., 2023).”

“AGIEval (Zhong et al., 2023) expanded upon these by incorporating fill-in-the-blank questions alongside multiple-choice.”

“• AGIEval (Zhong et al., 2023): A human-centric benchmark crafted to assess the general capabilities of foundation models in tasks aligned with human cognition and problem-solving.”

“Question-answering evaluations have made use of text (3; 4) and multimodal (5; 6) benchmarks to understand model capabilities to perceive and reason.”

“…benchmark often paves the path in a machine learning field, reflecting the trends in technological progress, including Scientific Machine Learning (Takamoto et al., 2022; Thiyagalingam et al., 2022), large language models (Valmeekam et al., 2022), and human-centric evaluations (Zhong et al., 2023).”

“To bridge this knowledge gap, researchers are investigating their capabilities across diverse benchmark datasets [WSM ∗ 18, ZCG ∗ 23].”

“A single-task attack is optimized and tested on one specific task category while a multi-task attack is supposed to be effective on all task categories.”

“MMLU (Hendrycks et al., 2020) and Big-bench (Srivastava et al., 2022) are designed as multiple-choice questions to measure the knowledge and reasoning ability of LLMs. AGIEval (Zhong et al., 2023) constructs human-centric evaluation for LLMs from standardized exams.”

“[118, 131, 132] reading comprehension multiple choice 3 510 0.25 AGIEval LSAT RC [118, 131, 132] reading comprehension multiple choice 3 268 0.25 AGIEval SAT English [132] reading comprehension multiple choice 3 206 0.25 ARC-Challenge [23] world knowledge multiple choice 10 2376 0.25 ARC-Easy [23]…”

“…al., 2021), MBPP (Austin et al., 2021), DS-1000 (Lai et al., 2023) ArXiv 101M 23/09-23/10 GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), SAT-Math (Zhong et al., 2023), MMLU-Math that an agent’s capacity to achieve goals across a wide array of scenarios should define its intelligence.”

“As a problem format listing alternative answers, MCQs are prevalent in real world and have led to numerous related datasets, such as MMLU (Hendrycks et al., 2020), BIG-bench (Srivastava et al., 2022), AGIEval (Zhong et al., 2023), CEVAL (Huang et al., 2023).”

“Most existing evaluation studies present results as single or aggregated metrics (Rein et al., 2023; Zhong et al., 2023), which support standardized comparison but offer limited guidance for practical deployment decisions.”

“While significant research has examined LLM performance on knowledge benchmarks [Hendrycks et al., 2021, Zhong et al., 2023] and reasoning tasks [Wei et al., 2022], comparatively little attention has been paid to how these models handle geopolitically contested issues—particularly across different…”

“We also use AGIEval to assess overall performance in real-world scenarios.”

“• AGIEval-LSAT-LR (Zhong et al., 2024) (510 items; 4-choice): LSAT Logical Reasoning—draw conclusions and identify assumptions from short arguments.”

“We evaluate our method on a diverse suite of benchmarks that cover different aspects of LLM capabilities, including MMLU (Hendrycks et al., 2021), TriviaQA (Joshi et al., 2017), TruthfulQA (Lin et al., 2021), ARC-Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), TyDiQA (Clark et al., 2020), Winogrande (Sa…”

“Other notable automatic evaluation approaches include PandaLM [145], HELM [146], AGIEval [147], and PICO [148].”

“…test used for college admissions in the United States; the LSAT (Law School Admission Test), which evaluates logical reasoning, reading comprehension, and analytical thinking for law school appli-cants; and the Gaokao, China’s highly competitive national college entrance exam [Zhong et al ., 2024].”

“While several datasets such as AGIEval [46], AlpacaFarm [47], BBQ [48], BoolQ [49], GSM8K [50], MMLU [51], WinoGrande [52], ASAP [53], SAF [43], Mohler [54], and others, have been developed to benchmark feedback and scoring tasks, they remain underutilized in large-scale studies or lack alignment…”

“Evaluation is conducted on eight diverse mathematical benchmarks: GSM8K [16], SVAMP [62], MATH [31], NumGLUE [56], SimulEq [42], AQuA [47], SAT [100], and MMLU-Math [30], covering competencies from basic arithmetic to multi-step symbolic reasoning and standardized test preparation.”

“To measure domain knowledge in English, we leverage the MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024b) and SuperGPQA (Team et al., 2025) benchmarks as well as for creative problem-solving and AGIEval (Zhong et al., 2023) for standardized exam performance.”

“The 7B LLMs have achieved noteworthy accomplishments on mainstream benchmarks, such as MMLU [38] or AGIEval [39].”

“…target deeper, multi-step problem solving and domain-specific expertise—either by repurposing advanced human exams and problem set (e.g., AGIEval [19], JEEBench [20], SciBench [21]) or by probing complex reasoning dimensions (e.g., MMLU-Pro [22], SciEval [23], TheoremQA [24]), up to the extreme…”

“To fairly assess and compare different LLMs’ capabilities, many benchmarks for evaluating LLMs have emerged and developed continuously (Liu et al., 2024; Hendrycks et al., 2021; Li et al., 2024; Zhong et al., 2024; Liang et al., 2023).”

“…2021), MBPP (Austin et al., 2021), HumanEval+, and MBPP+ (Liu et al., 2023); 3 math problem solving tasks: GSM8k (Cobbe et al., 2021), SAT-Math (Zhong et al., 2023), and MATH (Hendrycks et al., 2021); and 2 logic reasoning tasks ARC-Challenge (Clark et al., 2018) and LogiQA (Zhong et al., 2023).”

“In AGIEval benchmark, DFPE achieves a 4.”

“Since LLMs are susceptible to irrelevant context, they may also be influenced by other sub-questions [14].”

“Additional results on AGIEval English (Zhong et al., 2023) and Big-Bench Hard (Suzgun et al., 2022) can be found in Appendix F. MMLU is a question-answering dataset containing 57 tasks with diverse subjects and levels of difficulty, mainly focusing on the knowledge of the language models.”

“Besides the MMLU (Hendrycks et al., 2020) benchmark, we also evaluate our proposed data selection application on three benchmarks, i.e. , AGIEval (Zhong et al., 2024), BBH (Suzgun et al., 2023) and BoolQ (Clark et al., 2019), using different LLM backbones including Llama2-7b (Tou-vron et al., 2023), Mistral-7b (Jiang…”

“The benchmarks encompass MMLU, CMMLU, AGIEval (Zhong et al., 2023), C-Eval, MMLU-Pro (Wang et al., 2024), and LiveBench (White et al., 2024).”

“While Mukherjee et al. (2023) suggest evaluating world knowledge acquisition through tasks like AGIEval (Zhong et al., 2023) and Big-Bench Hard (Suzgun et al., 2023), we defer assessing larger multi-token models on such complex benchmarks to future work.”

“Traditional methods rely on carefully curated test sets to evaluate the model performance on specific task [6, 32], ranging from simple sentiment classification to more diverse Multilingual Multi-Genre Language Understanding (MMLU) benchmarks [33].”

“The selection of the five categories of evaluation datasets was informed by extensive prior work(Zhong et al., 2024; Polo et al., 2024; Liang et al., 2023), as these datasets have been widely adopted in the literature to represent diverse task types across multiple domains.”

“MCQs are extensively employed both for benchmarking LLM performance (Hendrycks et al., 2020; Zhong et al., 2023) and as the basis for automated evaluation frameworks (Zheng et al., 2023b).”

“• AGIEval : AGIEval [39] is a human-centric benchmark that evaluates a model’s abilities in tasks pertinent to human-cognition and problem-solving.”

“AGIEval [97] acts as a specialized evaluation framework dedicated to assessing the performance of foundation models in human-centric standardized exams.”

“For our experiments, we use the GSM8k (Cobbe et al., 2021a) and MATH (Hendrycks et al., 2021) datasets for in-distribution tasks, and AIME 2024, AMC, and AGIEval’s SAT Math and Gaokao Math subsets (Zhong et al., 2023) for out-of-distribution evaluation.”

“…QA datasets including HotpotQA, MuSiQue, SQuAD, and Triv-iaQA (Geva et al., 2021; Saparov and He, 2022; Tafjord et al., 2021; Han et al., 2024; Zhong et al., 2024; Lin et al., 2025a; Yu et al., 2020; Clark et al., 2020; Cobbe et al., 2021; Patel et al., 2021; Roy and Roth, 2015; Miao et al.,…”

“For OOD evaluation, ARC-C, MMLU-Pro, GPQA, AGIEval-English, BigBenchHard, and ZebraLogic-Grid are all averaged over four generations.”

“Over the last two years, Large Language Models (LLMs) have achieved superhuman performance on a variety of standardized benchmarks, from professional exams to programming competitions [2, 4, 10, 17, 29, 30, 36].”

“…Gaokao (Zhang et al., 2023), Math (Hendrycks et al., 2021), Min-erva Math (Lewkowycz et al., 2022), TabMWP (Lu et al., 2022), OlympiadBench (He et al., 2024), Mmlu Stem (Hendrycks et al., 2020), Sat Math (Zhong et al., 2024), Mawps (Koncel-Kedziorski et al., 2016), and Svamp (Patel et al., 2021).”

“AGIEval-Gaokao MathQA, AGIEval-Gaokao Geography, and AGIEval-Gaokao History adopt the same data split to facilitate the heterogeneous datasets’ construction.”

“By providing static questions, these benchmarks aim to achieve comprehensive coverage (Hendrycks et al., 2020; Rein et al., 2024; Zhong et al., 2024) or construct more complex problems (Wang et al., 2024c; Chen et al., 2025c; Phan et al., 2025) to enhance discrim-inability.”

“For out-of-domain datasets, we evaluate on GPQA [34], LAST [35], and MMLU [36], which cover a broad range of tasks across different domains.”

“Further efforts such as 5 https://mera.a-ai.ru/en/multi/leaderboard MMLU (Hendrycks et al., 2021), AGIEval (Zhong et al., 2024), and C-Eval (Huang et al., 2023) extended evaluation to academic and professional domains, though primarily in English and Chinese.”

“C-EVAL (Huang et al., 2023) and GAOKAO-Bench (Zhang et al., 2024) assess Chinese educational content, while AGIEval (Zhong et al., 2023) uses real-world exam questions such as the SAT and LSAT.”

“Benchmarking efforts have shifted focus from high school-level exams such as MMLU (Hendrycks et al., 2020), CEval (Huang et al., 2023), and AGIEval (Zhong et al., 2023), to more advanced scientific tests including GPQA (Rein et al., 2024), SciEval (Sun et al., 2024), and SciKnowEval (Feng et al., 2024), and further to…”

“Moreover, AGIEval[Zhong et al. 2023] also presents a similar landscape, evaluating LLMs in real-world cognitive challenges through standardized exams, such as college entrance exams, math competitions, and lawyer qualification tests.”

“…ground-truth benchmarks such as MMLU [Hendrycks et al., 2021], HellaSwag [Zellers et al., 2019], GSM-8K [Cobbe et al., 2021], FinanceQA [Ma-teega et al., 2025], AGIEval [Zhong et al., 2023], BIG-Bench [Srivastava et al., 2023], and HumanEval [Chen et al., 2021] dominate language-model evaluation.”

“…to be solved (Gao et al., 2023; Hao et al., 2023; Alazraki and Rei, 2025), to competition-level problems on advanced topics such as number theory, combinatorics and calculus, on which LLMs score consistently below 50% (Glazer et al., 2024; He et al., 2024; Zhong et al., 2024; Gao et al., 2025).”

“S2K is evaluated on the JEC-QA-KD subset from AGIEval (Zhong et al., 2024).”

“We evaluate our method on three challenging math-reasoning benchmarks, MATH-500 (Light-man et al., 2023b), SAT-MATH from AGIEval (Zhong et al., 2023), and AIME 2024.”

“…(Hua et al., 2024) • Knowledge : ARC (Clark et al., 2018), GPQA (Rein et al., 2023) • Soft Reasoning : MuSR (Sprague et al., 2023), AGIEval LSAT (Zhong et al., 2023) • Commonsense : CommonsenseQA (Talmor et al., 2018), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), StrategyQA (Geva et al.,…”

“They evaluated LLaMA-2-7b-chat-hf [34] on the AGIEval benchmark [35], sampling three random prompts and measuring model outputs for interpolation coe ffi cients α ∈ { 0 , 0 .”

“…etal.,2020;Chenetal.,2021;Austinetal.,2021;Hendrycksetal.,2021;Dhamalaetal.,2021;Cobbeetal.,2021;Linetal., 2022;Hartvigsenetal.,2022;Parrishetal.,2022;Srivastavaetal.,2023;Zhongetal.,2023;GeminiTeametal.,2024) google/gemma-7b gemma 2 66776 27 Telugu-LLM-Labs/Indic-gemma-7b-finetuned-sft-Navarasa-2.”

“• AGIEval (AI General Intelligence Evaluation) [84]: Evalu-ates the model’s performance on tasks that require general intelligence, such as logical reasoning and problem-solving.”

“In the Chinese language domain, researchers have developed SuperCLUE[7], AGIEval[8], and C-Eval[9], among others.”

“…experimental setup (Section 5.1), including training datasets, models used for 2,999 (14 sub-tasks) English 0 Code HumanEval [12] 164 English 0 MBPP [6] 974 English 0 Other (General) AGIEval [74] 8,062 (20 sub-tasks) English, Chinese 0 HellaSwag [67] 10,003 English 0 experimentation and benchmarks.”

“The other two subsets are sourced from the benchmark AGI-Eval (Zhong et al., 2023), SAT contains math problems from SAT exam in U.S. while Gaokao-Mathcloze is sourced from Chinese College Entrance Exam.”

“For example, MMLU [42], C-Eval [43] and AGIEval [44], use multiple-choice and fill-in-the-blank questions to assess the knowledge ability, logical reasoning ability, etc. of the big models.”

“These certifications [4-5, 19, 40] include fields of medicine [6-7,15,17,20,23-25,27-28,30-31,36], law [13], physics [43], math [16,41], engineering [34], software [14], psychology [32], IQ [39], finance [37], language translation [8,10-12,21,29,44], and general task solving [12,9,18,26,35,38,42].”

“We use the first 1000 AQUA [Zhong et al., 2024] datapoints from the test split as our dataset, and use a 80 / 10 / 10 train/val/test split.”

“We evaluate models’ general performance using MMLU and AGIEval and its unlearning with the WMDP bio and cyber evaluations from Li et al. (2024a).”

“Here, we utilize popular mathematical reasoning benchmarks, namely GSM8K [16], MATH [26], MMLU-STEM [25], AGIEval-SAT-MATH [70], MathQA [3] and AQuA [39] to detect data contamination.”

“By leveraging the different sensitivities of abilities to scaling, we can make models learn specific abilities more efficiently and achieve practical performance improvements on two comprehensive benchmarks, CMMLU (Li et al., 2023) and AGIEval (Zhong et al., 2023).”

“Evaluation Benchmarks and Training Datasets Following (Ma et al. 2025), we construct our evaluation benchmark based on AGIEval (Zhong et al. 2024) and IFEval (Zhou et al. 2023).”

“…et al., 2021a; Huang et al., 2023; Kwiatkowski et al., 2019; M-A-P et al., 2025; Romanou et al., 2025; Wang et al., 2024; Wei et al., 2024; Zhong et al., 2024); reasoning (Cobbe et al., 2021; Dua et al., 2019; Sap et al., 2019; Suzgun et al., 2023; Zellers et al., 2019); and code (Austin…”

“• Aggregate: MMLU-Redux (Hendrycks et al., 2020)(5-shot), MMLU-Pro (Wang et al., 2024b)(5-shot), AGIEval en (Zhong et al., 2023)(3-shot), and SuperGPQA (Du et al., 2025)(5-shot).”

“Early efforts, such as MMLU [16], SciEval [44], AGIEval [57], and ChemLLMBench [14], mainly focus on text-based scientific knowledge, while ScienceQA [5] and TQA [24] add multimodal inputs in largely educational settings.”

“We employ 15 datasets spanning six domains for evaluation: • QA: AGIEval (Zhong et al., 2024), ARC-challenge (Clark et al., 2018), MMLU-redux (Gema et al., 2025) • reasoning: BigBench-hard (Suzgun et al., 2023), GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), TheoremQA (Chen et al.,…”