AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“This framework assesses performance across four key dimensions, utilizing the following standard benchmarks and their respective metrics: • General Ability: MMLU (Massive Multitask Language Understanding) (Hendrycks et al., 2021b,a) and AGIEval (Zhong et al., 2023).”

“AGI-Eval (Zhong et al., 2023) and C-Eval (Huang et al., 2023) target general-purpose, multi-disciplinary evaluation for LLMs and contain subsets specifically designed to assess mathematical abilities.”

“For pre-trained models, we evaluate on: • Knowledge: BBH (Suzgun et al., 2023), PIQA (Bisk et al., 2019), SCIQ (Welbl et al., 2017), SIQA (Sap et al., 2019) • Reasoning: ARC (Clark et al., 2018), BoolQ (Clark et al., 2019), CLUE (Xu et al., 2020), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), NaturalQA (K…”

“In this section, following the approach of previous benchmark constructions [18, 83, 88], we outline the three key steps used to build U2-B ENCH : (1) data collection and sampling (2) data cleaning, format unification and quality verification, and (3) task-specific prompt design.”

“The Generative Pretrained Transformer 4 (GPT-4) now surpasses an average human’s performance on various standardized tests, such as the Law School Admission Test (LSAT) and the SAT, and math competitions, including the American Mathematics Competitions and the American Invitational Mathematics Examination (Zhong et al…”

“Examples include GSM8K at elementary school level, Math and GaokaoBench-Math (Zhang et al., 2023) at high school level , College Math (Tang et al., 2024), TheoremQA (Chen et al., 2023), ARB (Sawada et al., 2023) at university level, and OlympiadBench (He et al., 2024), AGIeval-Math (Zhong et al., 2023) at competition…”

“AGIEval (Zhong et al., 2023) is a benchmark designed to evaluate foundational models in the context of human-centric standardized exams.”

“Prior research indicated that ChatGPT could underperform in question-answering scenarios if the instructions were not appropriately optimized [71].”

“Most work on ChatGPT has considered canonical NLP problems (Zhang et al., 2023; Liu et al., 2023; Zhong et al., 2023).”

“Methodologically,mostevaluationstudiesfocused onevaluatingthefunctionalcapabilities[20]andpsychological traits[21],[22],[23]ofLLMs,usingestablishedbenchmarks [24],[25],[26],andwell-designedscales[22],[27].”

“1: Mistral 7B (Jiang et al., 2023) is an open-source model that outperforms Llama2 13B on LLM benchmarks, including AGI Eval (Zhong et al., 2023)”

“…literature between benchmarking via point-wise metrics on a large number of models and scenarios (Liang et al., 2022; Srivastava et al., 2022; Zhong et al., 2023), and evaluations where models are aggregated post-hoc based on latent factors that capture variations over comparable tasks…”

“…we followed the lm-evaluation-harness [Gao et al., 2023] and evalute model on MMLU [Hendrycks et al., 2020], BBH [Suzgun et al., 2022], AGIEval [Zhong et al., 2023b], ARC-Easy and ARC-Challenge [Clark et al., 2018], BoolQ [Clark et al., 2019], PIQA [Bisk et al., 2020], Hellaswag [Zellers et…”

“Evaluates models’ ability on diverse domains of tasks MMLU [71], MMMU [249], MMStar [32], M3GIA [197], AGIEval [263]”

“…al., 2025) have demonstrated exceptional performance across a wide range of natural lan-* guage tasks (Hendrycks et al., 2020; Huang et al., 2023b; Zhong et al., 2024b) but often fail in short in domain-specific applications (Gururangan et al., 2020; Jia et al., 2025; Zhang et al., 2025; Susnjak…”

“As detailed in Section 3.2, we selected problems from OlympicArena (Huang et al., 2024b), AGIEval (Zhong et al., 2023), LiveCodeBench v4 (Jain et al., 2024), NuminaMATH (LI et al., 2024), and OmniMath (Gao et al., 2024).”

“These tasks include TruthfulQA (Lin et al., 2021), MMLU-Pro (Wang et al., 2024b), Big-BenchHard (Suzgun et al., 2022), AGIEval (Zhong et al., 2023), GSM8K (Cobbe et al., 2021), and MATH (Hendrycks et al., 2020).”

“To address these issues, we plan to optimize the data distribution in the future, open source multilingual datasets adapted for different regions, and update the data regularly.”

“…for the code domain is Pass@1 of HumanEval (Chen et al., 2021); and the performance for the math domain is the average result of GSM8K (Pass@1) (Cobbe et al., 2021), MATH (Pass@1) (Hendrycks et al., 2021), SAT-Math (Acc) (Zhong et al., 2023), and AQuA-RAT (Acc) (Ling et al., 2017) datasets.”

“Our model outperforms other models on MMLU, AGIEval and CMMLU benchmarks, even surpassing the Qwen-72B with a much larger parameter size.”

“The calculation capability of Yuan 2.0 is evaluated on the GSM8K [51] and the GaoKao-Math task in AGIEval [52].”

“2 11B [6] on popular multimodal benchmarks like MMMU [24] and MathVista [14], while outperforming most open-source models on popular text-only tasks like MATH [7] and HumanEval [26].”

“5 and GPT 4.0 developed by OpenAI were trained to prioritize not promoting discrimination against specific groups such as race and gender and to approach sensitive topics in a manner that safeguards human well-being (Ouyang et al., 2022; OpenAI, 2023; Zhong et al., 2023; Cheng et al., 2023a).”

“We take the 19-rule Item-Writing Flaws rubric from Tarrant et al. (2006), which avoids subjective rules in others (Haladyna and Downing, 1989, e.g., “ avoid trivial material ”) and has been used for over two decades across domains in education—mostly higher education (Schmucker and Moore, 2025).”

“These shortcomings are particularly stark when compared to standardized human assessments like the SAT, GRE, or bar examinations, which are proctored, regularly updated, and governed by rigorous procedures to uphold fairness, reliability, and data integrity [43, 54].”

“The post-training benchmarks can be categorized into several dimensions: • General Tasks: For general language understanding tasks, we utilize various benchmarks including MMLU (Hendrycks et al., 2020), MMLU-Pro (Wang et al., 2024a), CMMLU (Li et al., 2023), SuperGPQA (Du et al., 2025), BBH (Suzgun et al., 2022), DROP…”

“• AGI Eval [23] includes questions from human-centric standardized exams such as the SAT and LSAT.”

“AQuA-RAT [61] focuses on algebraic word problems from the AQuA-RAT dataset, requiring rationales and supporting steps.”

“As shown Further, we evaluate each model on the standard MMLU (Hendrycks et al., 2020), AGIEval English (Zhong et al., 2024), Hellaswag (Zellers et al., 2019), and ARC-Challenge dataset Clark et al. (2018) and report their results in Table 12.”

“JEEBench 56 draws on pre-engineering-level scientific problems from college entrance exams, and AGIEval 57 assesses LLM performance on standardized exams such as college entrance and lawyer qualification tests.”

“Also, real-world natural data [67] or tasks for real-world situations [119]in various linguistic contexts should be gathered with great importance.”

“In response, researchers are developing more challenging benchmarks that use human experts as the standard, such as GPQA (Rein et al., 2023) and AGIEval (Zhong et al., 2023), or employ real-world use cases for testing, such as SWE (Jimenez et al., 2024) and BigCodeBench (Zhuo et al., 2024).”

“As for open-source LLMs, there is a significant gap in parameter size and pre-training corpus compared to closed-source LLMs, resulting in significantly weaker general capabilities such as reasoning, and domain-specific knowledge extraction [9],[17],[18],[19].”

“This has led to the creation of many general benchmarks such as GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019), MMLU (Hendrycks et al., 2021), Super-NaturalInstructions (Wang et al., 2022a), HELM (Liang et al., 2022), and AGIEval (Zhong et al., 2023).”

“∗ Equal contribution † Corresponding Author benchmarks that comprise common standardized exams, such as AGIEval (Zhong et al., 2023), CEval (Huang et al.”

“AGIEval [46] is a bilingual benchmark that measures the capability of LLMs on tasks of the Chinese college entrance exam and American college admission test.”

“Early textual eval-Figure uations such as AGIEval [65], ARC [5], and MMLU [15] focus on standardized exams, while MMLU-Pro [51] introduces higher-order reasoning.”

“AGIEval SAT Math [44] is a benchmark derived from standardized exams such as the U.S. SAT.”

“1, for instance, achieves more than 90% accuracy on MMLU—new benchmarks continue to emerge, offering fresh evaluation challenges (Zhong et al., 2024; Phan et al., 2025; Guha et al., 2023).”

“Cloud LLM MATH-lighteval MATH-500 AMC23 MinervaMath AGI-Eval-Math Avg.”

“3) SAT-MATH (Zhong et al., 2024): Consisting of 102 questions, this dataset includes math problems from the U.S. high-school curriculum.”

“We evaluate on the centered core metric from DCLM, which consists of the following 22 tasks: AGI Eval LSAT Analytical Reasoning [33], ARC Challenge [5], ARC Easy [5], BIG-bench cs algorithms [28], BIG-bench dyck languages [28], BIG bench language identification [28], BIG-bench operators [28],…”

“These benchmarks range from general purpose question answering (e.g. MMLU (Hendrycks et al., 2020), AGIEval (Zhong et al., 2024)) to specialized benchmarks testing specific capabilities such as mathematics (Cobbe et al., 2021; Glazer et al., 2024), coding (Chen et al., 2021; Jimenez et al., 2024), scientific knowledge…”

“5% , and on AGIEval-Math (Zhong et al., 2023) from 23 .”

“One of the well-established benchmarking areas is question answering (QA), which tests the LM’s ability to apply knowledge acquired from diverse domains to answer user questions (Kwiatkowski et al., 2019; Hendrycks et al., 2021; Zhong et al., 2024).”

“Typical benchmarks include AGIEval (Zhong et al. 2023) for evaluating their ability of tackling human-level tasks, BIG-Bench (Srivastava et al. 2022) for measuring and extrapolating their capabilities (measured for 204 tasks at this moment), and MMLU (Hendrycks et al. 2020) for quantifying their multitask language und…”

“To this end, a large number of benchmarks specifically designed for LLMs, e.g., (Wang et al., 2019; Xu et al., 2020, 2023; Zhang et al., 2023b; Zhong et al., 2023; Huang et al., 2023; Li et al., 2023a; Suzgun et al., 2023; Gu et al., 2024), were proposed.”

“…2023), and the dataset includes commonly used datasets: C-Eval (Huang et al., 2024) and CMMLU (Li et al., 2023a) for Chinese knowledge, AGI-Eval (Zhong et al., 2023) for World Knowledge, MMLU (Hendrycks et al., 2020) for English knowledge, HumanEval (Chen et al., 2021) and MBPP (Austin et al.,…”

“…et al., 2019), MMLU (5-shot) (Hendrycks et al., 2021), and CMMLU (5-shot) (Li et al., 2023a), while the generative tasks include AGIEval (0-shot) (Zhong et al., 2023), Big-Bench Hard (BBH) (3-shot) (Suzgun et al., 2023), Human-Eval (0-shot) (Chen et al., 2021), and Gsm8k (5-shot) (Cobbe et al.,…”

“A single-task attack is optimized and tested on one specific task category while a multi-task attack is supposed to be effective on all task categories.”

“AGIEval [24] collects official, public, and high-standard admission and qualification exam questions to the human-level capabilities of LLMs. C-Eval [25] is a comprehensive Chinese evaluation suite and contains 13,948 multi-choice questions, including middle school, high school, college, and…”

“Currently, commonly used methods include Retrieval Augmented Generation (RAG) and Chain of Thought (CoT) techniques to guide the model to accomplish specific tasks.”

“Existing benchmarks like AGIEval [Zhong et al., 2023], ChenLLMBench [Guo et al., 2023], SCIEval [Sun et al., 2023], PIXIU [Xie et al., 2023a], and MME [Fu et al., 2024] provide comprehensive datasets for evaluating LLMs.”

“The evaluation of large language models has made significant strides in recent years, with researchers developing numerous benchmarks aimed at testing various aspects of model performance [12, 19, 38].”

“GPT-4 (Achiam et al., 2023), LLaMA (Touvron et al., 2023a,b), Gemini (Team et al., 2023), Minerva (Lewkowycz et al., 2022), Llemma (Azerbayev et al., 2023), Mistral (Jiang et al., 2023), WizardMath (Luo et al., 2023), MAM-MOTH (Yue et al., 2023), ToRA (Gou et al., 2023) and Deepseek (Bi et al., 2024; Guo et al., 2024;…”

“In recent times, Large language models (LLMs) have demonstrated an extraordinary ability to fathom enormous amount of world knowledge [13], [14].”

“More recently, AGIEval [32], C-Eval [10], and M3KE [16] collect various questions from a range of subjects or standardization examinations and convert them into multi-choice question answering to make it easy to calculate accuracy.”

“…to mitigate these challenges by using an intermediary LLM to evaluate generated text (Liu et al., 2023; Ding et al., 2023), designing AI tools to aid data annotation (Gao et al., 2023), or improve metrics and datasets for new LLMs (Jain et al., 2023; Zhong et al., 2023; Sawada et al., 2023).”

“While some studies focus on gathering numerous evaluation tasks [30], others have chosen to evaluate those models on human examinations rather than machine learning benchmarks [112].”

“Primarily, with GPT models, a lack of domain knowledge or analytical reasoning frequently causes fictitious responses to be generated in their overzealous pursuit of a reply [16, 25].”

“Logical Reasoning : We consider the logical reasoning part of the Law school admission test (LSAT) (Zhong et al., 2023).”

“Notable examples include MMLU (Hendrycks et al., 2020), MATH (Hendrycks et al., 2021), GSM-8K (Cobbe et al., 2021), HumanEval (Chen et al., 2021), Big-Bench (bench authors, 2023), HellaSwag (Zellers et al., 2019), and AGIEval (Zhong et al., 2023).”

“…benchmarks, including complex reasoning datasets BBH (Suzgun et al., 2022) and ARC Clark et al. (2018), knowledge-based datasets AGIEval (Zhong et al., 2023) and MMLU (Hendrycks et al., 2021), commonsense reasoning dataset CSQA (Talmor et al., 2019), and mathematical reasoning dataset…”

“…a wide range of tasks and languages to test general knowledge and reasoning (Hendrycks et al., 2020; Liang et al., 2022; Srivastava et al., 2022a; Zhong et al., 2023), where small language models are getting close performance compared to the state-of-the-art large language models Li et al.…”

“Other efforts include the EleutherAI LM Harness for few-shot evaluation (Gao et al., 2021), Liang et al. (2023)’s holistic evaluation framework, AGIEval for human-centric standardized exams (Zhong et al., 2023), and conversational model benchmarks like MT-Bench and ChatbotArena (Zheng et al., 2023).”

“– LSAT-AR [16]: LSAT analytical-reasoning problems from AGIEval and AR-LSAT, – Sci-XW [12]: college-level math, chemistry, and physics problems from SciBench.”

“Furthermore, we assess rigorous application skills using AGIEval (Zhong et al., 2024) on standardized exams, GSM8k (Cobbe et al., 2021) on mathematics, and OpenCompass LCBench (Contribu-tors, 2023) (a collection of programming problems from Leetcode weekly competitions) on code generation.”

“We evaluate our MTP-D on a diverse set of widely used pre-training benchmarks spanning multiple domains, including AGIEval-en (Zhong et al., 2024) for human-centric standardized examinations; GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) for mathematical reasoning; NaturalQues-tions (Kwiatkowski et al.,…”

“• General-purpose QA: AGIEval (Zhong et al., 2024), ARC-challenge (Clark et al., 2018), MMLU-redux (Gema et al., 2025), GPQA (Rein et al., 2024) • Math: GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021a) • Reasoning: BigBench-hard (Suzgun et al., 2023), TableMWP (Lu et al., 2023), TheoremQA…”

“For the General domain, we report results on DROP [18], IFEVAL [82], AGIEVAL [80], and MMLU-Pro [65].”

“For instance, AGIEval (Zhong et al., 2024) uses questions from exams like the SAT and Gaokao, and C-Eval (Huang et al., 2023) focuses on Chinese academic disciplines.”

“For mathematical reasoning, we select 11 diverse datasets: GSM8K [8], MATH [20], GSM8K-Hard [14], SVAMP [36], ASDIV [35], MAWPS [25], MinMATH [20], MMLU-STEM [19], TABMWP [30], MATHQA [1], and SAT [55].”

“The benchmark assesses the ability of code generation models to produce code that is not only syntactically correct but also semantically similar to the intended human-written code [58], [59].”