AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“…literature between benchmarking via point-wise metrics on a large number of models and scenarios (Liang et al., 2022; Srivastava et al., 2022; Zhong et al., 2023), and evaluations where models are aggregated post-hoc based on latent factors that capture variations over comparable tasks…”

“…the GPT series (Achiam et al., 2023; Brown et al., 2020), LLaMAs (Touvron et al., 2023a;b), and Gemma (Team et al., 2024), has demonstrated remarkable efficacy in a wide range of NLP tasks (Hendrycks et al., 2021; Srivastava et al., 2023; Stiennon et al., 2020; Yu et al., 2024; Zhong et al., 2024).”

“AGIEval English Zhong et al. (2024) evaluates general and professional reasoning on English-language examination questions drawn from several standardized and professional examinations.”

“We evaluate benchmark accuracy on Minerva Math [99] and 340 retained Gaokao-MathQA questions from AGIEval [100], both of which have deterministic reference answers.”

“5-0.5B-Instruct policies on two test problems where the two methods disagree: one from MMLU-math, and one from SAT-Math.”

“Assessment-based evaluation of LLMs has rapidly become common practice across domains, often by reusing instruments originally developed to measure human knowledge or reasoning (Kasagga et al., 2025; Balunovi´c et al., 2025; Zhong et al., 2024; Du et al., 2025).”

“From these results, we can observe that: (a) As evidenced by the evaluation results and loss curves of MoE-5B, our HyGA achieves consistently improvements over Gated attention across most metrics and training steps, and the overall improvement is significant.”

“AGIE VAL [247] uses human standardized exams, including SAT Math, math competitions, and college-entrance examinations, thereby measuring mathematical reasoning as part of a broader human-test-taking profile.”

“BloomBench operationalizes the cognitive hierarchy through six task families, each corresponding to a level in Bloom’s Taxonomy.”

“We evaluate on nine public English and Chinese math/STEM benchmarks: GSM8K [2], MATH [5], OCW [7], SAT [1], MMLU STEM [4], CodeContests [8], CMATH [18], GaokaoMath-Cloze, and GaokaoMath-QA [19].”

“We evaluate all model checkpoints using the lm-evaluation-harness (Gao et al., 2024) framework on 12 standard downstream benchmarks spanning three capability axes: common-sense reasoning (HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2020), OpenBookQA (Mihaylov et al., 2018)…”

“We train the controller on the training split of MATH-lighteval [8] and evaluate it on the corresponding test split and six additional math benchmarks: MATH500 [8], Omni-MATH [6], MinervaMath [16], AGI-Eval-MATH [37], AMC23 [13], and AIME24.”

“Subsequently, we perform test-time training driven by TEMPO on a test set consisting of BigBenchHard [26], AGI Eval [27], and ZebraLogic [28].”

“These settings differ substantially in structure, difficulty, and the diversity of locally valid continuations, providing a comprehensive assessment of our proposed method.”

“We conduct experiments on seven datasets spanning diverse arithmetic, algebraic, and symbolic reasoning skills: SAT [38], MATH [39], GSM8K [40], MultiA [41], MMLU [42], MAWPS [43], and GPQA [44].”

“• South Korea : Publicly released past examination questions”

“We evaluate Helix on 12 datasets covering diverse reasoning and knowledge-intensive tasks from established benchmarks, including BBH 1 (Suzgun et al., 2024), and AQuA-RAT 5 (Ling et al., 2017).”

“Finally, we consider two STEM benchmarks: MMLU-STEM (English) (Hendrycks et al., 2021a) and Gaokao2024 (Chinese)(Zhong et al., 2023).”

“We evaluate the transferability of this structured logic to general domains using BBH [25], AGIEval [39], GSM-8k [4], MATH [10], and MMLU-Pro [34].”

“To evaluate generalization beyond the training domain, we additionally report results on a suite of out-domain math reasoning benchmarks, including SAT Math (Zhong et al., 2024), AMC23 (math ai, 2023), AIME24 (Jia, 2024), and OLYMPIAD Bench (He et al., 2024).”

“AGIEval-MATH [29] evaluates mathematical problem solving with challenging questions from standardized exams.”

“…(Hendrycks et al., 2021), IFEval (Zhou et al., 2023), MedText (Melamud and Shivade, 2019), and the Gaokao Chinese subset of AGIEval (AGIEval_G) (Zhong et al., 2023)—covering mathematical reasoning, multi-domain knowledge, instruction following and other general-purpose knowledge and abilities,…”

“To highlight both domain-specific performance and generalization, we evaluated our suite of trained models (ORION 1.5B) on mathematical reasoning in the domain and out-of-the-domain tasks such as GPQA, LSAT, and MMLU.”

“…2021a) x x 5-shot scoring ARC (Challenge) (Clark et al., 2018) x x 0-shot scoring GSM8K (Cobbe et al., 2021) x x 8-shot sampling AGIEval (English) (Zhong et al., 2024) x 3-5-shot sampling BBH (Suzgun et al., 2023) x x 0-shot sampling Winogrande (Sakaguchi et al., 2021) x x scoring HellaSwag…”

“BIG-bench, MMLU, and AGIEval probe diverse skills and exam-style knowledge [10]–[12].”

“We further evaluate DPA-GRPO on public datasets targeting more general multi-answer scenarios, including Mmlu-Multi-Answer [40] and JEC-QA [41].”

“…and training sets, BBH Causal Judgement [15]: a task from the Big-Bench Hard suite focusing on textual causal reasoning based on moral, intentional, or counterfactual analysis, and AGIEval-MATH [23]: a subset of the AGIEval benchmark, focusing on mathematical problem solving skills and reasoning.”

“…Datasets natively constructed from Chinese educational corpora refer to benchmarks directly collected from Chinese educational text resources, such as C-Eval (Huang et al., 2023), Edubench (Xu et al., 2025), Scieval (Sun et al., 2024), AGIEval (Zhong et al., 2023), and SuperCLUE (Xu et al., 2023).”

“The major problems in the questionnaire are constructed based on the AGI-Eval [97] benchmark, covering multiple domains including mathematics, medicine, computer science, humanities, and history.”

“We evaluate performance across various benchmark categories: General (AGIEval (Zhong et al., 2024), RACE (Lai et al., 2017), SQuAD2.”

“Many probes on this question often focus on large composite benchmarks such as MMLU (Hendrycks et al., 2021) and AGIEval (Zhong et al., 2023), without consideration on how specific questions or tasks map into some latent notion of intelligence.”

“We evaluate HKGAI-V1 on four benchmarks — MMLU[18][50], AGI-Eval[62], Flames[22], and Beaver-zh-hk[20] — selected for their coverage of general knowledge, reasoning skills, real-time contextualization, and regional value alignment.”

“Performance We test our training method on the following popular benchmarks: ARC-C (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), SuperGLUE (Sarlin et al., 2020), MMLU-Redux (Gema et al., 2024), AGIEval (Zhong et al., 2023), BIG-Bench Hard (BBH) (Suzgun et al., 2022) and MedQA (Jin et al., 2021) with eight…”

“Generalization to Public Datasets Given that our dataset focuses on multi-answer spatial plausibility reasoning, we further evaluate DPA-GRPO on public datasets targeting more general multi-answer scenarios, including Mmlu-Multi-Answer [14] and JEC-QA [36].”

“Notable datasets include MMLU (Hendrycks et al., 2021), Xiezhi (Gu et al., 2024), and AGIEval (Zhong et al., 2023).”

“…(Rein et al., 2024), FOLIO Han et al. (2022), CommonsenseQA(CSQA) (Talmor et al., 2019), MUSR (Sprague et al., 2024b), MUSIQUE (Trivedi et al., 2022), the AR split of the AGIEval-LSAT (Zhong et al., 2024), the level 3 abductive and level 4 deductive reasoning from contexthub (Hua et al., 2024).”

“Notable examples in this category include AGIEval [29], MMLU-Redux [30], C-Eval [31], and SciKnowEval [32], which evaluate models through exam items covering diverse subjects and varying levels of difficulty.”

“Notably, ACLUE (Zhang & Li, 2023) evaluates ancient Chinese language ability, and AGIEVAL (Zhong et al., 2024) offers evaluation in both Chinese and English for various standardized competitions and exams.”

“Large language models (LLMs) have made remarkable strides in recent years [1–3], achieving near-human or even superhuman performance in various tasks, including traditional natural language processing [4], code generation [5], mathematical reasoning [6], and professional aptitude tests [7].”

“The following benchmark descriptions are taken from Li et al. (2024b) • AGI Eval LSAT-EN (Zhong et al., 2023) tests for model knowledge in the legal domain and evaluates analytical reasoning capabilities.”

“BigBench Hard[5] and AGIEVal[12] are used to measure general knowledge and English comprehension skills, respectively: 1, 2, 3,.”

“Specifically, we use gpt-4o-mini-2024-07-18 and gpt-4o-2024-08-06 (Ope-nAI, 2023) from OpenAI, and claude-3-5-sonnet-20240620 (Anthropic, 2023) from Anthropic.”

“AGIEval [14] and C-Eval [15] focus on standardized examinations and advanced knowledge evaluation in Chinese, respectively.”

“For both tasks testing reasoning abilities (AGIeval and BigBench), Orca (13B) retained up to 88% performance of ChatGPT, outperforming other models like the Vicuna 13B [49, 41].”

“Large language models (LLMs), particularly those designed for conversational tasks, have achieved state-of-the-art (SOTA) performance in a wide range of applications, including casual conversation, question answering, and personalized dialogue (Zhong et al., 2023; Liang et al., 2022).”

“This restricts the benchmark's capacity to assess the model's robustness to diverse prompting styles [6], [28].”

“While we draw parallels between reliance on shortcuts and fast thinking within human cognition, some research within the NLP field argues that such dependency on shortcuts (dataset biases) detracts from the models’ relevance to human-level cognition [Zhong et al. , 2023].”

“• AGIEval [ 9 ] AGIEval, a new benchmark developed by Microsoft’s Chinese research team, is specifically designed to evaluate foundational models’ human-like capabilities, particularly focusing on their performance in tasks related to human cognition and problem-solving.”

“CLUE (Xu et al., 2020) exemplifies traditional task-oriented evaluations, while datasets such as AGIEval (Zhong et al., 2023) offer a human-centered alternative assessment.”

“The second category includes subject-oriented benchmarks such as GAOKAO-Bench (Zhang et al., 2024), AGIEval (Zhong et al., 2023) and Xiezhi (Zhouhong et al., 2023).”

“…2019), HellaSwag (Zellers et al., 2019), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), Hu-manEval (Chen et al., 2021), and AGIEval (Zhong et al., 2023): text-and-image, such as MMMU (Yue et al., 2023), TextVQA (Singh et al., 2019), MathVista (Lu et al., 2024), MME (Fu et al.,…”

“AGIEval [39]: This benchmark selects 20 official, public, and high-standard qualification exams aimed at general human candidates, including general college entrance exams (such as China’s Gaokao and the US SAT), judicial examinations, math competitions, and so on.”

“To build an empathetic companion, we further finetune C-SFT 70B with emotional supporting conversation (ESC) samples, in both English and Chinese.”

“Even the arguably most powerful LLM, GPT-4, still exhibits hallucinations (OpenAI, 2023) and its performance in the real world legal tasks is still limited (Shui et al., 2023; Zhong et al., 2023; Martinez, 2023).”

“…coverage, including math-focused settings where verifiers, chain-of-thought prompting, and program-aided strategies reduce intermediate reasoning errors and improve reliability [20], [21], as well as broader multi-subject suites designed to probe general knowledge and reasoning [22], [23], [24].”

“• AGIEval (Zhong et al., 2024): A human-centric benchmark derived from high-standard official admission and qualification exams.”

“Factual evaluation of LLMs has garnered academic attention, leading to the development of several representative benchmarks (Chern et al., 2023; Zhong et al., 2024; Huang et al., 2023; Srivastava et al., 2022; Zhang et al., 2025b).”

“Despite AI models’ exceptional ability to solve a wide range of tasks, they nevertheless struggle with real-world issues that call for sophisticated mathematical computations or strong reasoning skills, for instance (Chang et al., 2024; Zhong et al., 2023).”

“2) There are specialized benchmarks designed for multilingual evaluation, including those targeting non-English languages (Huang et al., 2023b) as well as bilingual settings (Zhong et al., 2024).”

“Our study separates itself from previous studies in the research line of literary Chinese RC (Xu et al., 2020; Zhou et al., 2023; Zhong et al., 2024; Hou et al., 2024; Wei et al., 2024) by focusing on the questions that involve interpretation and processing of language (Rao et al., 2023).”

“Multiple-choice tasks, a common yet important task format in the research of LLMs [56, 19, 58, 57], provide a controlled setting for investigating the influence of error, with clear criteria for correctness facilitating analysis of the impact of different error types on performance.”

“Tasks are grouped into several categories, including: (a) general knowledge/reasoning (ARC (Bhak-thavatsalam et al., 2021), AGIEval (Zhong et al., 2024), OpenBookQA (Mihaylov et al., 2018), BBH (Suzgun et al., 2023), WorldSense (Hong et al., 2025), PIQA (Bisk et al., 2020), hellaswag (Zellers et al., 2019) and KOR-Ben…”

“While several existing LLM benchmarks [18, 42, 4] have incorporated scientific questions into their evaluations, and some benchmarks [27, 29, 3, 30, 20, 11] are specifically tailored for the scientific domain, we argue that the current benchmarks do not fully evaluate the potential of LLMs in…”

“Ideally, such agents should seamlessly integrate real-world interaction with verbal reasoning, and continuously learn from experience to improve themselves, much like human intelligence [29, 72, 86, 132].”

“The benchmarks are: MMLU (Hendrycks et al., 2021b), MMLU-Pro (Wang et al., 2024b), AGIEval (Zhong et al., 2024), CommensenseQA (Talmor et al., 2019), ARC-Challenge (Clark et al., 2018), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021c), HumanEval (Chen et al., 2021), and MBPP (Austin et al., 2021).”

“These multi-agent systems (MAS) integrate various techniques, including CoT prompting (Wei et al., 2022), iterative refinement (Shinn et al., 2023), self-improvement (Huang et al., 2023), and external tool usage (Hao et al., 2023; Qin et al., 2024; Schick et al., 2023), to support multi-step…”

“Thus, our techniques would equally work well for other evaluations [17, 43, 12, 36, 44, 8, 22, 21] as well as cost and latency prediction.”

“While LLMs excel in many scenarios, their reliance on fast, intuitive thinking often falls short when faced with complex reasoning challenges, such as advanced mathematics [6, 235] and coding tasks.”

“These included non-proving problems from OlympiadBench (He et al., 2024), mathematical problems from AGIEval (Zhong et al., 2023) benchmark, and various problems in calculus, differential, statistics domains from SciBench (Wang et al., 2024) and JEEBench (Arora et al., 2023).”

“For OOD evaluation, we use BB-Sub filtered by CasCoD, and AGIEval (Zhong et al., 2023) sub-tasks about English multiple-choice questions.”

“Notable benchmarks in this category include MMLU [9], HellaSwag [17], GSM-8K [18], HELM [19], BigBench [12], AGIEval [20], HumanEval [21], and ToxicChat [22].”

“Table 1 reports performance on MMLU [8], MMLU-Pro [31], AGIEval [37], WinoGrande [22], BBH [28], DROP [6], GSM8k [4], MATH [9], HumanEval [3], MBPP+ [16], and TriviaQA [12].”

“Qwen3-8B (Yang et al., 2025) et al., 2025), and AGI-Eval (Zhong et al., 2024) and knowledge on MMLU (Hendrycks et al., 2020) and PopQA (Mallen et al., 2023).”

“To verify that LLMON post-training preserves general capabilities, we evaluate across seven categories, reporting the unweighted mean over the benchmarks in each: General (MMLU (Hendrycks et al., 2021), MMLU-Pro (Wang et al., 2024), BBH (Suzgun et al., 2023), AGI-Eval (Zhong et al., 2024), GPQA (Rein et al., 2023)); A…”

“…PaperText, and FetaTab (Hui et al., 2024)), mathematical reasoning (AQuA (Ling et al., 2017), MATH-500 (Lightman et al., 2024), and AGIEval-MATH (Zhong et al., 2024)), code generation (HumanEval-ET (Dong et al., 2025) and MBPP (Austin et al., 2021)), and video understanding (LongTVQA+ (Liu et…”

“The extended suite includes SciQ (Welbl et al., 2017), SocialIQA (Sap et al., 2019), MuSR (Sprague et al., 2023), AGIEval (Zhong et al., 2023), GSM8K-Platinum (Vendrow et al., 2025), GSM-Plus (Li et al., 2024), MathQA (Amini et al., 2019), ASDiv (Miao et al., 2020) and both CRUXEval tasks (Gu et al., 2024).”