AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“We further evaluate generalization on the challenging AGIEval-Gaokao (Zhong et al., 2024) benchmark, which spans nine subjects (Biology, Chemistry, Chinese, English, Geography, History, MathCloze, MathQA, Physics).”

“However, existing QA and evaluation benchmarks mainly target general knowledge (Rajpurkar et al., 2016; Joshi et al., 2017; Hendrycks et al., 2021) or domains such as medicine and law (Jin et al., 2019; Zhong et al., 2024), leaving disaster contexts underexplored.”

“(4) AGIEval (Zhong et al., 2023), which includes a wide range of high-quality official entrance exams, qualifying exams, and advanced competitions tailored to human participants.”

“In contrast, standardized exams offer a more reliable framework for evaluating the general cognitive abilities of large foundation models across a range of academic disciplines [23].”

“Some efforts, such as AGIEval (Zhong et al., 2023), explicitly model human-like cognitive testing by evaluating performance on standardized exam questions.”

“We et al., 2020a), Alignment ( TruthfulQA (Lin et al., 2021) and Sycophancy (Perez et al., 2023)), and Intelligence ( OpenBookQA (Mihaylov et al., 2018), ARC Challenge (Clark et al., 2018), and LSAT (Zhong et al., 2023)).”

“Furthermore, some datasets draw directly from actual human exams, offering a realistic portrayal of diverse scenarios such as M3Exam [14], VNHSGE [16], AGIEval [13] and EXAMS [38].”

“To quantify the discrepancy between dynamic and static evaluation, we compare the model rankings from LLMEval-Fair with two representative static benchmarks, AGIEval (Zhong et al., 2024) and C-Eval (Huang et al., 2023).”

“…on a wide range of OOD datasets beyond GSM8K, including other math datasets MATH (Hendrycks et al., 2021 (Ling et al., 2017), and general reasoning datasets MMLU (Hendrycks et al., 2020), BBH (Suzgun et al., 2023), ARC-Challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018).”

“We select several representative reasoning benchmarks: AIME24, AMC23, AIME25, Math-500 (Hendrycks et al. 2021b), OlympiadBench (He et al. 2024), MMLU-STEM (Hendrycks et al. 2021a), GPQA diamond (Rein et al. 2023), and GaoKao Math QA (Zhong et al. 2024).”

“This section outlines the five-step construction of SNS-B ENCH -VL, following prior benchmarks [77, 78, 79], and presents its distributional analysis.”

“The development of so-called “challenge sets”, initially explored in machine translation (Goyal et al., 2022) but now extended to LLMs (Zhong et al., 2024) also falls into this space.”

“Numerous high-quality benchmarks—such as GLUE [13], SuperGLUE [14], Big-Bench [15], GSM8K [9], MATH [10], C-Eval [16], and AGIEval [17]—have been introduced to test LLMs on different dimensions.”

“We conduct evaluations on a total of seven publicly available datasets in the field of mathematical reasoning, including both in-distribution (IND) ( Conic10K , MATH , GSM8k ) and OOD ( MMLU-pro-Math (Wang et al. 2024d), JEEBench-Math (Arora, Singh, and Mausam 2023), SciBench (Wang et al. 2024c), SAT-Math (Zhong et al…”

“Additionally, we also test all methods on AGI Eval (Zhong et al., 2024), an aggregated benchmark dataset designed to assess overall model capabilities.”

“AGIEval (Zhong et al., 2024) is another benchmark that uses tasks from general college admission tests to simulate real-world AI assistant use cases.”

“The chosen datasets include OpenBookQA [25], ARC Challenge [5], SciQ [26], AGI Eval Gaokao English [27] and AGI Eval SAT Math [27]; while the tested LLMs are Mistral 7B v1.”

“Performance on Downstream Tasks Furthermore, we investigate the performance of different training paradigms across nine downstream tasks, including C 3 (Sun et al., 2020), GSM8K (Cobbe et al., 2021), MMLU (Hendrycks et al., 2021), CSL (Li et al., 2022), C-EVAL (Huang et al., 2023), BBH (Suzgun et al., 2023), CMMLU (Li…”

“AGIEval (Zhong et al., 2023) includes standardized tests and civil service exams to assess reasoning and problem-solving skills in academic and professional scenarios.”

“…knowledge tests (Hendrycks et al., 2020; Liu et al., 2020; Cobbe et al., 2021; Zellers et al., 2019; Clark et al., 2018; Lin et al., 2021), complex logical reasoning (Cobbe et al., 2021), school admission tests (Zhong et al., 2023) and fine-grained "skill sets" evaluation (Ye et al., 2023).”

“AGIEval Zhong et al. (2023) is an exam dataset designed for general human test-takers, including entrance exams for colleges (like China’s Gaokao and the US SAT), law school entrance exams, math competitions, and U.S. civil service exams.”

“…assessing ChatGPT’s performance on existing NLP tasks (Pan et al., 2023; Wang et al., 2023b; Hendy et al., 2023; Zhu et al., 2023; Yang et al., 2023; Gao et al., 2023; Yuan et al., 2023), and proposing many new evaluation frameworks (OpenAI, 2023; Zhong et al., 2023; Kocmi and Federmann, 2023).”

“Furthermore, even some of those benchmarks [20] provide different difficulty levels; however, these difficulty levels are applied to a per-dataset granularity, which means the overall difficulty of the entire question dataset while not a specific question.”

“Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks (Qin et al., 2023a; Jiao et al., 2023; Zhong et al., 2023; Wang et al., 2023; Liang et al., 2023; Zhang et al., 2024c; Li et al., 2024b; Guo et al., 2023; Zhang et al., 2024b).”

“While general benchmarks like AGIEval [26] attempt to capture these nuances, they lack the domain-specific logic structures central to legal analysis, such as the IRAC (Issue, Rule, Application, Conclusion) framework.”

“For example, GPT-4o reached up to 80% accuracy on AQuA, which still lags behind typical human performance of 85% [46].”

“…such as GPT-4 (OpenAI, 2023), LLaMa-2 (Touvron et al., 2023), and Gemini (Team et al., 2023), successfully achieved strong zero-shot/few-shot performance on various natural language tasks (Qin et al., 2023a; Frieder et al., 2023; Kojima et al., 2022; Zhong et al., 2023; Hendrycks et al., 2020).”

“For our initial variance analysis, we use both benchmark-level scores (to compute variance and monotonicity) and sample level scores (to estimate 95% confidence intervals).”

“While LLMs have been shown to excel at these tasks (Wang et al., 2022; Zhong et al., 2023), there is also work demonstrating bias in using LLMs as evaluators (Wang et al., 2024; Zheng et al., 2023).”

“We evaluate our models on test sets of 4 different reasoning datasets: AIME 2025, MATH (Hendrycks et al., 2021b), AMC, Olympiad-Bench (He et al., 2024), and additionally GPQA (Rein et al., 2023), LSAT (Zhong et al., 2023), and MMLU (Hendrycks et al., 2021a).”

“Leaderboards based on multiple choice questions (MCQ) for evaluation (Wang et al., 2018, 2019; Nie et al., 2019; Zhong et al., 2023; Hendrycks et al., 2020) present both convenience and significant limitations (Pezeshkpour and Hr-uschka, 2023; Zheng et al., 2023).”

“In Figure 12, we present several examples of tokens selected by the SLM method, with content marked in blue indicating the tokens actually chosen during the pretraining process.”

“The distributions of B and B ′ are illustrated in Figure 8.”

“We use the GSM8k (Cobbe et al., 2021), MMLU (Hendrycks et al., 2020), and AGIEval (Zhong et al., 2023) to evaluate their general capabilities and show the average scores ( ↑ ).”

“We shuffle the order of the questions, presenting one generated and one real question side by side, and ask two psychological annotators to judge which was better or if there was a tie.”

“For our dataset for reinforcement learning training we combine public sources, specifically SkyWork (He et al., 2025a) and s1 (Muennighoff et al., 2025) (s1 further sources from Numi-naMATH (Li et al., 2024a), MATH (Hendrycks et al., 2021), OlympicArena (Huang et al., 2024), OmniMath (Gao et al., 2024), AGIEval (Zhong…”

“…2021; Wang et al., 2024), commonsense (Zellers et al., 2019; Bisk et al., 2020; Sap et al., 2019; Huang et al., 2019; Clark et al., 2019), factuality (Lin et al., 2022b), standardized exams (Zhong et al., 2024), mathematics (Amini et al., 2019), and multilingual reading (Bandarkar et al., 2024).”

“General benchmarks like C-EVAL 15 , AGI-Eval 16 ,andMMLU 17 canbeusedtoassessLLMsondiversetasksbutdo not cater speci fi cally to mental health.”

“We evaluate reasoning and logic capabilities across reasoning, math and logic benchmarks with text and multimodal datasets: Text: (1) AGIEval (Zhong et al., 2023): 8269 problems from college admission tests, law school admission tests, math competitions, lawyer qualification tests, and national…”

“Data and Evaluation We evaluate models and systems across 5 domains and 15 tasks in total: general QA ( AGIEval [26], ARC-challenge [27], MMLU-redux [28]), reasoning ( BigBench-Hard [29], GSM8k [30], MATH [31], Sciriff [32]), knowledge ( WikiDYK [33], PopQA [34], BLEND [35]), safety ( TruthfulQA…”

“…OpenBookQA (Mihaylov et al., 2018), LAMBADA (Paperno et al., 2016), Hel-laSwag (Zellers et al., 2019 (Sakaguchi et al., 2019), AGIEval LSAT-AR (Zhong et al., 2023), SQuAD (Rajpurkar et al., 2016), CoQA (Reddy et al., 2019), BoolQ (Clark et al., 2019), and the BIG-bench QA WikiData, Language…”

“AGIEval-LSAT-RC, AGIEval-LSAT-LR, and AGIEval-SAT-EN [47] evaluate reading comprehension, logical reasoning, and SAT-style English respectively, with 268 , 510 , and 206 four-choice examples.”

“The magnitude of bias varies substantially across datasets, with HLE and TruthfulQA showing particularly strong in-group favoritism for Qwen3-8B.”

“For contextual understanding, we evaluated models on 1000 passage-based inference questions drawn from the SAT English section (Zhong et al., 2023).”

“AGIEval (Zhong et al., 2024) evaluates foundation models in human-centric standardized exams, including college entrance tests, math competitions, and legal qualification tests.”

“…[ Kwiatkowski et al. , 2019 ] , TriviaQA [ Joshi et al. , 2017 ] , and DROP [ Dua et al. , 2019 ] ); AGIEval , a suite of college-level academic reasoning tasks [ Zhong et al. , 2023 ] ; and BBH , a collection of challenging multi-step reasoning tasks from BIG-Bench [ Suzgun et al. , 2022 ] .”

“The first, automated benchmarks, provides essential measures of technical capability through standardised tests like MMLU (Hendrycks et al., 2021), HELM (Liang et al., 2022), AGIEval (Zhong et al., 2023), and BIG-Bench (Srivastava et al., 2022).”

“Knowledge question answering typically involves reading comprehension and open-domain question answering, with benchmarks such as AGIEval utilizing large-scale standardized exams (e.g., Gaokao, SAT) to assess general capabilities [82].”

“Factual evaluation of LLMs has garnered academic attention, leading to the development of several representative benchmarks (Chern et al., 2023; Zhong et al., 2024; Huang et al., 2023; Srivastava et al., 2022; Zhang et al., 2025b).”

“Similarly, Chinese benchmarks like E-Eval (Hou et al., 2024), AGIEval (Zhong et al., 2024), and C-Eval (Huang et al., 2023) are still dominated by low level cognitive questions and single-turn formats, lacking systematic evaluation of higher-order abilities and providing limited diversity in task…”

“…answering, or summarization (e.g., XTREME (Hu et al., 2020b), XGLUE (Liang et al., 2020), MEGA (Ahuja et al., 2023)), while others emphasize factual knowledge access in multiple languages, such as M3Exam (Zhang et al., 2023), AGI-Eval (Zhong et al., 2023), and Global-MMLU (Singh et al., 2025).”

“The knowledge-infused ACER -trained models improved by more than 2 absolute points in ARC and GPQA, both of which emphasize knowledge recall and domain understanding, while maintaining stable performance on general reasoning, arithmetic, and common sense tasks such as AGIEval, GSM8K, and HellaSwag.”

“The evaluation framework is organized into distinct task categories, including: (a) General Knowledge ( e.g., ARC (Bhaktha-vatsalam et al., 2021), AGIEval (Zhong et al., 2024), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019)) (b) Language Understanding ( e.g., RACE (Lai et al., 2017)) (c) Professional Knowl…”

“Therefore, our experiment reproduces the training regime up to the 5B token mark in [4].”

“As a result, mathematical reasoning has become a key benchmark for evaluating the progress of Large Language Models (LLMs) toward Artificial General Intelligence (AGI) (Zhong et al., 2024).”

“…96k, discrete reasoning link PlanBench [40] 2022 ASU 11k, planning-specific link MuSR [41] 2023 Zayne Sprague 756 tasks, novel format link AGIEval [42] 2023 Microsoft Exam-style, static link ScienceQA [43] 2022 UCLA Multi-modal, vision bias link e-CARE [44] 2022 HIT 21k, causality specific link…”

“Chen et al. [22] examined model performance in high-stakes domains, including national licensing exams for lawyers, doctors, and teachers in China and the United States.”

“MMLU [19] (5-shot), BBH [23] (3-shot), ARC-Easy [24] (25-shot), ARC-Challenge (25-shot), HellaSwag [26] (10-shot), WinoGrande [25] (5-shot), NQ-Open [28, 29] (5-shot), AGIEval [30] (0-shot), TriviaQA [27] (5-shot).”

“Investigations into test-time compute [39], zero-RL in the wild [57], negative-preference optimisation [59], AGIEval benchmarking [60], minimalist RLHF pipelines [54], model-merging strategies [55], and foundational RL theory [44] likewise inform the context for our study.”

“There are two key challenges: AI can be misused in ways that undermine fair assessment, and its mistakes often appear convincing, potentially misleading students (Borges et al., 2024; Wang et al., 2023; Zhong et al., 2023; Arora et al., 2023).”

“Language comprehension: BBH [55], ARC-C [11], TruthfulQA [34], WinoGrande [51], HellaSwag [76], AGIEval-EN [87], CommonsenseQA [57].”

“Biases in LLM responses have been extensively studied (Dai et al., 2024), revealing issues such as social biases (Salewski et al., 2024; Liu et al., 2023a) and order sensitivity (Ak-ter et al., 2023).”

“Multi-task Chinese understanding benchmarks include C-Eval [23], AGIEval [24], GAOKAO 2 and CMMLU [25].”

“Jin et al., 2021; Pal et al., 2022; Alonso et al., 2024) and three general domain benchmarks (MMLU, MMLU-Pro, and AGIeval; Wang et al., 2023a; 2024b; Zhong et al., 2024).”

“We finetune the compressed LLMs on a new version of Alpaca, i.e. Alpaca-GPT4 (Peng et al., 2023), and evaluate them on both MMLU and AGIEval (Zhong et al., 2024).”

“Most conventional LLM benchmarks fall into this category, including ARC (Clark et al., 2018), GSM8K (Cobbe et al., 2021a), BIG-Bench (Zhong et al., 2024), AGIEval (Zhong et al., 2024) and MMLU (Hendrycks et al., 2020).”

“Our evaluations are performed using TruthfulQA [20]; AGIEval [21]; GPT4ALL [22], comprising piqa, openbookqa, boolq, arceasy, arcchallenge, winogrande, and hellaswag; MMLU 5-shot [23]; BBH-few shot [24]; the math domain, comprising MathQA [25] and GSM8K 5-shot [26]; and the Finance domain [27],…”

“To bridge this gap, researchers have introduced novel datasets such as the MMLU [41] and AGI-Eval [14].”

“…a structured taxonomy for thorough evaluations; 2) real-world application assessments, including standardized tests for human-like understanding (Zhong et al., 2023; Clark et al., 2018; Sakaguchi et al., 2021; Zellers et al., 2019), programming capabilities (Chen et al., 2021; Babe et al.,…”

“That is, K has a strong understanding of the basic concepts and executes the solution steps accurately, indicating proficiency in applying the elimination method (Al-Mutawah et al., Copyright © 2023; Author (s); 2339-224X (print) | 2579-3209 (online) 2019; Zhong et al., 2023).”

“Existing Chinese-inclusive LLM benchmarks (Huang et al., 2023; Li et al., 2024; Zhong et al., 2024; Chen et al., 2025) test clean-input knowledge; to our knowledge, FORGE is the first Chinese vulnerability benchmark under retrieval-time pollution.”

“Legal-domain General-purpose benchmarks, such as MMLU (Hendrycks et al., 2020) and AGIEval (Zhong et al., 2024), standardize the evaluation of LLMs through closed-form tasks and outcome-oriented metrics.”

“…school level, Math and GaokaoBench-Math (Zhang et al., 2023) at high school level , College Math (Tang et al., 2024), TheoremQA (Chen et al., 2023), ARB (Sawada et al., 2023) at university level, and Olympiad-Bench (He et al., 2024), AGIeval-Math (Zhong et al., 2023) at competition level.”

“Based on memory and manipulation for world knowledge, LLMs can provide more accurate and relevant responses to user questions (Hendrycks et al., 2020; Zhong et al., 2023).”

“Existing LLM testing predominantly relies on black-box testing by constructing benchmarks [24]–[27].”

“Miscellaneous : WinoGrande (Sakaguchi et al., 2020), AGIEval (Zhong et al., 2024a).”

“…LLM generations is by utilizing a strong LLM like GPT-4 (Bubeck et al., 2023; OpenAI, 2023a; Dubois et al., 2023; Zhou et al., 2023; Chiang and Lee, 2023; Wang et al., 2023c; Hada et al., 2023; Liu et al., 2023) on different benchmarks (Zhong et al., 2023; Köpf et al., 2023; Zheng et al., 2023).”

“• Domain-specific: LSAT-AR via AGIEval (Zhong et al., 2023), ChessInstruct 4 .”