AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“We fine-tune on the Alpaca GPT-4 dataset (Peng et al., 2023), which contains 52,000 instruction–response pairs generated by GPT-4 (OpenAI, 2023), and evaluate on three benchmarks: MMLU (Hendrycks et al., 2021), comprising 57 tasks and 14,079 questions covering a broad range of world knowledge; AGIEval (Zhong et al., 2…”

“…general intelligence (AGI) towards do-* However, existing multidisciplinary benchmarks are primarily focused on understanding tasks, including pure text benchmarks (Wang et al., 2024b; Zhong et al., 2023) and multimodal benchmarks (Yue et al., 2024a;b), but rarely focus on image generation tasks.”

“With a sufficient parameter size, pretrained language models have the ability to utilize their stored knowledge in various reasoning scenarios [2, 3, 35].”

“Another trend in MLLM assessment is the use of human exam questions [26, 25, 55, 53, 54].”

“One widely adopted approach for this is the multi-choice questions (MCQ) format, whose typical benchmark representatives are MMLU (Hendrycks et al., 2020), AGIEval(Zhong et al., 2023), and C-Eval(Huang et al., 2023), etc.”

“For example, AGIEval (Zhong et al., 2023a) provides a benchmark that covers standardized human exams, including college entrance exams, law school entrance exams, math competitions, and bar exams.”

“AGIEval (Zhong et al., 2023) focuses on standardized Chinese exams like the college entrance exam, while C-Eval (Huang et al., 2023) encompasses questions across four levels of difficulty from middle school to professional tests.”

“In line with the second design principle, we follow the recent mainstream approach of LLMs evaluating work, such as GSM8k (Cobbe et al., 2021), MMLU (Hendrycks et al., 2021), and AGIEval (Zhong et al., 2023), using a question-answering (Q&A) paradigm.”

“AGIEval [Zhong et al., 2023] is a benchmark tailored to assess models on human cognition and problem-solving, derived from 20 prominent admission and qualification exams including the Gaokao, SAT, law school tests, and civil service exams.”

“…and Llama (Touvron et al., 2023a)), have billions of parameters and show strong performance on various basic natu-ral language tasks (Qin et al., 2023), human examination (Hendrycks et al., 2020b; Zhong et al., 2023), and agent-related tasks (Guo et al., 2023; Liu et al., 2023; Zhou et al., 2023).”

“In language modeling, evaluation incurs relatively low overhead: newly saved model checkpoints can be rapidly assessed against standardized benchmarks [24, 28, 37, 40, 47, 61, 94, 121, 140, 141, 154], meaning evaluation rarely becomes a bottleneck.”

“Conventional evaluation paradigms rely heavily on human assessment (Zhong et al., 2024; Starace et al., 2025), which, while meticulous, presents a critical bottleneck due to its high cost and low scalability.”

“AGIEval (Zhong et al., 2023) is centered on standardized exams like the SAT, LSAT, and GRE, testing models’ reasoning, problem-solving, and language comprehension skills.”

“We evaluated the effectiveness of our data on five in-domain test sets: GSM8K, MATH, AQuA-RAT, NumGLUE, Mathematics, and four out-of-domain test sets: SVAMP [18], SimulEq [10], SAT-Math [27], MMLU-Math [7].”

“Among them, many benchmarks can perform relatively comprehensive evaluation on the models, such as HELM(Liang et al., 2022), MMLU (Hendrycks et al., 2020), C_EVAL(Huang et al., 2023), AGIEval(Zhong et al., 2023), BIG-Bench(Srivastava et al., 2022), etc.”

“Over time, numerous datasets of this kind have been created, and many have been adapted for LLM evaluation: GSM-8k (Cobbe et al., 2021), EXAMS (Hardalov et al., 2020), MMLU (Hendrycks et al., 2020), and AgiEval (Zhong et al., 2023), among many others.”

“It is worth noting that in some cases R E A LIGN will also provide a significant boost to knowledge, such as Open-Platypus on AGIEval.”

“Current benchmarks for LLMs mostly focus on the exam performance, such as MMLU (Hendrycks et al., 2021) and AGIEval (Zhong et al., 2023).”

“Multiple-Choice Question Answering (MCQA) is a fundamental and prevalent task for the evaluation of large language models (LLMs) (Gao et al., 2021; Zhong et al., 2024; OpenAI, 2023).”

“…has been a long-standing challenge in Natural Language Processing (NLP) and remains especially essential in the era of Large Language Models (LLMs) to understand their properties and behaviors (Liang et al., 2022; Chang et al., 2023; Zhong et al., 2023; Chia et al., 2023; Holtzman et al., 2023).”

“…(Zellers et al., 2019), MMLU (Hendrycks et al., 2020), Winogrande (Sakaguchi et al., 2019), DROP (Dua et al., 2019), Natural Questions (Kwiatkowski et al., 2019), AGIEval-English (Zhong et al., 2023), GSM8K (Cobbe et al., 2021), MMLU-Pro (Wang et al., 2024b), and TriviaQA (Joshi et al., 2017).”

“…al., 2022) for factual accuracy, GSM8K (Cobbe et al., 2021) for mathematical reasoning, DROP (Dua et al., 2019) for reading comprehension, AGI Eval (Zhong et al., 2024) for general intelligence capabilities, BigBench-Hard (Suzgun et al., 2023) for challenging reasoning tasks, and MMLU for…”

“Table 1 shows that the benefits of the ERC loss persist across various public benchmarks more challenging than those used for 3B models, including MMLU [13], C-Eval [15], MMLU-Pro [42], AGI-Eval [47], BBH [36], MATH [14], GSM8K [5], and TriviaQA [16].”

“Routers generalize out-of-domain (e.g., MMLU, GSM8k, AIME, TruthfulQA, SQuADv2, GPQA, AGIEval, PIQA) with only 0.85%p average accuracy drop while retaining efficiency, indicating that learned routing policies transfer beyond the supervised domains.”

“In this work , we utilize the dataset from Tang et al. (2024), which translated the math subset originally provided by Zhong et al. (2023) into English 2 .”

“Prior works (Zhang et al., 2023; Yue et al., 2024a; Das et al., 2024; Yue et al., 2024b; Zhong et al., 2023) have introduced multi-disciplinary benchmarks using questions from different competitive exams.”

“Existing LLM testing predominantly relies on black-box testing by constructing benchmarks [24]–[27].”

“More advanced tasks presented by benchmarks like AGIEval (Zhong et al., 2023) and BIG-Bench (Srivastava et al., 2022) test the limits of LLMs' problem-solving abilities, including their capacity for multi-step reasoning and understanding complex, real-world scenarios.”

“To ensure maximum device performance, we activated the Jetson clocks and monitored the power consumption of each We use the AGI Eval [14] provided by Meta to evaluate the model’s accuracy.”

“Some earlier benchmarks sourced from Wikipedia, the Children’s Book Test dataset (Hill et al., 2015), Quac (Choi et al., 2018) and SQuAD 2.0 (Rajpurkar et al., 2018), are nearly entirely contaminated.”

“…Free LILA (Mishra et al., 2022) High School Free MNLU (Hendrycks et al., 2020) High School & College MC CEval (Huang et al., 2023) High School & College MC AGIEval (Zhong et al., 2023) High School & College MC TheroemQA (Chen et al., 2023) College Free SciBench (Wang et al., 2023) College Free”

“AGIEval (Zhong et al., 2024), on the other hand, is designed to assess the capabilities of models in artificial general intelligence across a broad range of tasks and skills.”

“5B, outperforms all specific baselines on GSM8K, MATH, CMATH, GaoKao Math Cloze, and Gaokao Math QA.”

“…including close-book question-answering (QA) based knowledge testing(Hendrycks et al., 2020; Huang et al., 2023), human-centric standardized exams(Zhong et al., 2023), multi-turn dialogue(Lin and Chen, 2023), reasoning(Liu et al., 2023a; bench authors, 2023) and safety assessment(Sun et al.,…”

“AGI-Eval (Zhong et al., 2023) is a benchmark specifically designed to evaluate the general abilities in tasks pertinent to human cognition and problem-solving.”

“…[Cobbe et al., 2021] and SVAMP [Patel et al., 2021], which are math word problems; MATH [Hendrycks et al., 2021], a benchmark of high school competition level mathematics problems; and MMLU [Hendrycks et al., 2020] and SAT [Zhong et al., 2023], which are standard mathematical reasoning subsets.”

“NLP benchmarks include MMLU [128], CMMLU [129], AGIEval [130] and MATH [131].”

“The data sources are diverse, drawing from multiple academic datasets, including MMLU (Hendrycks et al., 2020a), Math500 (Hendrycks et al., 2020b), AGIEval (Zhong et al., 2023), AIME (Veeraboina, 2023), GPQA (Rein et al., 2024), SciKnowE-val (Feng et al., 2024), and UGPhysics (Xu et al., 2025b).”

“To evaluate the advantages and limitations of LLMs, some new benchmarks have been proposed to evaluate the broader knowledge and advanced abilities of LLM, mainly transitioning from traditional NLP tasks to simulate human exams, such as M3KE [30], AGIEval [17] and CEval [16].”

“This not only signifies maturity but also greatly advances the progress, as has been validated by ample work [142, 143, 144, 145, 146].”

“Core-knowledge benchmarks [16, 45, 10, 33, 9, 11, 48] assess LLMs through zero-shot and few-shot tasks, requiring concise, specific answers to questions that are easily validated.”

“…al., 2019; Baker et al., 2015; Pal et al., 2022; Shah et al., 2022; Chen et al., 2022; Amini et al., 2019; Hendrycks et al., 2021; Lin et al., 2022; Zhong et al., 2023b; bench authors, 2023; Suzgun et al., 2023; Liang et al., 2023) have been proposed for testing domain specific knowledge in LLMs.”

“…(OpenAI, 2023)) show strong strong performance on various basic natural language tasks and human examinations (Qin et al., 2023a; Jiao et al., 2023; Zhong et al., 2023; Wang et al., 2023d; Liang et al., 2023), * Equal contribution and arises the hope to help humans to complete tasks in complex…”

“…Chinese college entrance examination questions (GAOKAO) to assess the language comprehension and logical reasoning abilities of LLMs. • AGIEVAL (Zhong et al. (2023)): A bilingual evaluation dataset, encompassing standardized test questions such as the Chinese National College Entrance Exam…”

“7, 8, 9 and 10.”

“Logical Reasoning : We consider the logical reasoning part of the Law school admission test (LSAT) (Zhong et al., 2023).”

“AGIEval [11], [12] Knowledge & reasoning ARC-Challenge [13] Knowledge & reasoning BIG-Bench-Hard [14] Knowledge & reasoning GPQA [15], Diamond Knowledge & reasoning HellaSwag [16] Knowledge & reasoning MMLU [11], [17] Knowledge & reasoning GSM8K [11], [18] Mathematical & logical reasoning MATH…”

“These include benchmarks for code (Jimenez et al., 2023; Jain et al., 2024), mathematics (Zhong et al., 2023; Wang et al., 2023), city (Feng et al., 2025a), as well as agent-based tasks (Liu et al., 2023a; Qin et al., 2023).”

“As LLMs began to address a broader range of tasks across various domains (Wu et al., 2023; Wang et al., 2023), more comprehensive benchmark suites (Hendrycks et al., 2021a; Zhong et al., 2023) were developed to assess general capabilities rather than individual task performance.”

“AGIEval English [Zhong et al., 2024] includes the English language subset of the AGIEval benchmark, specifically these multiple-choice tasks: aqua-rat , logiqa-en , lsat-ar , lsat-lr , lsat-rc , sat-en , sat-math , and gaokao-english .”

“We evaluate these models across widely used benchmarks, including MMLU [Hendrycks et al., 2021], ARC-Challenge, AGIEval [Zhong et al., 2023], and SQuAD 2.0 [Rajpurkar et al., 2018].”

“Details of our downstream benchmarks are discussed below: Learning Rate 2 × 10 − 5 2 × 10 − 5 2 × 10 − 6 1 × 10 − 5 1 × 10 − 5 1 × 10 − 6 • AGIEval (Zhong et al., 2023): AGIEval is a general benchmark.”

“…2023) (zero-shot), MMLU (Hendrycks et al., 2021a) (5-shot), MMLU-pro (Wang et al., 2024a) (5-shot), CMMLU (Li et al., 2023b) (5-shot), GAOKAO (Zhang et al., 2024b) (zero-shot), AGIEval (Zhong et al., 2023) (zero-shot), GPQA (Rein et al., 2023) (5-shot) and TheoremQA (Chen et al., 2023) (5-shot).”

“Qualification examinations are rigorously verified by domain experts before public release, offering invaluable benefits to constructing realistic and reliable benchmark datasets (Yang et al., 2024b; Zhong et al., 2024).”

“We evaluated the performance of the fine-tuned models on the MATH Hard benchmark (Level 5 sub-set) [11], the GSM8K benchmark [6], and the SAT MATH dataset [50] to assess both in-domain accuracy and generalization capabilities.”

“…fine-tuning can also improve AbsAcc (Table 1), so we design another ablation study to investigate whether the improvement of MeanLearn only comes from additional data. precise, for each sample in AbsR, we use GPT-4 to generate a new explanation based on the question and label without the guidance…”

“we comprehensively evaluate LLM’s performance on datasets containing erroneous text by incorporating tasks such as AGIEval(Zhong et al., 2023), Gaokao Bench2, machine translation, and sentiment classification.”

“Particularly in terms of the results on the MMLU, AGIEVAL, CMMLU and CHID datasets, TeleChat’s performance surpasses that of other models of equivalent size.”

“The AGI Eval [38] benchmark assesses models on human-level cognition and problem-solving tasks, which tests the real-world applicability of model outputs.”

“…and 22 we report scores for our models on benchmarks that are generally considered more difficult than the standard suite used in the paper: MMLU-Pro (Wang et al., 2024), GPQA (Di-amond, Extended, and Main splits) (Rein et al., 2023), CulturalBench (Chiu et al., 2025 AGIEval (Zhong et al., 2024).”

“…(MMLU, MMLU-Pro, GPQA, AGIEval) and math benchmarks (GSM8K, MATH, OlympiadBench, Omni-MATH) emphasized harder, diagnostic items (Hendrycks et al., 2020; Wang et al., 2024b; Rein et al., 2024; Zhong et al., 2023; Cobbe et al., 2021; Hendrycks et al., 2021; He et al., 2024; Gao et al., 2024).”

“• Datasets where the effect of base models is unclear: general QA (MMLU-pro (Wang et al., 2024), AGIEval (Zhong et al., 2024), and PopQA (Mallen et al., 2023)) and scientific literature (Sci-enceMeter (Wang et al., 2025a)).”

“Many of these systems have demonstrated NLP capabilities at a level very close-to-human capabilities (Zhong et al., 2023).”

“At the same time, with new models arriving in quick succession, and ever-present scope for data to leak from evaluations to training (Ganguli et al, 2023), the evaluation problem is a dynamic one: it requires ongoing, adaptive gathering of new evaluation data.”

“However, as user demands grow more sophisticated, traditional benchmarks (Zhong et al., 2024; Chia et al., 2023), which focus on specific tasks, are insufficient to evaluate models’ comprehensive ability to handle multifaceted instructions.”

“…benchmarks spans elementary word problems (Cobbe et al., 2021; Patel et al., 2021; Amini et al., 2019; Miao et al., 2020; Ling et al., 2017; Koncel-Kedziorski et al., 2016; Shi et al., 2015) to higher levels (Hendrycks et al., 2021b,a; Zhong et al., 2024; Zhang et al., 2023; He et al., 2024).”

“AGIEval The AGIEval benchmark [65] is designed to evaluate models on human standardized tests including college entrance exams, law school admission tests, and graduate-level professional certification exams.”

“Their capabilities have been evaluated across a wide range of benchmarks, including general benchmarks like AGIEval [64], MMLU [26], and BIG-Bench Hard [48]; knowledge reasoning benchmarks such as TriviaQA-Wiki [30]; and reading comprehension benchmarks like SQuAD [42] and BoolQ [19].”

“For evaluation, we adopt the following widely used benchmarks, including GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), BBH (Suzgun et al., 2023), TriviaQA (Joshi et al., 2017), MBPP (Austin et al., 2021), AGIEval (Zhong et al., 2024), DROP (Dua et al., 2019), MMLU-pro (Wang et al., 2024).”

“To demonstrate the superiority of Adapt-Accel, three benchmarks are employed for evaluation, including BBH [Srivas-tava et al., 2022], TruthfulQA [Lin et al., 2021], AGIEval [Zhong et al., 2023], which assess all methods performance in different aspects beyond commonsense reasoning.”

“…is still limited beyond the standardized bar exam questions (Guha et al., 2023; Dahl et al., 2024; Magesh et al., 2024; Kang et al., 2023; Bern-sohn et al., 2024; Blair-Stanek et al., 2023), particularly in handling legal tasks in languages other than English (Fei et al., 2023; Zhong et al., 2023).”

“One group of research evaluates the general abilities of LLMs (Hendrycks et al., 2020; Zheng et al., 2023; Zhong et al., 2023; Bang et al., 2023) Besides, existing works evaluate LLMs in specific tasks (Bang et al., 2023; Bian et al., 2023; Gao et al., 2023; Wei et al., 2023; Li et al., 2024).”

“Benchmarks like AGIEval assess models via standardized exams [43], and MT-Bench focuses on conversational and instructional capabilities [42].”

“Therefore, a diverse array of benchmarks has been developed to examine the varied real-world aspects of MLLMs. Benchmarks like LLaVA-Bench (Liu et al., 2023c), MMBench (Liu et al., 2023d), MM-VET (Yu et al., 2023), TouchStone (Bai et al., 2023b), MLLM-bench (Ge et al., 2023), and SEED-Bench(Li et…”

“However, another recent bilingual benchmark (Zhong et al., 2023) found that GPT-4 could be largely biased to subjects with sufficient data (reproduced in Fig.”

“Some find that recent ChatGPT and GPT4 perform better than previous models (Zhong et al., 2023; Chia et al., 2023), and there may exist an evolutionary tree of language models (Yang et al., 2023).”

“Tasks and Datasets We selected a total of 17 datasets from both general and domain-specific tasks for extensive experimentation, which include: 6 BBH (Suzgun et al., 2022) general tasks, 6 MMLU (Wang et al., 2024b) general tasks, 3 C-Eval (Huang et al., 2024) Chinese domain tasks, 1 LSAT-AR (Zhong et al., 2023) law do…”

“Rentatrucknearzip08817onDecember10at11:30AMreturnedtotheexactlocationanddate.”