AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”

“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”

“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”

“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”

“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”

“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”

“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”

“The results of the model on other benchmarks [13, 35, 48] can be found in Table 8 in the supplementary materials [39].”

“Phi-3 is a relatively small LLM but with favorable reported performance characteristics on common benchmark logic evaluations [28, 13, 4, 5].”

“AGIEval [35] is derived from 20 official, public, and high-standard admission and qualification exams intended for general human test-takers.”

“We evaluate the models across various dimensions, including natural language understanding, knowledge, mathematics, code generation, and logical reasoning, using eleven popular benchmarks including C-Eval (Huang et al., 2024) and AGIEval (Zhong et al., 2023), etc. Evaluation experiments are conducted in zero-shot or f…”

“, (Zhong et al., 2023; Hendrycks et al., 2021).”

“Benzer ¸sekilde çok fazla sayıda kabiliyeti test etmek için hazırlanmı¸s BigBench [3], Big Glue [4], AGI Eval [5] gibi de˘gerlendirme veri setleri de Türkçe odaklı de˘gildir.”

“Following (Guo et al., 2024), we ran the data decontamination check in order to remove data contamination in the pretraining corpus from the various benchmark evaluations that we performed to test the performance of our math SLM.”

“With the increasing emphasis on the trustworthiness of models, dedicated benchmarks are proposed to evaluate robustness (Yang et al., 2022; Wang et al., 2023b), hallucination (Li et al., 2023), bias (Zhong et al., 2023), and generalizability (Wang et al., 2023a).”

“Notable examples include MMLU [6], TruthfulQA [7], Big-Bench [8] in English, and AGI-Eval [9], C-Eval [10], CMMLU [11] for the Chinese domain, which are now widely adopted and prominently featured on leaderboards.”

“During the filtering stage, we removed all samples containing images or tables that could not fit into text-based queries for LLMs.”

“Generalist suites like MMLU (Hendrycks et al., 2021), BIG-Bench (Srivastava et al., 2023), and AGIEval (Zhong et al., 2023) include science questions but rely on short, single-turn QA and fail to assess the multi-step problem solving required in real laboratory settings.”

“…(Zhang et al., 2025b), PHYSICS (Pang et al., 2025), CMPhys-Bench (Wang et al., 2025a), PhysicsEval (Siddique et al., 2025), UGPhysics (Xu et al., 2025b), For-mulaReasoning (Li et al., 2024), agieval-gaokao-physics (Zhong et al., 2023), MVPBench (Dong et al., 2025), and Phyx (Shen et al., 2025).”

“Arena-style systems avoid reliance on fixed reference answers and better capture subjective conversational quality than static benchmarks such as MMLU [13], AGIEval [14], and HellaSwag [15].”

“These concerns are not unique to Evil-AI Benchmark and are shared by many modern API-mediated benchmark studies [67], [68].”

“We also evaluate our approach and the baseline models using established academic benchmarks, including ARC-Challenge (Clark et al., 2018), GSM8K (Cobbe et al., 2021), HellaSwag (Zellers et al., 2019), TruthfulQA (Lin et al., 2021), AGIEval (Zhong et al., 2023), and ANLI (Nie et al., 2019).”

“The leading index in each cell denotes the model’s rank for the corresponding metric, while the value in parentheses indicates its actual score. consistent relationship between benchmark scores and human-alignment: models scoring lower on MMLU, AGIEval, and BBH, like Vicuna-13B-v1.”

“Another example is the AR-LSAT corpus [35], which contains analytical reasoning questions from law school admission tests; these are high-level logical puzzles in natural language, and zero-shot GPT-4 still finds many of them challenging [34].”

“Additionally, we randomly selected 800 samples from AGIEVAL (General)(Zhong et al., 2023) as a dataset to evaluate the general capabilities.”

“5 (Liu et al., 2024)), Gemma9B (Gemma-2-9B-it (Zhong et al., 2023)), Gemma27B (Gemma-27B-it (Zhong et al., 2023)), and Ministral8B (Ministral-8B-Instruct-2410).”

“Now, LLMs may have the capability to automate or assist with every step of this process and seamlessly transition bteween covariance-and logic-based causality. soning (Bubeck et al., 2023; Zhong et al., 2023; Ghazal et al., 2017).”

“We evaluated on the following benchmarks: General: MMLU [Hendrycks et al., 2020], MMLU-Redux [Perez et al., 2024], ARC-Challenge [Clark et al., 2018], RACE High [Lai et al., 2017], TriviaQA [Joshi et al., 2017], NaturalQS [Kwiatkowski et al., 2019], and AGIEval [Zhong et al., 2023].”