AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Open accessAbstract
Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have been shown to excel at following instructions and comprehending retrieved passages.
“While LLMs have been shown to excel at these tasks (Wang et al., 2022;Zhong et al., 2023)”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Zhong et al., 2023
Section 4.5.2 (Strengths; page unavailable) says the models excel at comprehending context and questions, and Section 4.3 (Main Results; page unavailable) links strong zero-shot performance to instruction tuning and understanding tasks. The paper does not establish that the models excel specifically at following instructions or comprehending retrieved passages; its comprehension results concern exam questions and their context. (LLM, conf=0.84)
Experimental — accuracy not yet measuredAGIEval serves as a dedicated evaluation framework for assessing performance of foundation models in human-centric standardized exams.
“AGIEval [262], serves as a dedicated evaluation framework for assessing the performance of foundation models in the domain of human-centric standardized exams.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [262]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 3× · see citing works
“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3× · see citing works
“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1× · see citing works
“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 2× · see citing works
“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 3× · see citing works
“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 2× · see citing works
“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 2,991 · cited 1× · see citing works
“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1× · see citing works
“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1× · see citing works
“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 3× · see citing works
“, 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate language model performance across diverse NLP tasks, including paraphrase identification, sentiment analysis, and more.”
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · cited 1× · see citing works
“A recent technical report (SenseTime, 2023) further evaluates foundation models (GLM130B (Du et al., 2022), LLaMa-65B (Touvron et al.”
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 522 · cited 1× · see citing works
“Lawyer Qualification Test: Including the bar exam, these tests assess legal knowledge, analytical skills, and ethical understanding.”
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 1× · see citing works
“The following categories of human-centric exams are included in our benchmark: General College Entrance Exams: Including the GRE, SAT, and Gaokao, these exams assess critical thinking, problem-solving, and analytical skills for entry into higher education.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · arXiv · doi:10.48550/arxiv.1803.05449 · Cited by 477 · cited 1× · see citing works
“SentEval (Conneau and Kiela, 2018) and De-caNLP (McCann et al., 2018) also set benchmarks for evaluating models’ general capabilities.”
The Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448 · cited 1× · see citing works
“SentEval (Conneau and Kiela, 2018) and De-caNLP (McCann et al., 2018) also set benchmarks for evaluating models’ general capabilities.”
Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning
Hua, Y.-C., Li, Y.-F., Haffari, G., Qi, G., & Wu, W. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/505 · Cited by 398 · see citing works
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · see citing works
Solving Quantitative Reasoning Problems with Language Models
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.14858 · Cited by 358 · cited 1× · see citing works
“For MATH, we use the same instances as in the appendices of Lewkowycz et al. (2022).”
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · cited 1× · see citing works
“Furthermore, we incorporate GRE and GMAT questions from AQaA-RAT (Ling et al., 2017), which emphasizes algebraic word problems.”
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 3× · see citing works
“(2) AGIEval is bilingual (English and Chinese), broadening the assessment scope across languages and cultures, whereas MMLU is solely English-based.”
InfluentialToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · arXiv · doi:10.48550/arxiv.2203.09509 · Cited by 276 · cited 1×
“The following categories of human-centric exams are included in our benchmark: General College Entrance Exams: Including the GRE, SAT, and Gaokao, these exams assess critical thinking, problem-solving, and analytical skills for entry into higher education.”
Automatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274 · cited 2× · see citing works
“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
JEC-QA: A Legal-Domain Question Answering DatasetPreprint
Zhong, H., Xiao, C.-J., Tu, C.-C., Zhang, T., Liu, Z., & Sun, M. (2019) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1911.12011 · Cited by 250 · cited 4× · see citing works
“For LogiQA, JEC-QA and LSAT, we use ChatGPT to generate explanations given the questions and the answers.”
InfluentialBOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244 · cited 1× · see citing works
“Toxi-Gen (Hartvigsen et al., 2022) and BOLD (Dhamala et al., 2021) evaluate the bias in language models.”
The LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87 · cited 1× · see citing works
“The LAMBADA language modeling task (Paperno et al., 2016) assesses language models’ ability to capture long-range dependencies in text.”
From LSAT: The Progress and Challenges of Complex ReasoningPreprint
Wang, S., Liu, Z.-K., Zhong, W.-J., Zhou, M., Wei, Z.-Y., Chen, Z.-M., & Duan, N. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.48550/arxiv.2108.00648 · Cited by 71 · see citing works
Self-citationEhrmann, M., Romanello, M., Doucet, A., & Clematide, S. (2022) · European Conference on Information Retrieval · doi:10.1007/978-3-030-99739-7_44 · Cited by 22 · cited 1×
“Numerous benchmarks (Thorne et al., 2018; Rajpurkar et al., 2016; Bowman et al., 2015; Ehrmann et al., 2022) have been proposed and widely adopted for evaluating single-task performance, such as SQuAD (Rajpurkar et al.”
Zhong, W., Wang, S., Tang, D., Xu, Z., Guo, D., Chen, Y., Wang, J., Yin, J., Zhou, M., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.177 · Cited by 14 · cited 1× · see citing works
“For the LSAT, we utilize data from Wang et al. (2022) and Zhong et al. (2022), which encompasses three tasks (logical reasoning, reading comprehension, and analytical reasoning) from the LSAT administered between 1991 and 2016.”
Self-citationOPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068 · cited 1×
“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
Flagged first, then by citations
8 of the 41 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty- Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3622-3628, 2021.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“Furthermore, AGIEval [247], serves as a dedicated evaluation framework for assessing the performance of foundation models in the domain of human-centric standardized exams.”
BackgroundInfluentialSelf-citationLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
“In Table 24, we present fine-grained results from the English part of the AGI Eval (Zhong et al., 2023) benchmark.”
BackgroundInfluentialGemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.19786 · Cited by 1,156
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
MethodC-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.08322
“Recently, the AGIEval benchmark (Zhong et al., 2023) contains data from the Chinese College Entrance Exam, Chinese lawyer qualification test and Chinese civil service examination.”
MethodInstruction Tuning for Large Language Models: A Survey
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wang, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3777411 · Cited by 640 · see citing works
“A key distinction lies between LLM-centric designs (e.g., LLaVA), which excel at complex reasoning and detailed textual explanations, and vision-centric models (e.g., InstructPix2Pix), which offer superior performance on fine-grained visual manipulation tasks.”
DeepSeek-VL: Towards Real-World Vision-Language Understanding
Lu, H., Liu, W., Zhang, B., Wang, B.-L., Dong, K., Liu, B., Sun, J., Ren, T.-Z., Li, Z.-S., Yang, H., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2403.05525 · Cited by 941
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q.-H., Xu, H., Ye, J., Yan, M.-S., Hu, A., Liu, H.-W., Qian, Q., Zhang, J., Huang, F., & Zhou, J.-R. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01239 · Cited by 763 · see citing works
“Data sets mPLUG-Owl2 is first pre-trained on image-text”
BackgroundFrom Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder PipelinePreprint
Li, T.-L., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T.-H., Zhu, B.-H., Gonzalez, J. E., & Stoica, I. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2406.11939 · Cited by 595 · see citing works
“Common ones include MMLU [16], MATH [17], GSM-8K [10], HumanEval [7], DROP [12], BigBench [37], Hel-laSwag [49], AGIEval [52], GPQA [35], as well as comprehensive collection like HELM [26].”
BackgroundCMMLU: Measuring massive multitask language understanding in Chinese
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., & Baldwin, T. (2023) · arXiv · doi:10.48550/arxiv.2306.09212
“AGIEval (Zhong et al., 2023) focuses on Chinese standardized exams, such as college entrance exams, math competitions, and lawyer qualification tests.”
BackgroundSelf-citationThe Lessons of Developing Process Reward Models in Mathematical ReasoningPreprint
Zhang, Z.-R., Zheng, C., Wu, Y., Zhang, B.-C., Lin, R.-J., Yu, B.-W., Liu, D., Zhou, J.-R., & Lin, J.-Y. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2501.07301 · Cited by 528 · see citing works
“We evaluate across three Chinese benchmarks including Chinese math benchmarks CMATH (Wei et al., 2023), GaoKao Math Cloze (Zhong et al., 2024), and GaoKao Math QA (Zhong et al., 2024) following Yang et al. (2024c), as shown in (Liao et al., 2024) and OlympiadBench (He et al., 2024).”
Orca: Progressive Learning from Complex Explanation Traces of GPT-4Preprint
Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., & Awadallah, A. (2023) · arXiv.org · doi:10.48550/arxiv.2306.02707 · Cited by 466 · see citing works
“7M SAT-English English 206 656 SAT-Math Math 220 54 Table 6: Breakdown of tasks in AGIEval benchmark as reported in [1].”
BackgroundMethodInfluentialBenchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2309.01431 · Cited by 455 · see citing works
“Recently, with the development of LLMs, a series of general evaluation benchmarks have emerged, such as AGIEval (Zhong et al. 2023), C-Eval (Huang et al. 2023), AlpacaEval (Li et al. 2023b), OpenLLM Leaderboard (Edward Beeching 2023), etc.”
BackgroundLarge Language Models for Education: A survey and outlook
Wang, S., Xu, T., Li, H., Zhang, C.-L., Liang, J., Tang, J.-L., Yu, P. S., & Wen, Q.-S. (2024) · IEEE Signal Processing Magazine · doi:10.1109/msp.2025.3594309 · Cited by 438 · see citing works
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · see citing works
“This strategy has been successfully adopted in benchmarks such as MMLU [24] and AGIEval [85], but only text-based questions are considered, while human experts are capable of solving multimodal problems.”
MethodSelf-citationTo CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningPreprint
Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D.-W., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., & Durrett, G. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2409.12183 · Cited by 405 · see citing works
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454 · see citing works
Large Language Models for Mathematical Reasoning: Progresses and ChallengesPreprint
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., & Yin, W. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.00157 · Cited by 382 · see citing works
OpenChat: Advancing Open-source Language Models with Mixed-Quality DataPreprint
Wang, G., Cheng, S., Zhan, X., Li, X.-G., Song, S., & Liu, Y. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.11235 · Cited by 368 · see citing works
“Furthermore, we also use AGIEval (Zhong et al., 2023) to prove the gen-1 eralization, where openchat-13b also achieves the top-1 average accuracy among all 13b open-source language models.”
BackgroundMethodInfluentialJiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · see citing works
“…with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and AGI Eval [29] (3-5-shot, English multiple-choice questions only) Detailed results for Mistral 7B, Llama 2 7B/13B, and Code-Llama 7B are reported in Table 2.”
BackgroundMT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues
Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.401 · Cited by 79 · see citing works
“Subsequent benchmarks (Huang et al., 2023; Li et al., 2023a; Zhong et al., 2023; Srivastava et al., 2022; Yu et al., 2023) aim to rigorously assess LLMs’ knowledge and logic.”
BackgroundBai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
Guo, T., Guo, K., Nan, B., Liang, Z., Guo, Z.-C., Chawla, N., Wiest, O., & Zhang, X. (2023) · Neural Information Processing Systems · doi:10.52202/075280-2607 · Cited by 323 · see citing works
“These studies underscored the complexity of achieving task-specific accuracy and functionality with LLMs. Lastly, AGIEval [62] assessed LLMs’ general abilities but noted struggles in complex reasoning tasks.”
BackgroundScaling Diffusion Language Models via Adaptation from Autoregressive Models
Gong, S., Agarwal, S., Zhang, Y.-Z., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J.-W., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.17891 · Cited by 315
“We evaluate Dif-fuLLaMA’s math reasoning and in-context learning ability by evaluating on MAWPS (Koncel-Kedziorski et al., 2016) consisting of math word problems and SATMATH from AGI-eval consisting of math problems from SAT exam (Zhong et al., 2024).”
He, Z., Liang, T., Xu, J.-H., Liu, Q.-Z., Chen, X., Wang, Y., Song, L.-F., Yu, D., Liang, Z.-W., Wang, W., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.11456 · Cited by 307
“We performed decontamination against a comprehensive suite of mathematics and STEM benchmarks, including MATH (Hendrycks et al., 2021b), AIME (MAA, a), AMC (MAA, b), Minerva Math (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), Omni-MATH (Gao et al., 2024), MathOdyssey (Fang et al., 2024), GAOKAO (Zhong et a…”
Self-citationSecAlign: Defending Against Prompt Injection with Preference Optimization
Chen, S., Zharmagambetov, A. S., Mahloujifar, S., Chaudhuri, K., Wagner, D., & Guo, C. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744836 · Cited by 54 · see citing works
“We run more utility benchmarks (MMLU [56], Winogrande [57], AGIEval[58],andCommonSenseQA[59])onMistral-7BandLlama3- 8Btocheckthemodel’sfunctionoutsidetheAlpacaEval2bench-markpresentedinthemainexperiments.”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language ModelsPreprint
Wang, X.-X., Hu, Z.-N., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., & Wang, W. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2307.10635 · Cited by 300 · see citing works
“Previous works have relied on human labor to annotate error reasons into different categories, which is both expensive and time-consuming [49].”
BackgroundMethodInfluentialGemma: Open Models Based on Gemini Research and Technology
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.08295 · Cited by 284 · see citing works
“These specific benchmarks are: ARC (Clark et al., 2018), CommonsenseQA (Talmor et al., 2019), Big Bench Hard (Suzgun et al., 2022), and AGI Eval (English-only) (Zhong et al., 2023).”
BackgroundFLASK: Fine-grained Language Model Evaluation based on Alignment Skill SetsPreprint
Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J., & Seo, M. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2307.10928 · Cited by 253 · see citing works
“Conventionally, the performance of LLMs is measured by assessing the model on separate benchmarks using automatic metrics such as accuracy for knowledge/reasoning tasks or ROUGE for longform text generation (Chung et al., 2022; Hendrycks et al., 2020; Suzgun et al., 2022; Wang et al., 2022c; Gao et al., 2021; Zhong et…”
MethodDefending Large Language Models Against Jailbreaking Attacks Through Goal PrioritizationPreprint
Zhang, Z., Yang, J., Ke, P., Mi, F., Wang, H.-N., & Huang, M.-L. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.09096 · Cited by 238 · see citing works
“Consequently, models are capable of learning to prioritize safety over helpfulness in straightforward inquiries but struggle to discern the goal priority in out-of-distribution (OOD) jailbreaking cases.”
BackgroundEvaluating the Performance of Large Language Models on GAOKAO BenchmarkPreprint
Zhang, X., Li, C.-Y., Zong, Y., Ying, Z., He, L., Qiu, X.-P., Sun, T., Li, P., Meng, S.-Q., Zheng, Y.-J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.12474 · Cited by 236 · see citing works
The effect of sampling temperature on problem solving in large language models
Renze, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.432 · Cited by 212 · see citing works
VITA: Towards Open-Source Interactive Omni Multimodal LLM
Fu, C., Lin, H., Long, Z., Shen, Y.-H., Dai, Y.-H., Zhao, M., Zhang, Y.-F., Dong, S.-Q., Li, Y.-Z., Wang, X., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2408.05211 · Cited by 208
“To validate the efficacy of our training process for language model, we evaluate our trained model “Mixtral 8x7B Ours” against the official version “Mixtral 8x7B Instruct”, on four datasets: C-EVAL [20], AGIEVAL [58], MMLU [18], and GSM8K [10].”
MethodLarge Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning
Yue, M., Zhao, J., Zhang, M., Du, L., & Yao, Z. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.03094 · Cited by 207
“Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in reasoning tasks (Rae et al., 2021; Lewkowycz et al., 2022; Zhong et al., 2023).”
BackgroundScientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 155 · see citing works
Self-citationRemoving RLHF Protections in GPT-4 via Fine-Tuning
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T., & Kang, D. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.05553 · Cited by 202
“We chose the informativeness metric on Truth-fulQA (Lin et al., 2021), AGIEval (Zhong et al., 2023), MMLU (Hendrycks et al., 2020), gsm-8k (Cobbe et al., 2021), and WinoGrande (Sakaguchi et al., 2021) as orthogonal evaluations.”
MethodMathScale: Scaling Instruction Tuning for Mathematical ReasoningPreprint
Tang, Z., Zhang, X., Wang, B., & Wei, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.02884 · Cited by 31 · see citing works
A Comprehensive Study of Knowledge Editing for Large Language ModelsPreprint
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S.-M., Wang, M.-R., Xi, Z.-K., Mao, S., Zhang, J., Ni, Y., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2401.01286 · Cited by 194 · see citing works
“These benchmarks include Com-monsenseQA [187], PIQA [188], Xsum [189], and TriviaQA [190], as well as specific tasks from the MMLU [191] and AGIEval [192] suites, which are known for their distinguished evaluation criteria suites.”
MethodBeyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling LawsPreprint
Sardana, N., Doubov, S., & Frankle, J. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2401.00448 · Cited by 192 · see citing works
“• Reading Comprehension : SQuAD (Rajpurkar et al., 2016), BoolQ (Clark et al., 2019), CoQA (Reddy et al., 2019), and AGI Eval (Zhong et al., 2023).”
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific ResearchPreprint
Sun, L., Han, Y., Zhao, Z., Ma, D., Shen, Z.-W., Chen, B.-C., Chen, L., & Yu, K. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.13149 · Cited by 187 · see citing works
“AGIEval (Zhong et al. 2023) serves as an evaluation framework for assessing the performance of foundation models in human-centric standardized exams.”
BackgroundParameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186 · see citing works
CMB: A Comprehensive Medical Benchmark in Chinese
Wang, X., Chen, G. H., Song, D., Zhang, Z., Chen, Z., Xiao, Q., Jiang, F., Li, J., Wan, X., Wang, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08833
“…ability benchmarks include comprehensive test suites, each targeting different aspects of LLM’s proficiency, ranging from handling multi-turn dialogues (Zheng et al., 2023) to gauging language comprehension and reasoning abilities (Srivastava et al., 2022; Zhang et al., 2023a; Zhong et al., 2023).”
BackgroundMMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Zhao, Y., Zhang, H., Xie, L., Hu, T., Gan, G., Long, Y., Hu, Z., Chen, W., Li, C., Xu, Z., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00793 · Cited by 14 · see citing works
“Early benchmarks focused on domain-specific tasks for textual domains, establishing a foundation for assessing the models’ strengths and limitations in expert reasoning [21, 27, 60, 133, 147, 150, 168, 169].”
Self-citationLawBench: Benchmarking Legal Knowledge of Large Language ModelsPreprint
Fei, Z.-W., Shen, X., Zhu, D., Zhou, F., Han, Z., Zhang, S., Chen, K., Shen, Z.-W., & Ge, J.-D. (2023) · arXiv.org · doi:10.48550/arxiv.2309.16289 · Cited by 176 · see citing works
“In contrast to assessments that primarily focus on testing model’s ability to pass legal bar exams [5; 86], which are not always representative of the actual use-cases for LLMs 2 , our emphasis is on examining a structured set of legal skills required in real-world scenarios.”
BackgroundResultPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.14219 · Cited by 166 · see citing works
Self-citationA survey of multilingual large language models
Qin, L., Chen, Q., Zhou, Y., Zhi, C., Li, Y., Liao, L., Li, M., Che, W., & Yu, P. S. (2025) · Patterns · doi:10.1016/j.patter.2024.101118 · Cited by 197 · see citing works
M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language ModelsPreprint
Zhang, W., Aljunied, S. M., Gao, C., Chia, Y. K., & Bing, L. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.05179 · Cited by 153 · see citing works
“Firstly, the majority of these benchmarks focus on questions in English [17, 43], neglecting the evaluation of a model’s performance in a multilingual context.”
BackgroundMethodInfluentialRetrieval-Augmented Generation (RAG) Chatbots for Education: A Survey of Applications
Swacha, J., & Gracel, M. (2025) · Applied Sciences · doi:10.3390/app15084234 · Cited by 148 · see citing works
Causal Inference Meets Deep Learning: A Comprehensive Survey
Jiao, L., Wang, Y., Liu, X., Li, L., Liu, F., Ma, W., Guo, Y., Chen, P., Yang, S., & Hou, B. (2024) · Research · doi:10.34133/research.0467 · Cited by 122 · see citing works
“Zhong et al. [139] and Nori et al. [140] used benchmark tests and question-and-answer evaluations to score large models by evaluating the accuracy of their answers to a set of questions.”
BackgroundSelf-citationHuang, H., Bu, X.-Y., Zhou, H., Qu, Y., Liu, J., Yang, M.-Y., Xu, B., & Zhao, T.-J. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2025.findings-acl.306 · Cited by 146 · see citing works
“Therefore, after the emergence of LLMs, numerous efforts have been made to construct benchmark testsets that cover a variety of tasks for evaluation (Srivastava et al., 2023; Zhong et al., 2024).”
BackgroundJiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143 · see citing works
“…with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16] (5-shot), BBH [29] (3-shot), and AGI Eval [34] (3-5-shot, English multiple-choice questions only) Figure 2: Performance of Mixtral and different Llama models on a wide range of benchmarks .”
BackgroundBaichuan-Omni-1.5 Technical ReportPreprint
Li, Y.-D., Liu, J., Zhang, T., Chen, S., Li, T.-P., Li, Z., Liu, L., Ming, L., Dong, G., Pan, D., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.15368 · Cited by 142 · see citing works
“5, we utilize 4 comprehensive benchmarks, incuding MMLU [55], CMMLU [79], AGIEval [192], C-Eval [59] and GAOKAO-Bench [185].”
Self-citationBuehler, M. J. (2023) · Applied Mechanics Reviews · doi:10.1115/1.4063843 · Cited by 132 · see citing works
“For instance, in the AGI Eval test [31] it averages 0.”
BackgroundMMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.01574 · Cited by 131 · see citing works
“For example, AGIEval [48] focuses on general-exam questions from SAT, Gaokao, GRE, etc. ARC [12] focuses on science-based questions.”
BackgroundSelf-citationLISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Pan, R., Liu, X., Diao, S., Pi, R.-J., Zhang, J., Han, C., & Zhang, T. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2403.17919 · Cited by 127
“…assess LLMs on multiple aspects; MMLU [68] includes a total of 57 tasks with 14,079 questions covering a broad spectrum of world knowledge; AGIEval [69] serves as a human-centric benchmark for general abilities, comprising 9,316 instances; WinoGrande [70] is a large-scale dataset for commonsense…”
DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
Zhu, K., Chen, J., Wang, J.-D., Gong, N., Yang, D., & Xie, X. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.17167 · Cited by 127
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 72 · see citing works
“Evaluates models’ ability on diverse domains of tasks MMLU [73], MMMU [236], MMStar [27], M3GIA [191], AGIEval [253]”
Baichuan 2: Open Large-scale Language Models
Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.10305 · Cited by 125 · see citing works
“AGIEval (Zhong et al., 2023) is a human-centric benchmark specifically designed to evaluate general abilities like human cognition and problem-solving.”
BackgroundSelf-citationCValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility
Xu, G., Liu, J., Yan, M., Xu, H., Si, J., Zhou, Z., Yi, P., Gao, X., Sang, J., Zhang, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09705
“Most of these focus on assessing the helpfulness of the LLMs, such as AGIEval (Zhong et al., 2023b) and MMCU (Zeng, 2023) for Chinese and English College Entrance Exams, M3KE (Liu et al.”
BackgroundInfluentialChatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118
“Prominent examples in this category are MMLU (Hendrycks et al., 2020), Hel-laSwag (Zellers et al., 2019), GSM-8K (Cobbe et al., 2021), BigBench (Srivastava et al., 2023), AGIEval (Zhong et al., 2023), and HumanEval (Chen et al., 2021).”
BackgroundSeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning
Wang, B., Liu, Z., Huang, X., Jiao, F., Ding, Y., Aw, A., & Chen, N. F. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2309.04766 · Cited by 118
“Furthermore, we incorporate DREAM for English dialogue comprehension, OCNLI and C3 for Chinese comprehension, and Indo-Emotion dataset (Saputri et al., 2018; Wilie et al., 2020) to gauge emotion comprehension in Indonesian.”
BackgroundMethodInfluentialExploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
Xie, T., Foster, D. J., Krishnamurthy, A., Rosset, C., Awadallah, A., & Rakhlin, A. (2024) · arXiv.org · doi:10.48550/arxiv.2405.21046 · Cited by 117
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115 · see citing works
Can Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Chen, L., Wang, X., Huang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.09150
“Evaluated Models We evaluate a total of 34 models that demonstrated exceptional performance on other benchmarks (Huang et al. 2023; Dubois et al. 2023; Zhong et al. 2023), ranging from their model size, supported context length, and instruction tuning data size, as illustrated in Appendix.”
BackgroundMethodInfluentialThe Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models
Renze, M., & Guven, E. (2024) · 2024 2nd International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852493 · Cited by 110 · see citing works
“We preprocessed the data into a standard data format and randomly selected 100 questions from each of the ten benchmarks to create an exam with 1,000 MCQA problems.”
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110 · see citing works
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.04434 · Cited by 109 · see citing works
“Note that AGIEval includes both English and Chinese subsets.”
Large Language Models for Education: A SurveyPreprint
Xu, H., Gan, W.-S., Qi, Z.-L., Wu, J.-Y., & Yu, P. S. (2024) · arXiv.org · doi:10.48550/arxiv.2405.13001 · Cited by 105 · see citing works
“MathGPT has an accuracy rate of 60.34% in the benchmark test AGIEval, which can help students solve mathematical problems efficiently [142].”
BackgroundInfluentialInadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
“Prevalence : 18/23 A critical issue identified in LLM benchmarks is the inconsistency in implementation across different research teams, as observed in [7]–[12], [14], [31], [39]–[42], [44], [46]– [49], [52] (Appendix B-A).”
BackgroundMethodInfluentialLuo, G., Yang, X., Dou, W., Wang, Z., Liu, J., Dai, J., Qiao, Y., & Zhu, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02324 · Cited by 23 · see citing works
“NLP benchmarks include MMLU [29], CMMLU [36], AGIEval [89] and MATH [30].”
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
Huang, Z., Wang, Z., Xia, S., Li, X.-F., Zou, H.-Y., Xu, R.-J., Fan, R.-Z., Ye, L., Chern, E., Ye, Y.-X., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.12753 · Cited by 104
“This leads to the creation of benchmarks such as MMLU [18], AGIEval[63], C-Eval [22], and CMMLU [26], which pushed the limits of language models with multidisciplinary, multilingual, and knowledge-intensive tasks.”
BackgroundMethodALLaM: Large Language Models for Arabic and EnglishPreprint
Bari, M. S., Alnumay, Y., Alzahrani, N. A., Alotaibi, N. M., Alyahya, H., Alrashed, S., Mirza, F., Alsubaie, S. Z., Alahmed, H. A., Alabduljabbar, G., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2407.15390 · Cited by 103 · see citing works
“…Easy and Challenge (Clark et al., 2018), TriviaQA (Joshi et al., 2017), BoolQ (Clark et al., 2019), NQ Open (Kwiatkowski et al., 2019), AGIEval (Zhong et al., 2023), Exams-Ar (Hardalov et al., 2020), MMLU Arabic (tr) (Huang et al., 2023), MMLU Arabic (MBZU) (Koto et al., 2024) , and ETEC…”
BackgroundDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.03300 · Cited by 101 · see citing works
“Chinese benchmarks include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong et al., 2023), and Gaokao-MathQA (Zhong et al., 2023).”
BackgroundMethodInfluentialLLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement
Lee, N., Wattanawong, T., Kim, S., Mangalam, K., Shen, S., Anumanchipalli, G., Mahoney, M. W., Keutzer, K., & Gholami, A. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2403.15042 · Cited by 101
“For many of these general benchmarks (Hendrycks et al., 2020; Zhong et al., 2023), LLMs are prompted with custom instructions or in-context examples.”
BackgroundDeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99 · see citing works
“We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
Self-citationKe, P., Wen, B., Feng, A., Liu, X., Lei, X., Cheng, J., Wang, S., Zeng, A., Dong, Y., Wang, H., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.704 · Cited by 40 · see citing works
“…such as multi-choice QA to measure the performance of LLMs via simple objective metrics (such as accuracy and F1 score) (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023b), which may deviate from the common usage of LLMs and may not exactly reflect the ability of LLMs in…”
BackgroundCMATH: Can Your Language Model Pass Chinese Elementary School Math Test?Preprint
Wei, T., Luan, J., Liu, W., Dong, S., & Wang, B. (2023) · arXiv.org · doi:10.48550/arxiv.2306.16636 · Cited by 97 · see citing works
Data-Juicer: A One-Stop Data Processing System for Large Language Models
Chen, D.-Y., Huang, Y., Ma, Z.-J., Chen, H., Pan, X.-C., Ge, C., Gao, D.-W., Xie, Y.-X., Liu, Z., Gao, J.-Y., et al. (2023) · SIGMOD Conference Companion · doi:10.1145/3626246.3653385 · Cited by 94 · see citing works
Large language model assisted fine-grained knowledge graph construction for robotic fault diagnosis
Liao, X., Chen, C., Wang, Z., Liu, Y., Wang, T., & Cheng, L. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103134 · Cited by 88 · see citing works
EmoBench: Evaluating the Emotional Intelligence of Large Language ModelsPreprint
Sabour, S., Liu, S., Zhang, Z., Liu, J. M., Zhou, J., Sunaryo, A., Li, J.-Z., Lee, T. M. C., Mihalcea, R., & Huang, M.-L. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.12071 · Cited by 89 · see citing works
“However, despite their apparent proficiency in a variety of downstream tasks, such as question answering, and summarization (Zhou et al., 2023a; Zhong et al., 2023), research on evaluating EI capabilities for LLMs has been limited.”
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge EvaluationPreprint
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y.-X., Jiang, S.-H., Xiong, Z., Li, Z.-H., Wu, W.-J., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2306.05783 · Cited by 89 · see citing works
“Meanwhile, other research, such as MMLU (Hendrycks et al. 2021), AGI-Eval (Zhong et al. 2023), MMCU (Zeng 2023), C-Eval (Huang et al. 2023), M3KE (Liu et al. 2023), Lex-Treme (Niklaus et al. 2023), Big-Bench (Srivastava et al. 2022) and BIG-Bench-Hard (Suzgun et al. 2022) target at evaluating the…”
BackgroundFunctional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning GapPreprint
Srivastava, S., AnnaroseM, B., AntoP, V., Menon, S., Sukumar, A., AdwaithSamod, T., Philipose, A., Prince, S., & Thomas, S. (2024) · arXiv.org · doi:10.48550/arxiv.2402.19450 · Cited by 88 · see citing works
“Subparts of the aggregated benchmarks (MMLU [28], HELM [43], BBH [69], and AGI Eval [94]) could be functionalized.”
BackgroundPost Hoc Explanations of Language Models Can Improve Language Models
Krishna, S., Ma, J. W., Slack, D., Ghandeharioun, A., Singh, S., & Lakkaraju, H. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.11426 · Cited by 87
“[5, 7] Human-Rater [8] GPT-3 GPT-3. multiple-choice question answering platform that necessitates a wide variety of commonsense knowledge for accurately determining the correct responses; (7) Lastly, the Coin Flip [5] dataset, a synthetically generated dataset used for assessing the symbolic…”
BackgroundMethodReinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tPreprint
Dang, Q.-A., & Ngo, C. (2025) · arXiv.org · doi:10.48550/arxiv.2503.16219 · Cited by 86 · see citing works
Unintended Impacts of LLM Alignment on Global RepresentationPreprint
Ryan, M. J., Held, W. B., & Yang, D.-Y. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.15018 · Cited by 86 · see citing works
“…Sakaguchi et al., 2021; Cobbe et al., 2021), multitask knowledge (Hendrycks et al., 2021; Suzgun et al., 2023), truthfulness (Lin et al., 2022), multi-turn instruction following (Zheng et al., 2023), and similar variations of broad knowledge/reasoning/skills (Chen et al., 2021; Zhong et al., 2023).”
InfluentialAlignBench: Benchmarking Chinese Alignment of Large Language ModelsPreprint
Liu, X., Lei, X., Wang, S., Huang, Y., Feng, A., Wen, B.-S., Cheng, J., Ke, P., Xu, Y.-F., Tam, W., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.18743 · Cited by 85 · see citing works
“Faced with the challenges, researchers have proposed benchmarks (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023a; Cobbe et al., 2021; Chen et al., 2021; Bai et al., 2023b; Zhang et al., 2023) focused on measuring atomic abilities such as knowledge, math, and coding.”
BackgroundSimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models
Cheng, X.-F., Zhang, W., Zhang, S.-W., Yang, J., Guan, X.-Y., Wu, X.-J., Li, X., Zhang, G., Liu, J.-H., Mai, Y.-Y., et al. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00441 · Cited by 84 · see citing works
“Recently, Various benchmarks have been developed to evaluate factuality in LLMs [11, 16, 21, 27, 28, 36–38 , 44].”
CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean
Kim, E., Suk, J., Oh, P., Yoo, H., Thorne, J., & Oh, A. (2024) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2403.06412 · Cited by 82
“To collect data, we employ two approaches: 1 ) following the AGIEval dataset ( Zhong et al. , 2023 ) to select questions from standardized Korean exams, and 2 ) using GPT-4 to generate new questions based on textbooks.”
BackgroundMethodDo Large Language Models Know about Facts?Preprint
Hu, X., Chen, J.-Z., Li, X.-C., Guo, Y., Wen, L.-J., Yu, P. S., & Guo, Z. (2023) · arXiv.org · doi:10.48550/arxiv.2310.05177 · Cited by 81 · see citing works
“We only select questions of a multi-choice format, similar to other benchmarks (Hendrycks et al., 2021b; Zhong et al., 2023), because metrics are clearly defined (i.e. accuracy), and multi-choice questions are a simple but good proxy to evaluate the potential of advanced abilities of LLMs, which we…”
BackgroundSelf-citationSuperCLUE: A Comprehensive Chinese Large Language Model BenchmarkPreprint
Xu, L., Li, A.-Q., Zhu, L., Xue, H., Zhu, C.-T., Zhao, K., He, H., Zhang, X., Kang, Q.-Y., & Lan, Z. (2023) · arXiv.org · doi:10.48550/arxiv.2307.15020 · Cited by 81 · see citing works
“Recent benchmarks place more emphasis on assessment in human-centric scenarios by collecting high-standard admission and qualification exams on testing human-level reasoning and real-world relevance (Zhong et al. 2023; Huang et al. 2023; Zeng 2023; Gu et al. 2023).”
BackgroundInfluentialFinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language ModelsPreprint
Guo, X., Xia, H.-T., Liu, Z.-W., Cao, H.-Y., Yang, Z., Liu, Z.-Q., Wang, S., Niu, J.-Y., Wang, C.-Q., Wang, Y.-H., et al. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2308.09975 · Cited by 80 · see citing works
“This particular scenario serves to assess the models’ intrinsic capability to reason and solve problems without the aid of explicit training, as demonstrated in recent work (Zhong et al., 2023).”
BackgroundMethodSelf-citationEvaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning
Zhang, B., Zhou, K., Wei, X., Zhao, W. X., Sha, J., Wang, S., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2306.02408
“Third, evaluation metrics for intermediate reasoning steps based on formulated annotations are employed to better analyze the multi-step reasoning performance of LLMs, while existing datasets mainly focus on evaluating the outcome accuracy of solutions [6–8].”
BackgroundMethodXiao, T., & Xu, P. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114691 · Cited by 80 · see citing works
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
Weck, B., Manco, I., Benetos, E., Quinton, E., Fazekas, G., & Bogdanov, D. (2024) · International Society for Music Information Retrieval Conference · doi:10.48550/arxiv.2408.01337 · Cited by 78
CFBench: A Comprehensive Constraints-Following Benchmark for LLMsPreprint
Zhang, T., Zhu, C.-L., Shen, Y.-B., Luo, W.-J., Zhang, Y., Liang, H., Yang, F., Lin, M.-G., Qiao, Y., Chen, W.-P., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2408.01122 · Cited by 77 · see citing works
“By integrating a multitude of existing datasets, many have assessed the overall capabilities of LLMs (Bommasani, Liang, and Lee 2023; Zhong et al. 2024; Dubois et al. 2024; Chia et al. 2024; Hendrycks et al. 2021).”
Self-citationParcae: Scaling Laws For Stable Looped Language Models
Prairie, H., Novack, Z., Berg-Kirkpatrick, T., & Fu, D. Y. (2026) · arXiv.org · doi:10.48550/arxiv.2604.12946 · Cited by 76
PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
Zhang, X., Dong, Y., Wu, Y., Huang, J.-X., Jia, C.-Y., Fernando, B., Shou, M., Zhang, L.-L., & Liu, J. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.12054 · Cited by 75
“, AGIEval (Zhong et al., 2024), JEEBench (Arora et al.)”
ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse RelationsPreprint
Chan, C., Jiayang, C., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y.-Q. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14827 · Cited by 75 · see citing works
“These achievements have garnered significant attention from academia and industry, and many efforts have been made to estimate the potential of artificial general intelligence (AGI) (Bang et al., 2023; Zhong et al., 2023b; Frieder et al., 2023; Davis, 2023; Yuan et al., 2023; Wang et al., 2024).”
BackgroundPsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation
Hu, J., Dong, T., Luo, G., Ma, H., Zou, P., Sun, X. N., Guo, D., Yang, X., & Wang, M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3497725 · Cited by 46 · see citing works
“Zhong et al. [29] designed a human-centric benchmark for assessing the performance of foundation models via 20 diverse tasks across a wide variety of subjects.”
Chan, C., Jiayang, C., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y.-Q. (2024) · Findings · doi:10.18653/v1/2024.findings-eacl.47 · Cited by 73
“These achievements have garnered significant attention from academia and industry, and many efforts have been made to estimate the potential of artificial general intelligence (AGI) (Bang et al., 2023; Zhong et al., 2023b; Frieder et al., 2023; Davis, 2023; Yuan et al., 2023; Wang et al., 2024).”
Background
- Authors
- Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 2299-2314
- arXiv
- 2304.06364
- OpenAlex
- W4401042689
- Cited by
- 211
- References
- 41
- Citations by year
- 2026: 28 · 2025: 97 · 2024: 68 · 2023: 18
- FWCI
- 56.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- standardized testing, college entrance examination, LSAT, GPT-4, artificial general intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
