The DOI matches its DataCite record; 33 of the 41 references it cites were found in a registry or an index.
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Open accessAbstract
Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While LLMs have been shown to excel at these tasks (Wang et al., 2022;Zhong et al., 2023)”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Zhong et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“AGIEval [262], serves as a dedicated evaluation framework for assessing the performance of foundation models in the domain of human-centric standardized exams.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [262]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 2×
“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 3×
“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 2×
“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1×
“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602 · cited 1×
“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Research works printed but not linked: 2
- Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty- Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3622-3628, 2021.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“Furthermore, AGIEval [247], serves as a dedicated evaluation framework for assessing the performance of foundation models in the domain of human-centric standardized exams.”
BackgroundInfluentialSelf-citationLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
“In Table 24, we present fine-grained results from the English part of the AGI Eval (Zhong et al., 2023) benchmark.”
BackgroundInfluentialGemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.19786 · Cited by 1,156
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
MethodC-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.08322 · Cited by 373
“Recently, the AGIEval benchmark (Zhong et al., 2023) contains data from the Chinese College Entrance Exam, Chinese lawyer qualification test and Chinese civil service examination.”
MethodInstruction Tuning for Large Language Models: A Survey
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wang, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3777411 · Cited by 1,271
“A key distinction lies between LLM-centric designs (e.g., LLaVA), which excel at complex reasoning and detailed textual explanations, and vision-centric models (e.g., InstructPix2Pix), which offer superior performance on fine-grained visual manipulation tasks.”
DeepSeek-VL: Towards Real-World Vision-Language UnderstandingPreprint
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.05525 · Cited by 899
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., & Huang, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01239 · Cited by 431
“Data sets mPLUG-Owl2 is first pre-trained on image-text”
BackgroundFrom Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder PipelinePreprint
Li, T.-L., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T.-H., Zhu, B.-H., Gonzalez, J. E., & Stoica, I. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2406.11939 · Cited by 595
“Common ones include MMLU [16], MATH [17], GSM-8K [10], HumanEval [7], DROP [12], BigBench [37], Hel-laSwag [49], AGIEval [52], GPQA [35], as well as comprehensive collection like HELM [26].”
BackgroundCMMLU: Measuring massive multitask language understanding in Chinese
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., & Baldwin, T. (2023) · arXiv · doi:10.48550/arxiv.2306.09212 · Cited by 464
“AGIEval (Zhong et al., 2023) focuses on Chinese standardized exams, such as college entrance exams, math competitions, and lawyer qualification tests.”
BackgroundSelf-citationThe Lessons of Developing Process Reward Models in Mathematical ReasoningPreprint
Zhang, Z.-R., Zheng, C., Wu, Y., Zhang, B.-C., Lin, R.-J., Yu, B.-W., Liu, D., Zhou, J.-R., & Lin, J.-Y. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2501.07301 · Cited by 528
“We evaluate across three Chinese benchmarks including Chinese math benchmarks CMATH (Wei et al., 2023), GaoKao Math Cloze (Zhong et al., 2024), and GaoKao Math QA (Zhong et al., 2024) following Yang et al. (2024c), as shown in (Liao et al., 2024) and OlympiadBench (He et al., 2024).”
Orca: Progressive Learning from Complex Explanation Traces of GPT-4Preprint
Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., & Awadallah, A. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02707 · Cited by 603
“7M SAT-English English 206 656 SAT-Math Math 220 54 Table 6: Breakdown of tasks in AGIEval benchmark as reported in [1].”
BackgroundMethodInfluentialBenchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2309.01431 · Cited by 455
“Recently, with the development of LLMs, a series of general evaluation benchmarks have emerged, such as AGIEval (Zhong et al. 2023), C-Eval (Huang et al. 2023), AlpacaEval (Li et al. 2023b), OpenLLM Leaderboard (Edward Beeching 2023), etc.”
BackgroundLarge Language Models for Education: A survey and outlook
Wang, S., Xu, T., Li, H., Zhang, C.-L., Liang, J., Tang, J.-L., Yu, P. S., & Wen, Q.-S. (2024) · IEEE Signal Processing Magazine · doi:10.1109/msp.2025.3594309 · Cited by 438
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429
“This strategy has been successfully adopted in benchmarks such as MMLU [24] and AGIEval [85], but only text-based questions are considered, while human experts are capable of solving multimodal problems.”
MethodSelf-citationTo CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningPreprint
Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D.-W., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., & Durrett, G. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2409.12183 · Cited by 405
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454
Large Language Models for Mathematical Reasoning: Progresses and ChallengesPreprint
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., & Yin, W. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.00157 · Cited by 382
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2309.11235
“Furthermore, we also use AGIEval (Zhong et al., 2023) to prove the gen-1 eralization, where openchat-13b also achieves the top-1 average accuracy among all 13b open-source language models.”
BackgroundMethodInfluentialJiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
“…with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and AGI Eval [29] (3-5-shot, English multiple-choice questions only) Detailed results for Mistral 7B, Llama 2 7B/13B, and Code-Llama 7B are reported in Table 2.”
BackgroundMT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues
Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.401 · Cited by 79
“Subsequent benchmarks (Huang et al., 2023; Li et al., 2023a; Zhong et al., 2023; Srivastava et al., 2022; Yu et al., 2023) aim to rigorously assess LLMs’ knowledge and logic.”
BackgroundBai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332
What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
Guo, T., Guo, K., Nan, B., Liang, Z., Guo, Z., Chawla, N. V., Wiest, O., & Zhang, X. (2023) · neural information processing systems · doi:10.52202/075280-2607 · Cited by 179
“These studies underscored the complexity of achieving task-specific accuracy and functionality with LLMs. Lastly, AGIEval [62] assessed LLMs’ general abilities but noted struggles in complex reasoning tasks.”
BackgroundScaling Diffusion Language Models via Adaptation from Autoregressive ModelsPreprint
Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.17891 · Cited by 160
“We evaluate Dif-fuLLaMA’s math reasoning and in-context learning ability by evaluating on MAWPS (Koncel-Kedziorski et al., 2016) consisting of math word problems and SATMATH from AGI-eval consisting of math problems from SAT exam (Zhong et al., 2024).”
He, Z., Liang, T., Xu, J.-H., Liu, Q.-Z., Chen, X., Wang, Y., Song, L.-F., Yu, D., Liang, Z.-W., Wang, W., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.11456 · Cited by 307
“We performed decontamination against a comprehensive suite of mathematics and STEM benchmarks, including MATH (Hendrycks et al., 2021b), AIME (MAA, a), AMC (MAA, b), Minerva Math (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), Omni-MATH (Gao et al., 2024), MathOdyssey (Fang et al., 2024), GAOKAO (Zhong et a…”
Self-citationSecAlign: Defending Against Prompt Injection with Preference Optimization
Chen, S., Zharmagambetov, A. S., Mahloujifar, S., Chaudhuri, K., Wagner, D., & Guo, C. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744836 · Cited by 193
“We run more utility benchmarks (MMLU [56], Winogrande [57], AGIEval[58],andCommonSenseQA[59])onMistral-7BandLlama3- 8Btocheckthemodel’sfunctionoutsidetheAlpacaEval2bench-markpresentedinthemainexperiments.”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language ModelsPreprint
Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., & Wang, W. W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10635 · Cited by 289
“Previous works have relied on human labor to annotate error reasons into different categories, which is both expensive and time-consuming [49].”
BackgroundMethodInfluentialGemma: Open Models Based on Gemini Research and Technology
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.08295 · Cited by 284
“These specific benchmarks are: ARC (Clark et al., 2018), CommonsenseQA (Talmor et al., 2019), Big Bench Hard (Suzgun et al., 2022), and AGI Eval (English-only) (Zhong et al., 2023).”
BackgroundFLASK: Fine-grained Language Model Evaluation based on Alignment Skill SetsPreprint
Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J. P., Kim, J., & Seo, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10928 · Cited by 138
“Conventionally, the performance of LLMs is measured by assessing the model on separate benchmarks using automatic metrics such as accuracy for knowledge/reasoning tasks or ROUGE for longform text generation (Chung et al., 2022; Hendrycks et al., 2020; Suzgun et al., 2022; Wang et al., 2022c; Gao et al., 2021; Zhong et…”
MethodDefending Large Language Models Against Jailbreaking Attacks Through Goal PrioritizationPreprint
Zhang, Z., Yang, J., Ke, P., Mi, F., Wang, H.-N., & Huang, M.-L. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.09096 · Cited by 238
“Consequently, models are capable of learning to prioritize safety over helpfulness in straightforward inquiries but struggle to discern the goal priority in out-of-distribution (OOD) jailbreaking cases.”
BackgroundEvaluating the Performance of Large Language Models on GAOKAO BenchmarkPreprint
Zhang, X., Li, C.-Y., Zong, Y., Ying, Z., He, L., Qiu, X.-P., Sun, T., Li, P., Meng, S.-Q., Zheng, Y.-J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.12474 · Cited by 236
The effect of sampling temperature on problem solving in large language models
Renze, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.432 · Cited by 212
VITA: Towards Open-Source Interactive Omni Multimodal LLMPreprint
Fu, C., Lin, H., Long, Z., Shen, Y., Dai, Y., Meng, Z., Zhang, Y.-F., Dong, S., Li, Y., Wang, X., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.05211 · Cited by 183
“To validate the efficacy of our training process for language model, we evaluate our trained model “Mixtral 8x7B Ours” against the official version “Mixtral 8x7B Instruct”, on four datasets: C-EVAL [20], AGIEVAL [58], MMLU [18], and GSM8K [10].”
MethodLarge Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient ReasoningPreprint
Yue, M., Zhao, J., Zhang, M., Du, L., & Yao, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.03094 · Cited by 86
“Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in reasoning tasks (Rae et al., 2021; Lewkowycz et al., 2022; Zhong et al., 2023).”
BackgroundScientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 216
Self-citationRemoving RLHF Protections in GPT-4 via Fine-Tuning
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T., & Kang, D. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.05553 · Cited by 202
“We chose the informativeness metric on Truth-fulQA (Lin et al., 2021), AGIEval (Zhong et al., 2023), MMLU (Hendrycks et al., 2020), gsm-8k (Cobbe et al., 2021), and WinoGrande (Sakaguchi et al., 2021) as orthogonal evaluations.”
MethodMathScale: Scaling Instruction Tuning for Mathematical ReasoningPreprint
Tang, Z., Zhang, X., Wang, B., & Wei, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.02884 · Cited by 114
A Comprehensive Study of Knowledge Editing for Large Language ModelsPreprint
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S., Wang, M., Xi, Z., Mao, S., Zhang, J., Ni, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.01286 · Cited by 322
“These benchmarks include Com-monsenseQA [187], PIQA [188], Xsum [189], and TriviaQA [190], as well as specific tasks from the MMLU [191] and AGIEval [192] suites, which are known for their distinguished evaluation criteria suites.”
MethodBeyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling LawsPreprint
Sardana, N., Portes, J., Doubov, S., & Frankle, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.00448 · Cited by 112
“• Reading Comprehension : SQuAD (Rajpurkar et al., 2016), BoolQ (Clark et al., 2019), CoQA (Reddy et al., 2019), and AGI Eval (Zhong et al., 2023).”
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific ResearchPreprint
Sun, L., Han, Y., Zhao, Z., Ma, D., Shen, Z.-W., Chen, B.-C., Chen, L., & Yu, K. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.13149 · Cited by 187
“AGIEval (Zhong et al. 2023) serves as an evaluation framework for assessing the performance of foundation models in human-centric standardized exams.”
BackgroundParameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186
CMB: A Comprehensive Medical Benchmark in Chinese
Wang, X., Chen, G. H., Song, D., Zhang, Z., Chen, Z., Xiao, Q., Jiang, F., Li, J., Wan, X., Wang, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08833 · Cited by 94
“…ability benchmarks include comprehensive test suites, each targeting different aspects of LLM’s proficiency, ranging from handling multi-turn dialogues (Zheng et al., 2023) to gauging language comprehension and reasoning abilities (Srivastava et al., 2022; Zhang et al., 2023a; Zhong et al., 2023).”
BackgroundMMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Zhao, Y., Zhang, H., Xie, L., Hu, T., Gan, G., Long, Y., Hu, Z., Chen, W., Li, C., Xu, Z., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00793 · Cited by 80
“Early benchmarks focused on domain-specific tasks for textual domains, establishing a foundation for assessing the models’ strengths and limitations in expert reasoning [21, 27, 60, 133, 147, 150, 168, 169].”
Self-citationLawBench: Benchmarking Legal Knowledge of Large Language ModelsPreprint
Fei, Z., Shen, X., Zhu, D., Zhou, F., Han, Z., Zhang, S., Chen, K., Shen, Z., & Ge, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.16289 · Cited by 166
“In contrast to assessments that primarily focus on testing model’s ability to pass legal bar exams [5; 86], which are not always representative of the actual use-cases for LLMs 2 , our emphasis is on examining a structured set of legal skills required in real-world scenarios.”
BackgroundResultPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.14219 · Cited by 166
Self-citationA survey of multilingual large language models
Qin, L., Chen, Q., Zhou, Y., Zhi, C., Li, Y., Liao, L., Li, M., Che, W., & Yu, P. S. (2025) · Patterns · doi:10.1016/j.patter.2024.101118 · Cited by 282
M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models
Zhang, W., Aljunied, S. M., Gao, C., Chia, Y. K., & Bing, L. (2023) · arXiv · doi:10.48550/arxiv.2306.05179
“Firstly, the majority of these benchmarks focus on questions in English [17, 43], neglecting the evaluation of a model’s performance in a multilingual context.”
BackgroundMethodInfluentialRetrieval-Augmented Generation (RAG) Chatbots for Education: A Survey of Applications
Swacha, J., & Gracel, M. (2025) · Applied Sciences · doi:10.3390/app15084234 · Cited by 198
Causal Inference Meets Deep Learning: A Comprehensive Survey
Jiao, L., Wang, Y., Liu, X., Li, L., Liu, F., Ma, W., Guo, Y., Chen, P., Yang, S., & Hou, B. (2024) · Research · doi:10.34133/research.0467 · Cited by 141
“Zhong et al. [139] and Nori et al. [140] used benchmark tests and question-and-answer evaluations to score large models by evaluating the accuracy of their answers to a set of questions.”
BackgroundSelf-citationHuang, H., Bu, X., Zhou, H., Qu, Y., Liu, J., Yang, M., Xu, B. L., & Zhao, T. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.306 · Cited by 176
“Therefore, after the emergence of LLMs, numerous efforts have been made to construct benchmark testsets that cover a variety of tasks for evaluation (Srivastava et al., 2023; Zhong et al., 2024).”
BackgroundJiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143
“…with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16] (5-shot), BBH [29] (3-shot), and AGI Eval [34] (3-5-shot, English multiple-choice questions only) Figure 2: Performance of Mixtral and different Llama models on a wide range of benchmarks .”
BackgroundBaichuan-Omni-1.5 Technical ReportPreprint
Li, Y., Liu, J., Zhang, T., Zhang, T., Chen, S., Li, T., Li, Z., Liu, L., Ming, L., Dong, G., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.15368 · Cited by 147
“5, we utilize 4 comprehensive benchmarks, incuding MMLU [55], CMMLU [79], AGIEval [192], C-Eval [59] and GAOKAO-Bench [185].”
Self-citationBuehler, M. J. (2023) · Applied Mechanics Reviews · doi:10.1115/1.4063843 · Cited by 140
“For instance, in the AGI Eval test [31] it averages 0.”
BackgroundMMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.01574 · Cited by 131
“For example, AGIEval [48] focuses on general-exam questions from SAT, Gaokao, GRE, etc. ARC [12] focuses on science-based questions.”
BackgroundSelf-citationLISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-TuningPreprint
Pan, R., Liu, X., Diao, S., Pi, R., Zhang, J., Han, C., & Zhang, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.17919 · Cited by 63
“…assess LLMs on multiple aspects; MMLU [68] includes a total of 57 tasks with 14,079 questions covering a broad spectrum of world knowledge; AGIEval [69] serves as a human-centric benchmark for general abilities, comprising 9,316 instances; WinoGrande [70] is a large-scale dataset for commonsense…”
DyVal: Dynamic Evaluation of Large Language Models for Reasoning TasksPreprint
Zhu, K., Chen, J., Wang, J., Gong, N. Z., Yang, D., & Xie, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.17167 · Cited by 70
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 91
“Evaluates models’ ability on diverse domains of tasks MMLU [73], MMMU [236], MMStar [27], M3GIA [191], AGIEval [253]”
Baichuan 2: Open Large-scale Language Models
Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.10305 · Cited by 125
“AGIEval (Zhong et al., 2023) is a human-centric benchmark specifically designed to evaluate general abilities like human cognition and problem-solving.”
BackgroundSelf-citationCValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility
Xu, G., Liu, J., Yan, M., Xu, H., Si, J., Zhou, Z., Yi, P., Gao, X., Sang, J., Zhang, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09705 · Cited by 107
“Most of these focus on assessing the helpfulness of the LLMs, such as AGIEval (Zhong et al., 2023b) and MMCU (Zeng, 2023) for Chinese and English College Entrance Exams, M3KE (Liu et al.”
BackgroundInfluentialChatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118
“Prominent examples in this category are MMLU (Hendrycks et al., 2020), Hel-laSwag (Zellers et al., 2019), GSM-8K (Cobbe et al., 2021), BigBench (Srivastava et al., 2023), AGIEval (Zhong et al., 2023), and HumanEval (Chen et al., 2021).”
BackgroundSeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning
Wang, B., Liu, Z., Huang, X., Jiao, F., Ding, Y., Aw, A., & Chen, N. F. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2309.04766 · Cited by 118
“Furthermore, we incorporate DREAM for English dialogue comprehension, OCNLI and C3 for Chinese comprehension, and Indo-Emotion dataset (Saputri et al., 2018; Wilie et al., 2020) to gauge emotion comprehension in Indonesian.”
BackgroundMethodInfluentialExploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
Xie, T., Foster, D. J., Krishnamurthy, A., Rosset, C., Awadallah, A., & Rakhlin, A. (2024) · arXiv.org · doi:10.48550/arxiv.2405.21046 · Cited by 117
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
Can Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Chen, L., Wang, X., Huang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.09150 · Cited by 43
“Evaluated Models We evaluate a total of 34 models that demonstrated exceptional performance on other benchmarks (Huang et al. 2023; Dubois et al. 2023; Zhong et al. 2023), ranging from their model size, supported context length, and instruction tuning data size, as illustrated in Appendix.”
BackgroundMethodInfluentialThe Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models
Renze, M., & Guven, E. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852493 · Cited by 69
“We preprocessed the data into a standard data format and randomly selected 100 questions from each of the ten benchmarks to create an exam with 1,000 MCQA problems.”
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.04434 · Cited by 109
“Note that AGIEval includes both English and Chinese subsets.”
Large Language Models for Education: A SurveyPreprint
Xu, H., Gan, W., Qi, Z., Wu, J., & Yu, P. S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.13001 · Cited by 137
“MathGPT has an accuracy rate of 60.34% in the benchmark test AGIEval, which can help students solve mathematical problems efficiently [142].”
BackgroundInfluentialInadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
“Prevalence : 18/23 A critical issue identified in LLM benchmarks is the inconsistency in implementation across different research teams, as observed in [7]–[12], [14], [31], [39]–[42], [44], [46]– [49], [52] (Appendix B-A).”
BackgroundMethodInfluentialLuo, G., Yang, X., Dou, W., Wang, Z., Liu, J., Dai, J., Qiao, Y., & Zhu, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02324 · Cited by 23
“NLP benchmarks include MMLU [29], CMMLU [36], AGIEval [89] and MATH [30].”
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
Huang, Z., Wang, Z., Xia, S., Li, X.-F., Zou, H.-Y., Xu, R.-J., Fan, R.-Z., Ye, L., Chern, E., Ye, Y.-X., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.12753 · Cited by 105
“This leads to the creation of benchmarks such as MMLU [18], AGIEval[63], C-Eval [22], and CMMLU [26], which pushed the limits of language models with multidisciplinary, multilingual, and knowledge-intensive tasks.”
BackgroundMethodALLaM: Large Language Models for Arabic and EnglishPreprint
Bari, M. S., Alnumay, Y., Alzahrani, N. A., Alotaibi, N. M., Alyahya, H. A., Alrashed, S., Mirza, F., Alsubaie, S., Alahmed, H. A., Alabduljabbar, G., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.15390 · Cited by 131
“…Easy and Challenge (Clark et al., 2018), TriviaQA (Joshi et al., 2017), BoolQ (Clark et al., 2019), NQ Open (Kwiatkowski et al., 2019), AGIEval (Zhong et al., 2023), Exams-Ar (Hardalov et al., 2020), MMLU Arabic (tr) (Huang et al., 2023), MMLU Arabic (MBZU) (Koto et al., 2024) , and ETEC…”
BackgroundDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.03300 · Cited by 101
“Chinese benchmarks include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong et al., 2023), and Gaokao-MathQA (Zhong et al., 2023).”
BackgroundMethodInfluentialLLM2LLM: Boosting LLMs with Novel Iterative Data EnhancementPreprint
Lee, N., Wattanawong, T., Kim, S., Mangalam, K., Shen, S., Anumanchipalli, G., Mahoney, M. W., Keutzer, K., & Gholami, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.15042 · Cited by 51
“For many of these general benchmarks (Hendrycks et al., 2020; Zhong et al., 2023), LLMs are prompted with custom instructions or in-context examples.”
BackgroundDeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
Self-citationKe, P., Wen, B., Feng, A., Liu, X., Lei, X., Cheng, J., Wang, S., Zeng, A., Dong, Y., Wang, H., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.704 · Cited by 103
“…such as multi-choice QA to measure the performance of LLMs via simple objective metrics (such as accuracy and F1 score) (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023b), which may deviate from the common usage of LLMs and may not exactly reflect the ability of LLMs in…”
BackgroundCMATH: Can Your Language Model Pass Chinese Elementary School Math Test?
Wei, T., Luan, J., Liu, W., Dong, S., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2306.16636
Data-Juicer: A One-Stop Data Processing System for Large Language Models
Chen, D., Huang, Y., Ma, Z., Chen, H., Pan, X., Ge, C., Gao, D., Xie, Y., Liu, Z., Gao, J., et al. (2024) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3626246.3653385 · Cited by 72
Large language model assisted fine-grained knowledge graph construction for robotic fault diagnosis
Liao, X., Chen, C., Wang, Z., Liu, Y., Wang, T., & Cheng, L. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103134 · Cited by 88
EmoBench: Evaluating the Emotional Intelligence of Large Language ModelsPreprint
Sabour, S., Liu, S., Zhang, Z., Liu, J. M., Zhou, J., Sunaryo, A., Li, J.-Z., Lee, T. M. C., Mihalcea, R., & Huang, M.-L. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.12071 · Cited by 89
“However, despite their apparent proficiency in a variety of downstream tasks, such as question answering, and summarization (Zhou et al., 2023a; Zhong et al., 2023), research on evaluating EI capabilities for LLMs has been limited.”
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge EvaluationPreprint
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y.-X., Jiang, S.-H., Xiong, Z., Li, Z.-H., Wu, W.-J., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2306.05783 · Cited by 89
“Meanwhile, other research, such as MMLU (Hendrycks et al. 2021), AGI-Eval (Zhong et al. 2023), MMCU (Zeng 2023), C-Eval (Huang et al. 2023), M3KE (Liu et al. 2023), Lex-Treme (Niklaus et al. 2023), Big-Bench (Srivastava et al. 2022) and BIG-Bench-Hard (Suzgun et al. 2022) target at evaluating the…”
BackgroundFunctional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap
Srivastava, S., B, A. M., P, A., Menon, S., Sukumar, A., T, A. S., Philipose, A., Prince, S., & Thomas, S. (2024) · arXiv · doi:10.48550/arxiv.2402.19450 · Cited by 79
“Subparts of the aggregated benchmarks (MMLU [28], HELM [43], BBH [69], and AGI Eval [94]) could be functionalized.”
BackgroundPost Hoc Explanations of Language Models Can Improve Language ModelsPreprint
Krishna, S., Ma, J., Slack, D., Ghandeharioun, A., Singh, S., & Lakkaraju, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11426 · Cited by 16
“[5, 7] Human-Rater [8] GPT-3 GPT-3. multiple-choice question answering platform that necessitates a wide variety of commonsense knowledge for accurately determining the correct responses; (7) Lastly, the Coin Flip [5] dataset, a synthetically generated dataset used for assessing the symbolic…”
BackgroundMethodReinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tPreprint
Dang, Q.-A., & Ngo, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.16219 · Cited by 94
Unintended Impacts of LLM Alignment on Global RepresentationPreprint
Ryan, M. J., Held, W. B., & Yang, D.-Y. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.15018 · Cited by 86
“…Sakaguchi et al., 2021; Cobbe et al., 2021), multitask knowledge (Hendrycks et al., 2021; Suzgun et al., 2023), truthfulness (Lin et al., 2022), multi-turn instruction following (Zheng et al., 2023), and similar variations of broad knowledge/reasoning/skills (Chen et al., 2021; Zhong et al., 2023).”
InfluentialAlignBench: Benchmarking Chinese Alignment of Large Language ModelsPreprint
Liu, X., Lei, X., Wang, S., Huang, Y., Feng, A., Wen, B.-S., Cheng, J., Ke, P., Xu, Y.-F., Tam, W., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.18743 · Cited by 85
“Faced with the challenges, researchers have proposed benchmarks (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023a; Cobbe et al., 2021; Chen et al., 2021; Bai et al., 2023b; Zhang et al., 2023) focused on measuring atomic abilities such as knowledge, math, and coding.”
BackgroundSimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models
Cheng, X., Zhang, W., Zhang, S., Yang, J. T., Guan, X., Wu, X., Li, X., ZHANG, G., Liu, J., Mai, Y., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00441 · Cited by 40
“Recently, Various benchmarks have been developed to evaluate factuality in LLMs [11, 16, 21, 27, 28, 36–38 , 44].”
CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean
Kim, E., Suk, J., Oh, P., Yoo, H., Thorne, J., & Oh, A. (2024) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2403.06412 · Cited by 82
“To collect data, we employ two approaches: 1 ) following the AGIEval dataset ( Zhong et al. , 2023 ) to select questions from standardized Korean exams, and 2 ) using GPT-4 to generate new questions based on textbooks.”
BackgroundMethodDo Large Language Models Know about Facts?Preprint
Hu, X., Chen, J., Li, X., Guo, Y., Wen, L., Yu, P. S., & Guo, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05177 · Cited by 51
“We only select questions of a multi-choice format, similar to other benchmarks (Hendrycks et al., 2021b; Zhong et al., 2023), because metrics are clearly defined (i.e. accuracy), and multi-choice questions are a simple but good proxy to evaluate the potential of advanced abilities of LLMs, which we…”
BackgroundSelf-citationSuperCLUE: A Comprehensive Chinese Large Language Model BenchmarkPreprint
Xu, L. J., Li, A., Zhu, L., Xue, H., Zhu, C., Zhao, K., He, H., Zhang, X., Kang, Q., & Lan, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.15020 · Cited by 128
“Recent benchmarks place more emphasis on assessment in human-centric scenarios by collecting high-standard admission and qualification exams on testing human-level reasoning and real-world relevance (Zhong et al. 2023; Huang et al. 2023; Zeng 2023; Gu et al. 2023).”
BackgroundInfluentialFinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language ModelsPreprint
Guo, X., Xia, H.-T., Liu, Z.-W., Cao, H.-Y., Yang, Z., Liu, Z.-Q., Wang, S., Niu, J.-Y., Wang, C.-Q., Wang, Y.-H., et al. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2308.09975 · Cited by 80
“This particular scenario serves to assess the models’ intrinsic capability to reason and solve problems without the aid of explicit training, as demonstrated in recent work (Zhong et al., 2023).”
BackgroundMethodSelf-citationEvaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning
Zhang, B., Zhou, K., Wei, X., Zhao, W. X., Sha, J., Wang, S., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2306.02408 · Cited by 32
“Third, evaluation metrics for intermediate reasoning steps based on formulated annotations are employed to better analyze the multi-step reasoning performance of LLMs, while existing datasets mainly focus on evaluating the outcome accuracy of solutions [6–8].”
BackgroundMethodXiao, T., & Xu, P. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114691 · Cited by 86
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language ModelsPreprint
Weck, B., Manco, I., Benetos, E., Quinton, E., Fazekas, G., & Bogdanov, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.01337 · Cited by 53
CFBench: A Comprehensive Constraints-Following Benchmark for LLMsPreprint
Zhang, T., Zhu, C.-L., Shen, Y.-B., Luo, W.-J., Zhang, Y., Liang, H., Yang, F., Lin, M.-G., Qiao, Y., Chen, W.-P., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2408.01122 · Cited by 77
“By integrating a multitude of existing datasets, many have assessed the overall capabilities of LLMs (Bommasani, Liang, and Lee 2023; Zhong et al. 2024; Dubois et al. 2024; Chia et al. 2024; Hendrycks et al. 2021).”
Self-citationParcae: Scaling Laws For Stable Looped Language ModelsPreprint
Prairie, H., Novack, Z., Berg-Kirkpatrick, T., & Fu, D. Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.12946 · Cited by 66
PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
Zhang, X., Dong, Y., Wu, Y., Huang, J.-X., Jia, C.-Y., Fernando, B., Shou, M., Zhang, L.-L., & Liu, J. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.12054 · Cited by 75
“, AGIEval (Zhong et al., 2024), JEEBench (Arora et al.)”
ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse RelationsPreprint
Chan, C., Jiayang, C., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y.-Q. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14827 · Cited by 75
“These achievements have garnered significant attention from academia and industry, and many efforts have been made to estimate the potential of artificial general intelligence (AGI) (Bang et al., 2023; Zhong et al., 2023b; Frieder et al., 2023; Davis, 2023; Yuan et al., 2023; Wang et al., 2024).”
BackgroundPsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation
Hu, J., Dong, T., Luo, G., Ma, H., Zou, P., Sun, X. N., Guo, D., Yang, X., & Wang, M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3497725 · Cited by 58
“Zhong et al. [29] designed a human-centric benchmark for assessing the performance of foundation models via 20 diverse tasks across a wide variety of subjects.”
Chan, C., Jiayang, C., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y.-Q. (2024) · Findings · doi:10.18653/v1/2024.findings-eacl.47 · Cited by 73
“These achievements have garnered significant attention from academia and industry, and many efforts have been made to estimate the potential of artificial general intelligence (AGI) (Bang et al., 2023; Zhong et al., 2023b; Frieder et al., 2023; Davis, 2023; Yuan et al., 2023; Wang et al., 2024).”
Background
Metadata
- Authors
- Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 2299-2314
- arXiv
- 2304.06364
- OpenAlex
- W4401042689
- Cited by
- 211
- References
- 41
- Citations by year
- 2026: 28 · 2025: 97 · 2024: 68 · 2023: 18
- FWCI
- 56.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- standardized testing, college entrance examination, LSAT, GPT-4, artificial general intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
