AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Open accessAbstract
Assessing foundation models' abilities for human-level tasks is crucial for Artificial General Intelligence (AGI) development.Traditional benchmarks, which rely on artificial datasets, may not accurately represent these capabilities.In this paper, we introduce AGIEval, a novel bilingual benchmark designed to assess foundation models in the context of humancentric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests.We evaluate several state-of-the-art foundation models on our benchmark.Impressively, we show that GPT-4 exceeds the average human performance in SAT, LSAT, and math contests, with 95% accuracy on SAT Math and 92.5% on the Chinese college entrance English exam.This demonstrates the exceptional performance of contemporary foundation models.In contrast, we also find that GPT-4 is less proficient in tasks requiring complex reasoning or specific domain knowledge.Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal their strengths and limitations, providing valuable insights into future directions for enhancing general capabilities.By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a meaningful and robust evaluation of foundation models' performance in real-world scenarios 1 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks.
“The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks [3], [13], [30].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [13]
Unlike traditional benchmarks with clear numerical metrics, LLM benchmarking often involves subjective judgments and complex evaluation criteria.
“Unlike traditional benchmarks with clear numerical metrics, LLM benchmarking often involves subjective judgments and complex evaluation criteria [13], [25].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872.
- Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509. · arXiv
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4):0-6.
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167.
- Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. Logiqa: a challenge dataset for machine reading comprehen- sion with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on Interna- tional Joint Conferences on Artificial Intelligence, pages 3622-3628.
- Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word pre- diction requiring a broad discourse context. arXiv preprint arXiv:1606.06031. · arXiv
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
- Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of com- plex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2201-2216.
- Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Jec- qa: A legal-domain question answering dataset. In Proceedings of AAAI.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationMMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · see citing works
Self-citationBenchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454 · see citing works
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q.-H., Xu, H., Ye, J., Yan, M.-S., Hu, A., Liu, H.-W., Qian, Q., Zhang, J., Huang, F., & Zhou, J.-R. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01239 · Cited by 763 · see citing works
Parameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186 · see citing works
Retrieval-Augmented Generation (RAG) Chatbots for Education: A Survey of Applications
Swacha, J., & Gracel, M. (2025) · Applied Sciences · doi:10.3390/app15084234 · Cited by 148 · see citing works
Buehler, M. J. (2023) · Applied Mechanics Reviews · doi:10.1115/1.4063843 · Cited by 132 · see citing works
Scientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 155 · see citing works
Self-citationDatasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Causal Inference Meets Deep Learning: A Comprehensive Survey
Jiao, L., Wang, Y., Liu, X., Li, L., Liu, F., Ma, W., Guo, Y., Chen, P., Yang, S., & Hou, B. (2024) · Research · doi:10.34133/research.0467 · Cited by 122 · see citing works
Self-citationA survey of multilingual large language models
Qin, L., Chen, Q., Zhou, Y., Zhi, C., Li, Y., Liao, L., Li, M., Che, W., & Yu, P. S. (2025) · Patterns · doi:10.1016/j.patter.2024.101118 · Cited by 197 · see citing works
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
Large language model assisted fine-grained knowledge graph construction for robotic fault diagnosis
Liao, X., Chen, C., Wang, Z., Liu, Y., Wang, T., & Cheng, L. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103134 · Cited by 88 · see citing works
Xiao, T., & Xu, P. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114691 · Cited by 80 · see citing works
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 82 · see citing works
Liu, Y., Li, D., Wang, K., Xiong, Z., Shi, F., Wang, J., Li, B., & Hang, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103809 · Cited by 81 · see citing works
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 72 · see citing works
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115 · see citing works
Exploring the Impact of Temperature on Large Language Models: Hot or Cold?
Li, L., Sleem, L., Gentile, N., Nichil, G., & State, R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.135 · Cited by 62 · see citing works
The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S. (2026) · ACM Computing Surveys · doi:10.1145/3773080 · Cited by 58 · see citing works
Ray, P. P. (2023) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2023.100136 · Cited by 57 · see citing works
Assessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57 · see citing works
Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 47 · see citing works
PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation
Hu, J., Dong, T., Luo, G., Ma, H., Zou, P., Sun, X. N., Guo, D., Yang, X., & Wang, M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3497725 · Cited by 46 · see citing works
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51 · see citing works
Self-citationScaling Large Language Models for Next-Generation Single-Cell AnalysisPreprint
Rizvi, S. A., Levine, D., Patel, A. M., Zhang, S., Wang, E., Perry, C. J., Vrkic, I., Constante, N. M., Fu, Z., He, S., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.04.14.648850 · Cited by 49 · see citing works
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 38 · see citing works
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
Sun, L., Yang, H., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., & Yu, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29872 · Cited by 48 · see citing works
Lion: Adversarial Distillation of Proprietary Large Language Models
Jiang, Y., Chan, C., Chen, M., & Wang, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.189 · Cited by 55 · see citing works
INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Chia, Y. K., Hong, P., Bing, L., & Poria, S. (2023) · arXiv · doi:10.48550/arxiv.2306.04757 · Cited by 36 · see citing works
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 37 · see citing works
Survey on Factuality in Large Language Models
Wang, C., Liu, X., Yue, Y., Guo, Q., Hu, X., Tang, X., Zhang, T., Cheng, J., Yao, Y., Hu, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3742420 · Cited by 221 · see citing works
Self-citationJia, J.-Y., Wang, T.-R., Zhang, Y., & Wang, G.-D. (2024) · Asia Pacific Journal of Education · doi:10.1080/02188791.2023.2286920 · Cited by 36 · see citing works
Gao, L., Lu, J., Shao, Z., Lin, Z., Yue, S., Ieong, C., Sun, Y., Zauner, R. J., Wei, Z., & Chen, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456145 · Cited by 33 · see citing works
Evaluating large language models on geospatial tasks: a multiple geospatial task benchmarking study
Xu, L.-C., Zhao, S., Lin, Q.-M., Chen, L., Luo, Q.-Q., Wu, S.-S., Ye, X., Feng, H.-L., & Du, Z.-H. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2480268 · Cited by 40 · see citing works
Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A SurveyPreprint
Li, Z., Wu, X., Du, H., Nghiem, H. T., & Shi, G. (2025) · Qeios · doi:10.32388/gxr68q · Cited by 30 · see citing works
Berti, A., Kourani, H., Häfke, H., Li, C.-Y., & Schuster, D. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-61007-3_2 · Cited by 31 · see citing works
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y., Jiang, S., Xiong, Z., Li, Z., Wu, W., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29767 · Cited by 28 · see citing works
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43 · see citing works
Self-citationCan Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Liang, J., & Xiao, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29777 · Cited by 24 · see citing works
Evaluating Large Language Models for Material Selection
Grandi, D., Jain, Y. P., Groom, A., Cramer, B., & McComb, C. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4066730 · Cited by 24 · see citing works
Post Hoc Explanations of Language Models Can Improve Language ModelsPreprint
Krishna, S., Ma, J., Slack, D., Ghandeharioun, A., Singh, S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3006112/v1 · Cited by 23 · see citing works
Jiao, J., Afroogh, S., Murali, A., Chen, K., Atkinson, D., & Dhurandhar, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-18489-7 · Cited by 23 · see citing works
Feedback on Feedback: Student’s Perceptions for Feedback from Teachers and Few-Shot LLMs
Rüdian, S., Podelo, J., Kuzilek, J., & Pinkwart, N. (2025) · International Conference on Learning Analytics and Knowledge · doi:10.1145/3706468.3706479 · Cited by 35 · see citing works
SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models
Qin, C., Chen, X., Wang, C., Wu, P., Chen, X., Cheng, Y., Zhao, J., Xiao, M., Dong, X., Long, Q., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737403 · Cited by 24 · see citing works
A Comprehensive Review on the Application of Large Language Models (LLMs) in Power Systems
Ghafari, M., Sami, A., Rezapour, H., Ghayour, S. S., Lin, F., Başaran, K., Lăzăroiu, G. C., & Siano, P. (2025) · IEEE Access · doi:10.1109/access.2025.3637226 · Cited by 20 · see citing works
So you want your private LLM at home? A survey and benchmark of methods for efficient GPTs
Tuggener, L., Sager, P., Taoudi-Benchekroun, Y., Grewe, B. F., & Stadelmann, T. (2024) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds60720.2024.00036 · Cited by 22 · see citing works
Goto, T., Ono, K., & Morita, A. (2024) · doi:10.36227/techrxiv.171173447.70655950/v1 · Cited by 19 · see citing works
Large Language Model Evaluation Criteria Framework in Healthcare: Fuzzy MCDM Approach
Alabool, H. (2025) · SN Computer Science · doi:10.1007/s42979-024-03533-6 · Cited by 22 · see citing works
Artificial Intelligence in the 21st Century
奥宮, 正. (2023) · Research on Intelligent Manufacturing and Assembly · doi:10.25082/rima.2023.01.002 · Cited by 18 · see citing works
The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models
Renze, M., & Guven, E. (2024) · 2024 2nd International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852493 · Cited by 110 · see citing works
Mylrea, M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29246-0.00007-9 · Cited by 17 · see citing works
LLMEval: A Preliminary Study on How to Evaluate Large Language Models
Zhang, Y., Zhang, M., Yuan, H., Liu, S., Shi, Y., Gui, T., Zhang, Q., & Huang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29934 · Cited by 19 · see citing works
Using GPT-4 to guide causal machine learning
Constantinou, A. C., Kitson, N. K., & Zanga, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126120 · Cited by 14 · see citing works
Luo, G., Yang, X., Dou, W., Wang, Z., Liu, J., Dai, J., Qiao, Y., & Zhu, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02324 · Cited by 23 · see citing works
Generative AI-Based Text Generation Methods Using Pre-Trained GPT 2 ModelPreprint
Pandey, R., Waghela, H., Rakshit, S., Rangari, A., Singh, A., Kumar, R., Ghoshal, R., & Sen, J. (2024) · doi:10.36227/techrxiv.171216659.95569463/v1 · Cited by 15 · see citing works
Generative Inference of Large Language Models in Edge Computing: An Energy Efficient Approach
Yuan, X., Li, H., Ota, K., & Dong, M.-X. (2024) · International Conference on Wireless Communications and Mobile Computing · doi:10.1109/iwcmc61514.2024.10592339 · Cited by 15 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12 · see citing works
MindLLM: Lightweight large language model pre-training, evaluation and domain application
Yang, Y., Sun, H., Li, J., Liu, R., Li, Y., Liu, Y., Gao, Y., & Huang, H. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.001 · Cited by 12 · see citing works
Investigating the Efficacy of ChatGPT-3.5 for Tutoring in Chinese Elementary Education Settings
Bai, Y., Li, J., Shen, J., & Zhao, L. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3464560 · Cited by 8 · see citing works
A Framework for Domain-Specific Dataset Creation and Adaptation of Large Language Models
Balaskas, G., Papadopoulos, H., Pappa, D., Loisel, Q. E. A., & CHASTIN, S. F. M. (2025) · Computers · doi:10.3390/computers14050172 · Cited by 12 · see citing works
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15124 · Cited by 12 · see citing works
Self-citationThe ethical evaluation of large language models and its optimization
Lyu, Y., & Du, Y. (2025) · AI and Ethics · doi:10.1007/s43681-024-00654-9 · Cited by 11 · see citing works
ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination
Li, D., Yu, J., Hu, B., Xu, Z., & Zhang, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.129 · Cited by 10 · see citing works
Gong, L., Chen, J., & Wu, F. Q. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3564177 · Cited by 10 · see citing works
Retrieval augmented generation for building datasets from scientific literature
Maharana, P. R., Verma, A., & Joshi, K. (2025) · Journal of Physics Materials · doi:10.1088/2515-7639/ade1fa · Cited by 11 · see citing works
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Zhao, Y., Zhang, H., Xie, L., Hu, T., Gan, G., Long, Y., Hu, Z., Chen, W., Li, C., Xu, Z., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00793 · Cited by 14 · see citing works
Self-citationArtificial intelligence techniques for human-machine interaction
Mukhtar, H. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-29150-0.00010-x · Cited by 9 · see citing works
The Antecedents of Transformer Models
Dennis, S., Shabahang, K. D., & Yim, H. (2024) · Current Directions in Psychological Science · doi:10.1177/09637214241279504 · Cited by 10 · see citing works
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 9 · see citing works
A Survey on Human Preference Learning for Aligning Large Language Models
Jiang, R., Chen, K., Bai, X., He, Z., Li, J., Yang, M., Zhao, T., Nie, L., & Zhang, M. (2025) · ACM Computing Surveys · doi:10.1145/3773279 · Cited by 25 · see citing works
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9 · see citing works
Performance evaluations of large language models for customer service
Li, F., Wang, Y., Xu, Y., Wang, S., Liang, J., Chen, Z., Liu, W., Feng, Q., Duan, T., Huang, Y., et al. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02432-9 · Cited by 7 · see citing works
Self-citationDetection of Disease Features on Retinal OCT Scans Using RETFound
Du, K., Nair, A. R., Shah, S. V., Gadari, A., Vupparaboina, S. C., Bollepalli, S. C., Sutharahan, S., Sahel, J., Jana, S., Chhablani, J. K., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11121186 · Cited by 9 · see citing works
Low-cost language models: Survey and performance evaluation on Python code generation
Espejel, J. L., Alassan, M. S. Y., Bouhandi, M., Dahhane, W., & Ettifouri, E. H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109490 · Cited by 10 · see citing works
Du, R. F., Carbonell, E. L., Huang, J., Liu, S., Wang, X., Shen, D. G., & Ke, J. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3551913 · Cited by 9 · see citing works
Design Patterns for Large Language Model Based Neuro-Symbolic Systems
de Boer, M., Smit, Q. T. S., van Bekkum, M. A., Meyer-Vitali, A., & Schmid, T. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251377499 · Cited by 9 · see citing works
Li, S., Han, C., Yu, P., Edwards, C., Li, M., Wang, X., Fung, Y. R., Yu, C., Tetreault, J., Hovy, E. H., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.799 · Cited by 8 · see citing works
Measuring the Perceived IQ of Multimodal Large Language Models Using Standardized IQ TestsPreprint
Wasilewski, E., & Jablonski, M. (2024) · doi:10.36227/techrxiv.171560572.29045385/v1 · Cited by 8 · see citing works
AIBench: Towards trustworthy evaluation under the 45°law
Zhang, Z., Wang, J., Guo, Y., Wen, F., Chen, Z., Wang, H., Li, W., Sun, L., Zhou, Y., Zhang, J., et al. (2025) · Displays · doi:10.1016/j.displa.2025.103255 · Cited by 7 · see citing works
Self-citationSESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews
Huotala, A., Kuutila, M., & Mäntylä, M. V. (2025) · International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00053 · Cited by 13 · see citing works
Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
Barrak, A. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00064 · Cited by 15 · see citing works
An astronomical question answering dataset for evaluating large language models
Li, J., Zhao, F., Chen, P.-F., Xie, J., Zhang, X., Li, H., Chen, M., Wang, Y.-H., & Zhu, M.-Y. (2025) · Scientific Data · doi:10.1038/s41597-025-04613-9 · Cited by 17 · see citing works
Self-citationArch-Eval benchmark for assessing chinese architectural domain knowledge in large language models
Wu, J., Jiang, M., Fan, J., Li, S., Xu, H., & Zhao, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-98236-0 · Cited by 9 · see citing works
Li, Y., Branco, P., Hoole, A. M., Marwah, M., Koduvely, H. M., Jourdan, G.-V., & Jou, S. (2025) · IEEE Symposium on Security and Privacy · doi:10.1109/sp61157.2025.00191 · Cited by 27 · see citing works
Mathematical Language Models: A Survey
Liu, W., Hu, H., Zhou, J., Ding, Y., Li, J., Zeng, J., He, M., Chen, Q., Jiang, B., Zhou, A., et al. (2025) · ACM Computing Surveys · doi:10.1145/3773985 · Cited by 22 · see citing works
Mitigating the Bias of Large Language Model Evaluation
Zhou, H., Huang, H., Long, Y., Xu, B., Zhu, C., Cao, H., Yang, M., & Zhao, T. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_27 · Cited by 9 · see citing works
Martínez, J., Llinás, B., Botello, J. G., Padilla, J. J., & Frydenlund, E. F. (2024) · Online World Conference on Soft Computing in Industrial Applications · doi:10.1109/wsc63780.2024.10838967 · Cited by 7 · see citing works
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
Hu, R., Cheng, Y., Meng, L., Xia, J., Zong, Y., Shi, X., & Lin, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715265 · Cited by 7 · see citing works
Liu, T., Chen, Z., Fang, Z., Luo, W., Tian, M., & Liu, Z. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0053-z · Cited by 6 · see citing works
D’Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., & Mehri, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00748 · Cited by 12 · see citing works
DocTER: Evaluating document-based knowledge editing
Wu, S., Wang, A., Peng, M., Lin, Y., Li, W., Sun, M., & Su, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104299 · Cited by 17 · see citing works
Game Theory Combined Empowerment-Cloud Evaluation System
Dai, H. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.224 · Cited by 4 · see citing works
User-defined trade-offs in LLM benchmarking: balancing accuracy, scale, and sustainability
Gjorgjevikj, A., Nikolikj, A., Seljak, B. K., & Eftimov, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114405 · Cited by 5 · see citing works
Benchmarking Knowledge and Capability of Large Language Models in Building Science Domain
Jiang, G., Chen, K., Liu, J., Chen, J.-L., Zhe, S.-D., & Wang, Z. (2025) · Energy Use · doi:10.59717/ipj.energy-use.2025.100026 · Cited by 4 · see citing works
Wang, Y., Chen, H., Wei, X.-F., Chang, C., & Zuo, X. (2025) · Comput. Hum. Behav. Artif. Humans · doi:10.1016/j.chbah.2025.100198 · Cited by 6 · see citing works
Self-citationAssis, G., Freitas, C., & Paes, A. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5814 · Cited by 5 · see citing works
Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
Lyu, H., Luo, J., Kang, J. L., & Koenecke, A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732182 · Cited by 4 · see citing works
Goorts, L., Hollevoet, R., Xia, V., Cammaerts, F., & Güngör, A. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100475 · Cited by 6 · see citing works
Towards an Exhaustive Evaluation of Vision-Language Foundation Models
Salin, E., Ayache, S., & Favre, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00041 · Cited by 4 · see citing works
- Authors
- Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N.
- Venue
- Findings of the Association for Computational Linguistics: NAACL 2024
- Published
- 2024
- Pages
- 2299-2314
- OpenAlex
- W4401042689
- Cited by
- 211
- References
- 0
- Citations by year
- 2026: 28 · 2025: 97 · 2024: 68 · 2023: 18
- FWCI
- 56.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- standardized testing, college entrance examination, LSAT, GPT-4, artificial general intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
