The DOI matches its Crossref record; 8 of the 13 references it cites were found in a registry or an index.
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Open accessAbstract
Assessing foundation models' abilities for human-level tasks is crucial for Artificial General Intelligence (AGI) development.Traditional benchmarks, which rely on artificial datasets, may not accurately represent these capabilities.In this paper, we introduce AGIEval, a novel bilingual benchmark designed to assess foundation models in the context of humancentric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests.We evaluate several state-of-the-art foundation models on our benchmark.Impressively, we show that GPT-4 exceeds the average human performance in SAT, LSAT, and math contests, with 95% accuracy on SAT Math and 92.5% on the Chinese college entrance English exam.This demonstrates the exceptional performance of contemporary foundation models.In contrast, we also find that GPT-4 is less proficient in tasks requiring complex reasoning or specific domain knowledge.Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal their strengths and limitations, providing valuable insights into future directions for enhancing general capabilities.By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a meaningful and robust evaluation of foundation models' performance in real-world scenarios 1 .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks [3], [13], [30].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [13]
In the citing paper
“Unlike traditional benchmarks with clear numerical metrics, LLM benchmarking often involves subjective judgments and complex evaluation criteria [13], [25].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 10
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872.
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4):0-6.
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167.
- Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. Logiqa: a challenge dataset for machine reading comprehen- sion with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on Interna- tional Joint Conferences on Artificial Intelligence, pages 3622-3628.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
- Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of com- plex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2201-2216.
- Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Jec- qa: A legal-domain question answering dataset. In Proceedings of AAAI.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Self-citationMMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429
Self-citationBenchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., & Huang, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01239 · Cited by 431
Parameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186
Retrieval-Augmented Generation (RAG) Chatbots for Education: A Survey of Applications
Swacha, J., & Gracel, M. (2025) · Applied Sciences · doi:10.3390/app15084234 · Cited by 198
Buehler, M. J. (2023) · Applied Mechanics Reviews · doi:10.1115/1.4063843 · Cited by 140
Scientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 216
Self-citationDatasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 268
Causal Inference Meets Deep Learning: A Comprehensive Survey
Jiao, L., Wang, Y., Liu, X., Li, L., Liu, F., Ma, W., Guo, Y., Chen, P., Yang, S., & Hou, B. (2024) · Research · doi:10.34133/research.0467 · Cited by 141
Self-citationA survey of multilingual large language models
Qin, L., Chen, Q., Zhou, Y., Zhi, C., Li, Y., Liao, L., Li, M., Che, W., & Yu, P. S. (2025) · Patterns · doi:10.1016/j.patter.2024.101118 · Cited by 282
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
Large language model assisted fine-grained knowledge graph construction for robotic fault diagnosis
Liao, X., Chen, C., Wang, Z., Liu, Y., Wang, T., & Cheng, L. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103134 · Cited by 88
Xiao, T., & Xu, P. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114691 · Cited by 86
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Liu, Y., Li, D., Wang, K., Xiong, Z., Shi, F., Wang, J., Li, B., & Hang, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103809 · Cited by 91
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 91
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
Exploring the Impact of Temperature on Large Language Models: Hot or Cold?
Li, L., Sleem, L., Gentile, N., Nichil, G., & State, R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.135 · Cited by 62
The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S. (2026) · ACM Computing Surveys · doi:10.1145/3773080 · Cited by 58
Ray, P. P. (2023) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2023.100136 · Cited by 57
Assessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57
Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 56
PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation
Hu, J., Dong, T., Luo, G., Ma, H., Zou, P., Sun, X. N., Guo, D., Yang, X., & Wang, M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3497725 · Cited by 58
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51
Self-citationScaling Large Language Models for Next-Generation Single-Cell AnalysisPreprint
Rizvi, S. A., Levine, D., Patel, A. M., Zhang, S., Wang, E., Perry, C. J., Vrkic, I., Constante, N. M., Fu, Z., He, S., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.04.14.648850 · Cited by 54
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 41
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
Sun, L., Yang, H., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., & Yu, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29872 · Cited by 100
Lion: Adversarial Distillation of Proprietary Large Language Models
Jiang, Y., Chan, C., Chen, M., & Wang, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.189 · Cited by 93
INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Chia, Y. K., Hong, P., Bing, L., & Poria, S. (2023) · arXiv · doi:10.48550/arxiv.2306.04757 · Cited by 36
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
Survey on Factuality in Large Language Models
Wang, C., Liu, X., Yue, Y., Guo, Q., Hu, X., Tang, X., Zhang, T., Cheng, J., Yao, Y., Hu, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3742420 · Cited by 359
Self-citationJia, J., Wang, T., Zhang, Y., & Wang, G. (2024) · Asia Pacific Journal of Education · doi:10.1080/02188791.2023.2286920 · Cited by 47
Gao, L., Lu, J., Shao, Z., Lin, Z., Yue, S., Ieong, C., Sun, Y., Zauner, R. J., Wei, Z., & Chen, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456145 · Cited by 44
Evaluating large language models on geospatial tasks: a multiple geospatial task benchmarking study
Xu, L.-C., Zhao, S., Lin, Q.-M., Chen, L., Luo, Q.-Q., Wu, S.-S., Ye, X., Feng, H.-L., & Du, Z.-H. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2480268 · Cited by 40
Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A SurveyPreprint
Li, Z., Wu, X., Du, H., Nghiem, H. T., & Shi, G. (2025) · Qeios · doi:10.32388/gxr68q · Cited by 33
Berti, A., Kourani, H., Häfke, H., Li, C.-Y., & Schuster, D. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-61007-3_2 · Cited by 38
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y., Jiang, S., Xiong, Z., Li, Z., Wu, W., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29767 · Cited by 28
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43
Self-citationCan Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Liang, J., & Xiao, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29777 · Cited by 45
Evaluating Large Language Models for Material Selection
Grandi, D., Jain, Y. P., Groom, A., Cramer, B., & McComb, C. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4066730 · Cited by 28
Post Hoc Explanations of Language Models Can Improve Language ModelsPreprint
Krishna, S., Ma, J., Slack, D., Ghandeharioun, A., Singh, S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3006112/v1 · Cited by 24
Jiao, J., Afroogh, S., Murali, A., Chen, K., Atkinson, D., & Dhurandhar, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-18489-7 · Cited by 40
Feedback on Feedback: Student’s Perceptions for Feedback from Teachers and Few-Shot LLMs
Rüdian, S., Podelo, J., Kužílek, J., & Niels Pinkwart, C. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706479 · Cited by 33
SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models
Qin, C., Chen, X., Wang, C., Wu, P., Chen, X., Cheng, Y., Zhao, J., Xiao, M., Dong, X., Long, Q., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737403 · Cited by 31
A Comprehensive Review on the Application of Large Language Models (LLMs) in Power Systems
Ghafari, M., Sami, A., Rezapour, H., Ghayour, S. S., Lin, F., Başaran, K., Lăzăroiu, G. C., & Siano, P. (2025) · IEEE Access · doi:10.1109/access.2025.3637226 · Cited by 20
So you want your private LLM at home? A survey and benchmark of methods for efficient GPTs
Tuggener, L., Sager, P., Taoudi-Benchekroun, Y., Grewe, B. F., & Stadelmann, T. (2024) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds60720.2024.00036 · Cited by 23
Goto, T., Ono, K., & Morita, A. (2024) · doi:10.36227/techrxiv.171173447.70655950/v1 · Cited by 19
Large Language Model Evaluation Criteria Framework in Healthcare: Fuzzy MCDM Approach
Alabool, H. M. (2025) · SN Computer Science · doi:10.1007/s42979-024-03533-6 · Cited by 22
Artificial Intelligence in the 21st Century
奥宮, 正. (2023) · Research on Intelligent Manufacturing and Assembly · doi:10.25082/rima.2023.01.002 · Cited by 21
The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models
Renze, M., & Guven, E. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852493 · Cited by 69
Mylrea, M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29246-0.00007-9 · Cited by 19
LLMEval: A Preliminary Study on How to Evaluate Large Language Models
Zhang, Y., Zhang, M., Yuan, H., Liu, S., Shi, Y., Gui, T., Zhang, Q., & Huang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29934 · Cited by 31
Using GPT-4 to guide causal machine learning
Constantinou, A. C., Kitson, N. K., & Zanga, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126120 · Cited by 14
Luo, G., Yang, X., Dou, W., Wang, Z., Liu, J., Dai, J., Qiao, Y., & Zhu, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02324 · Cited by 23
Generative AI-Based Text Generation Methods Using Pre-Trained GPT 2 ModelPreprint
Pandey, R., Waghela, H., Rakshit, S., Rangari, A., Singh, A., Kumar, R., Ghoshal, R., & Sen, J. (2024) · doi:10.36227/techrxiv.171216659.95569463/v1 · Cited by 15
Generative Inference of Large Language Models in Edge Computing: An Energy Efficient Approach
Yuan, X., Li, H., Ota, K., & Dong, M. (2024) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc61514.2024.10592339 · Cited by 14
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12
MindLLM: Lightweight large language model pre-training, evaluation and domain application
Yang, Y., Sun, H., Li, J., Liu, R., Li, Y., Liu, Y., Gao, Y., & Huang, H. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.001 · Cited by 13
Investigating the Efficacy of ChatGPT-3.5 for Tutoring in Chinese Elementary Education Settings
Bai, Y., Li, J., Shen, J., & Zhao, L. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3464560 · Cited by 14
A Framework for Domain-Specific Dataset Creation and Adaptation of Large Language Models
Balaskas, G., Papadopoulos, H., Pappa, D., Loisel, Q. E. A., & CHASTIN, S. F. M. (2025) · Computers · doi:10.3390/computers14050172 · Cited by 17
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15124 · Cited by 12
Self-citationThe ethical evaluation of large language models and its optimization
Lyu, Y., & Du, Y. (2025) · AI and Ethics · doi:10.1007/s43681-024-00654-9 · Cited by 11
ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination
Li, D., Yu, J., Hu, B., Xu, Z., & Zhang, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.129 · Cited by 13
Gong, L., Chen, J., & Wu, F. Q. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3564177 · Cited by 11
Retrieval augmented generation for building datasets from scientific literature
Maharana, P. R., Verma, A., & Joshi, K. (2025) · Journal of Physics Materials · doi:10.1088/2515-7639/ade1fa · Cited by 12
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Zhao, Y., Zhang, H., Xie, L., Hu, T., Gan, G., Long, Y., Hu, Z., Chen, W., Li, C., Xu, Z., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00793 · Cited by 80
Self-citationArtificial intelligence techniques for human-machine interaction
Mukhtar, H. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-29150-0.00010-x · Cited by 9
The Antecedents of Transformer Models
Dennis, S. J., Shabahang, K. D., & Yim, H. (2024) · Current Directions in Psychological Science · doi:10.1177/09637214241279504 · Cited by 11
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 10
A Survey on Human Preference Learning for Aligning Large Language Models
Jiang, R., Chen, K., Bai, X., He, Z., Li, J., Yang, M., Zhao, T., Nie, L., & Zhang, M. (2025) · ACM Computing Surveys · doi:10.1145/3773279 · Cited by 67
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
Performance evaluations of large language models for customer service
Li, F., Wang, Y., Xu, Y., Wang, S., Liang, J., Chen, Z., Liu, W., Feng, Q., Duan, T., Huang, Y., et al. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02432-9 · Cited by 8
Self-citationDetection of Disease Features on Retinal OCT Scans Using RETFound
Du, K., Nair, A. R., Shah, S. V., Gadari, A., Vupparaboina, S. C., Bollepalli, S. C., Sutharahan, S., Sahel, J., Jana, S., Chhablani, J. K., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11121186 · Cited by 10
Low-cost language models: Survey and performance evaluation on Python code generation
Espejel, J. L., Alassan, M. S. Y., Bouhandi, M., Dahhane, W., & Ettifouri, E. H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109490 · Cited by 15
Du, R. F., Carbonell, E. L., Huang, J., Liu, S., Wang, X., Shen, D. G., & Ke, J. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3551913 · Cited by 10
Design Patterns for Large Language Model Based Neuro-Symbolic Systems
de Boer, M., Smit, Q. T. S., van Bekkum, M. A., Meyer-Vitali, A., & Schmid, T. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251377499 · Cited by 13
Li, S., Han, C., Yu, P., Edwards, C., Li, M., Wang, X., Fung, Y. R., Yu, C., Tetreault, J., Hovy, E. H., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.799 · Cited by 11
Measuring the Perceived IQ of Multimodal Large Language Models Using Standardized IQ TestsPreprint
Wasilewski, E., & Jablonski, M. (2024) · doi:10.36227/techrxiv.171560572.29045385/v1 · Cited by 9
AIBench: Towards trustworthy evaluation under the 45°law
Zhang, Z., Wang, J., Guo, Y., Wen, F., Chen, Z., Wang, H., Li, W., Sun, L., Zhou, Y., Zhang, J., et al. (2025) · Displays · doi:10.1016/j.displa.2025.103255 · Cited by 18
Self-citationSESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews
Huotala, A., Kuutila, M., & Mäntylä, M. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00053 · Cited by 12
Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
Barrak, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00064 · Cited by 22
An astronomical question answering dataset for evaluating large language models
Li, J., Zhao, F., Chen, P., Xie, J., Zhang, X., Li, H., Chen, M., Wang, Y., & Zhu, M. (2025) · Scientific Data · doi:10.1038/s41597-025-04613-9 · Cited by 10
Self-citationArch-Eval benchmark for assessing chinese architectural domain knowledge in large language models
Wu, J., Jiang, M., Fan, J., Li, S., Xu, H., & Zhao, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-98236-0 · Cited by 9
Li, Y., Branco, P., Hoole, A. M., Marwah, M., Koduvely, H. M., Jourdan, G.-V., & Jou, S. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00191 · Cited by 12
Mathematical Language Models: A Survey
Liu, W., Hu, H., Zhou, J., Ding, Y., Li, J., Zeng, J., He, M., Chen, Q., Jiang, B., Zhou, A., et al. (2025) · ACM Computing Surveys · doi:10.1145/3773985 · Cited by 47
Mitigating the Bias of Large Language Model Evaluation
Zhou, H., Huang, H., Long, Y., Xu, B., Zhu, C., Cao, H., Yang, M., & Zhao, T. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_27 · Cited by 16
Martínez, J., Llinás, B., Botello, J. G., Padilla, J. J., & Frydenlund, E. F. (2024) · Winter Simulation Conference · doi:10.1109/wsc63780.2024.10838967 · Cited by 9
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
Hu, R., Cheng, Y., Meng, L., Xia, J., Zong, Y., Shi, X., & Lin, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715265 · Cited by 14
Liu, T., Chen, Z., Fang, Z., Luo, W., Tian, M., & Liu, Z. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0053-z · Cited by 6
D’Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., & Mehri, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00748 · Cited by 31
DocTER: Evaluating document-based knowledge editing
Wu, S., Wang, A., Peng, M., Lin, Y., Li, W., Sun, M., & Su, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104299 · Cited by 34
Game Theory Combined Empowerment-Cloud Evaluation System
Dai, H. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.224 · Cited by 4
User-defined trade-offs in LLM benchmarking: balancing accuracy, scale, and sustainability
Gjorgjevikj, A., Nikolikj, A., Seljak, B. K., & Eftimov, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114405 · Cited by 5
Benchmarking Knowledge and Capability of Large Language Models in Building Science Domain
Jiang, G., Chen, K., Liu, J., Chen, J.-L., Zhe, S.-D., & Wang, Z. (2025) · Energy Use · doi:10.59717/ipj.energy-use.2025.100026 · Cited by 4
Wang, Y., Chen, H., Wei, X.-F., Chang, C., & Zuo, X. (2025) · Comput. Hum. Behav. Artif. Humans · doi:10.1016/j.chbah.2025.100198 · Cited by 6
Self-citationAssis, G., Freitas, C., & Paes, A. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5814 · Cited by 5
Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
Lyu, H., Luo, J., Kang, J. L., & Koenecke, A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732182 · Cited by 9
Goorts, L., Hollevoet, R., Xia, V., Cammaerts, F., & Güngör, A. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100475 · Cited by 7
Towards an Exhaustive Evaluation of Vision-Language Foundation Models
Salin, E., Ayache, S., & Favre, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00041 · Cited by 4
Metadata
- Authors
- Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N.
- Venue
- Findings of the Association for Computational Linguistics: NAACL 2024
- Published
- 2024
- Pages
- 2299-2314
- OpenAlex
- W4401042689
- Cited by
- 211
- References
- 0
- Citations by year
- 2026: 28 · 2025: 97 · 2024: 68
- FWCI
- 56.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- standardized testing, college entrance examination, LSAT, GPT-4, artificial general intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
