The DOI matches its Crossref record; 8 of the 13 references it cites were found in a registry or an index.
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Open accessAbstract
Assessing foundation models' abilities for human-level tasks is crucial for Artificial General Intelligence (AGI) development.Traditional benchmarks, which rely on artificial datasets, may not accurately represent these capabilities.In this paper, we introduce AGIEval, a novel bilingual benchmark designed to assess foundation models in the context of humancentric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests.We evaluate several state-of-the-art foundation models on our benchmark.Impressively, we show that GPT-4 exceeds the average human performance in SAT, LSAT, and math contests, with 95% accuracy on SAT Math and 92.5% on the Chinese college entrance English exam.This demonstrates the exceptional performance of contemporary foundation models.In contrast, we also find that GPT-4 is less proficient in tasks requiring complex reasoning or specific domain knowledge.Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal their strengths and limitations, providing valuable insights into future directions for enhancing general capabilities.By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a meaningful and robust evaluation of foundation models' performance in real-world scenarios 1 .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks [3], [13], [30].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [13]
In the citing paper
“Unlike traditional benchmarks with clear numerical metrics, LLM benchmarking often involves subjective judgments and complex evaluation criteria [13], [25].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 10
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872.
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4):0-6.
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167.
- Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. Logiqa: a challenge dataset for machine reading comprehen- sion with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on Interna- tional Joint Conferences on Artificial Intelligence, pages 3622-3628.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
- Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of com- plex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2201-2216.
- Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Jec- qa: A legal-domain question answering dataset. In Proceedings of AAAI.
Cited by
Sorted by citations
Items 201–220 of 220
DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs
Meng, Z., (8098), Y. Z., Feng, Z., & Liu, Z. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-16823-8_8 · Cited by 5
Study of the multimodal understanding of vision-language transformer models
Salin, E. (2023) · theses.fr (ABES) · doi:10.70675/2beca704z5d3bz421dzbf61z61e00610aaeb
Li, X., Shang, Y., Jiang, X., Zhang, C., & Liao, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_8
Tang, W. L., Zhou, Y., Cheng, K., 许尔强, Xiao, L., & Li, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460704 · Cited by 21
Physical plausibility reasoning via HCM-GRPO: Empowering compact model for superior performance
Hu, Z., Sun, Z., Wei, Y., & Yu, L. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114108
Dynamic Multi-view RAG: Mitigating Hallucinations of Large Language Models in Education
Zhao, W., & Dong, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-9694-2_5
Qin, S., Xue, J., Hao, F., Niu, S., & Chang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1947-6_10
PPAI: Enabling Personalized LLM Agent Interoperability for Collaborative Edge Intelligence
Wang, Z., Liu, Q., Guo, K., Wang, H., Lin, J., Hong, Z., & Guo, S. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571683 · Cited by 1
Pan, J., Pan, X., & Pan, H. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1818923
Standardized assessment of LLM English proficiency
Min, S., Wang, S., Gao, X., Wang, H., Jin, Z., Ling, C., & Ding, N. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100656
Triplet Decomposition and Extensions: A General Framework for Parameter-Efficient Fine-Tuning
Du, Z., Min, Y., Zhang, D., Lü, K., Li, J., & Shen, H. T. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3720458
Hou, Y., Wang, J., Xiang, K., Long, Q., Zhou, Y., & Zhen, M. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817576
Varol, A., Shaikh, A., Jalo, H., Muchenje, G., de Cerqueira, J. S., Kemell, K., Motlagh, N. H., Leino, M., Pirkkalainen, H., Abrahamsson, P., et al. (2026) · IEEE Access · doi:10.1109/access.2026.3727159
Qiu, J., Qin, Z., Ma, K., Hu, P., Zhang, Y., Xie, X., Zhao, W., & Wei, X. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.4079
Toward a Structured Evaluation of Intelligent Agents with a Five-Dimensional Framework
Zhang, H., Zhang, R., Liu, G., Li, Y., Zhou, K., & Lu, W. (2026) · Asia Conference on Machine Learning and Computing (ACMLC) · doi:10.1109/acmlc70381.2026.11700477
PolyReal: A Benchmark for Real-World Polymer Science Workflows
Liu, W.-H., Wang, W., Xie, J.-Q., Yang, S., Wang, J., Chen, B.-T., Mei, G.-J., Yang, Z., Zhang, S.-F., Mo, Y., et al. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00192 · Cited by 3
Lin, Z., Zhang, Z., Sun, S., Zhao, J., & Yang, H. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00205
A Comparative Analysis of LLM Performance on Standardized Examinations: Towards a Hybrid Model
Wang, Z.-P., Huang, L.-Y., Wang, A.-K., & Fan, Y.-Q. (2026) · 2026 IEEE 8th International Conference on Artificial Intelligence, Computer Science, and Information Processing (AICSIP) · doi:10.1109/aicsip69702.2026.11744030
Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate
Hao, X., Wu, Z., Qiu, Y.-X., Xiao, C., Xu, R., Zheng, S., & Qin, J. (2026) · arXiv · doi:10.48550/arxiv.2606.00820
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Tian, C., Wang, J., Zhao, Q., Chen, K., Liu, J., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2025) · arXiv · doi:10.48550/arxiv.2507.17634
Metadata
- Authors
- Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N.
- Venue
- Findings of the Association for Computational Linguistics: NAACL 2024
- Published
- 2024
- Pages
- 2299-2314
- OpenAlex
- W4401042689
- Cited by
- 211
- References
- 0
- Citations by year
- 2026: 28 · 2025: 97 · 2024: 68
- FWCI
- 56.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- standardized testing, college entrance examination, LSAT, GPT-4, artificial general intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
