The DOI matches its DataCite record; 33 of the 41 references it cites were found in a registry or an index.
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Open accessAbstract
Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While LLMs have been shown to excel at these tasks (Wang et al., 2022;Zhong et al., 2023)”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Zhong et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“AGIEval [262], serves as a dedicated evaluation framework for assessing the performance of foundation models in the domain of human-centric standardized exams.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [262]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The emergence of general language models (LMs) like BERT (Devlin et al., 2019) has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these LMs. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are popular benchmarks that evaluate…”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 2×
“Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 3×
“GLUE series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models’ generalization capabilities.”
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 2×
“Emphasis on human-level cognitive tasks: Our human-centric benchmark is designed to mimic human cognition and problem-solving, aiming for a comprehensive evaluation of foundation models.”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“(2022) conducted a holistic evaluation of dozens of language models (up to InstructGPT (Ouyang et al., 2022)) on several previous NLP tasks (e.”
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1×
“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602 · cited 1×
“, Openbook QA (Kwiatkowski et al., 2019; Mihaylov et al., 2018) and text classification (Clark et al.”
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Research works printed but not linked: 2
- Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty- Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3622-3628, 2021.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Cited by
Sorted by citations
Items 801–900 of 931
BeNice! Closing Looping's Inference-Memory Gap via KV-SharingPreprint
Prairie, H., Ryan, J. M., Berg-Kirkpatrick, T., & Fu, D. Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31633
Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs
Abbood, A., Martínez-Plumed, F., Hernández-Orallo, J., & Collier, N. (2026) · doi:10.48550/arxiv.2610.29507
“To test whether a stronger QL signal improves temporal arithmetic and semantics, we select non-temporal examples from ChemLLMBench (Guo et al., 2023), MMLU-Pro (Wang et al., 2024), AGIEval (Zhong et al., 2024), TruthQuest (Mon-dorf and Plank, 2024), OmniMath (Gao et al., 2025), and LiveBench (White et al., 2025).”
Li, Y., Bian, J., Wang, J., Ding, P., & Gu, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.26185
“Similarly, C-Eval [Huang et al. 2023] and AGIEval [Zhong et al. 2023] assess multilingual and exam-based reasoning abilities using standardized test questions derived from real-world academic and professional exams.”
Li, M., & Zhou, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19741
“We further evaluate on CEVAL-VALID-DIRECT and an AGIEVAL-CN direct subset under the same generation budgets (Huang et al., 2023b; Zhong et al., 2024); CMMLU provides a related large-scale reference point for Chinese multitask evaluation (Li et al., 2024).”
More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training
Wu, H., Wu, A.-Q., Hu, J.-N., & Yu, B. (2026) · doi:10.48550/arxiv.2610.19443
TPPO: Tracking-Value PPO For Language Model ReasoningPreprint
Wang, F., Li, C., Yan, J., Luo, Y., Wang, Z., Li, Y., Zhan, R., Cheng, Y., & Lin, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.16794
“…we evaluate on ten benchmarks: in-domain tasks (AIME24/25/26, AMC (Li et al., 2024), MATH-500 (Hendrycks et al., 2021b), Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024)) and OOD tasks (ARC-C (Clark et al., 2018), AGI-Eval (Zhong et al., 2024), and BBH (Suzgun et al., 2023)).”
Query-Adaptive Neuron Steering for LLMs
Pan, Y.-X., Wu, J.-W., Ma, Y., Chen, X., Zeng, X.-Y., Wang, X.-T., Zhang, Y.-Y., Zhang, Y., Tian, Y., Pan, J.-F., et al. (2026) · doi:10.48550/arxiv.2610.14519
“To assess cross-benchmark generalization, we directly evaluate the model trained on MMLU-Pro on GPQA-Main, GPQA-Diamond [26] , SuperGPQA, LiveBench [27] , AGIEval [28] , GSM8K [29] , and ARC-Challenge [30] , without using any training data from these target benchmarks.”
SAT-RL: Training Language Models for Logical Reasoning with Symbolic SolversPreprint
Sun, J., & Meel, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14402
“…models (LLMs) have demonstrated strong capabilities on mathematical and logical reasoning tasks, including the Law School Admission Test (LSAT), the International Mathematical Olympiad (IMO), and broad AGI datasets such as AGI-Eval (Zhong et al., 2021; Ren et al., 2025; Zhong et al., 2024).”
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-ExpertsPreprint
Chai, Y., Zhu, T., Qu, X., Hu, X., Chen, G., Guo, Q., & Cheng, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.11575
“For SFT, we report Avg(9) over a nine-task suite covering Knowledge : MMLU (Hendrycks et al., 2021a), AGIEval (En) (Zhong et al., 2023), GPQA (Main) (Rein et al., 2023), Reasoning : BBH (Suzgun et al., 2022), Mathematics : GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), Coding : MBPP…”
One Model, Multiple Compute Budgets: Multi-Ratio ConceptMoE for Flexible Compute Across TurnsPreprint
Fu, Y., Huang, Z., Han, D., Gong, S., Dong, Z., Min, Q., & Zhou, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14074
Self-Evolve With a Reference:Anchored Training of Tool-Integrated Agents
Liao, W.-J., Zhao, L., & Cao, Z.-H. (2026) · doi:10.48550/arxiv.2610.09856
More Than Words: Compositional Tokenization for Efficient Language ModelsPreprint
Reif, Y., Kaplan, G., & Schwartz, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.05597
TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-TrainingPreprint
Peng, M., Zhang, Q., Chen, N., Li, Y., Yan, G., Gu, X., Wu, H., Wang, H., Huang, L., Zhang, W., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.04973 · Cited by 1
“AGIEval-CN (Zhong et al., 2024) and reasoning on Chinese examination questions.”
Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?Preprint
Lyu, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.20639
“Out of domain, BBH (Suzgun et al., 2022) supplies free-form tasks; in domain, MATH-500 (Hendrycks et al., 2021b; Lightman et al., 2023) and GSM8K (Cobbe et al., 2021) supply free-form mathematics, and AQuA-RAT (Ling et al., 2017) and SAT-Math (Zhong et al., 2024) multiple choice.”
Steering Tree-of-Thought Reasoning via Deductive Verification
Haoliang, C., Tang, E., Zhang, S., Mao, J., Fu, Y., Ma, J., Cui, K., Liu, Y., Tian, Y., Gao, X., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832200
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Dutulescu, A., Ruseti, S., Masala, M., Rebedea, T., & Dascalu, M. (2026) · doi:10.48550/arxiv.2610.01511
“…& Math-AI, 2024) and 2025 (Zhang & Math-AI, 2025), MATH (Hendrycks et al., 2021b), and Omega 500 (Sun et al., 2026); Reasoning: AGI Eval English (Zhong et al., 2024) and BigBenchHard-BBH (Suzgun et al., 2023) with chain-of-thought prompting; Code: HumanEval+ (Chen et al., 2021; Liu et al.,…”
CODER: Data-efficient activation of mathematical reasoning via difficulty-aware core-set.
Wang, L., Liu, G.-A., & Zhao, X.-Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109765
Xu, D., Hooi, B., & Qiao, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38964
“AQuA-RAT consists of algebraic word problems derived from GRE and GMAT exams; we use the version included in AGIEval (Zhong et al., 2024).”
Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories
Huang, Z., Tian, C.-X., Yang, Q.-Y., Chen, K.-L., Liu, Z.-Q., Zhang, Z.-Q., Huang, X.-L., & Zhou, J. (2026) · doi:10.48550/arxiv.2609.37169
“(Clark et al., 2018), AGIEval (Zhong et al., 2023), BBH and its Chinese configuration (Suzgun et al., 2023), WorldSense (Benchekroun et al., 2023), PIQA (Bisk et al., 2020), and HellaSwag (Zellers et al., 2019); (ii) language understanding , including RACE-Middle and RACE-High (Lai et al., 2017),…”
InfluentialOrthogonal Yet Coupled: Decoupling Geometric Components for Model Merging
Wang, Z.-J., Liu, Y.-K., Wang, M.-Y., Nie, E., Zhao, M.-J., Ma, Y.-P., Liu, K., Wang, Z.-H., Feng, S., Wang, D.-L., et al. (2026) · doi:10.48550/arxiv.2609.37564
“AGIEval (Zhong et al., 2024) is a bilingual, human-centric benchmark comprising questions from official admission and qualification examinations, as well as academic competitions, in both Chinese and English.”
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Tirosh, D., Amos, I., & Geva, M. (2026) · doi:10.48550/arxiv.2609.38149
MassAlloc Attention: Let Attention Allocate Its Own ComputePreprint
Shi, J., Peng, Z., Li, X., Qi, Y., Lin, X., Chen, H., Wang, L., Liu, G., & Luo, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.32712
CoWindow Attention: Full Causal Coverage Is a Collective PropertyPreprint
Shi, J., Peng, Z., Li, X., Qi, Y., Lin, X., Chen, H., Wang, L., Liu, G., & Luo, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.32704
“The standard benchmark suite covers knowledge, reasoning, and retrieval through MMLU [23, 36], MMLU-Pro [54], BBH [49], HellaSwag [62], OBQA [37], WinoGrande [44], PIQA [6], GSM8K [9], Hendrycks-Math [24], ARC-C [8], AGIEval [68], GPQA-Diamond [43], and RULER [26].”
He, J.-F., Liu, Y.-Z., Zhang, K., Han, W.-D., Key, & Rayying (2026) · doi:10.48550/arxiv.2609.32398
Scholz, N., Thulke, D., Nasir, A., Allred, W. S., Matusov, E., & Ney, H. (2026) · doi:10.48550/arxiv.2609.28395
“From its unseen split, we take AGIEval English (Zhong et al., 2024) and GPQA (Rein et al., 2024), both zero-shot with chain-of-thought prompting.”
Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA
Fang, W.-Z., Tzou, N., Valkov, L., & Chappidi, S. (2026) · doi:10.48550/arxiv.2609.25618
“After fine-tuning on the target tasks, we evaluate reasoning retention on both the anchor benchmark and several held-out benchmarks whose data are not used to construct the preservation constraint, including MATH-500, AGIEval-EN-MATH (Zhong et al., 2024), and MMLU-STEM (Hendrycks et al., 2020).”
Robust Failure, Conservative Repair: Textual Knowledge Distillation from Cross-Model Failures
Ren, A., Liu, H., & Tan, C.-H. (2026) · doi:10.48550/arxiv.2609.25400
“AGIEval is excluded from the main claim because the available CS-ICL anchor underperforms raw prompting on all three tasks (Table 12, top).”
InfluentialRole of foundation models in data-driven tissue diagnostics
Bilal, M., Aadam, Raza, M., Alsuhaibani, A., Altherwy, Y. N., Alabduljabbar, A., Almarshad, F. A., Golding, P., & Rajpoot, N. (2026) · Molecular Aspects of Medicine · doi:10.1016/j.mam.2026.101504
CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
Jiang, J.-X., He, L., & Fang, Z.-X. (2026) · doi:10.48550/arxiv.2609.18779
MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse LookupPreprint
Li, M., Sigal, L., & Liao, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.15126
Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
Tu, Y.-L., Li, Z.-H., Ji, S.-X., Tiedemann, J., & Yuan, F. (2026) · doi:10.48550/arxiv.2609.13770
ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic SearchPreprint
He, J., Liang, G., Liu, H., Guan, H., Sun, J., Guo, J., Feng, W., Xie, Y., Shen, Y., Shao, B., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.13356
“(Zhong et al., 2023), AQuA-RAT (Ling et al., 2017), HARDMath-mini (Fan et al., 2024), IMO-AnswerBench (Luong et al., 2025), MATH-P-Hard (Huang et al., 2025), and the text-only mathematics subset of OlympiadBench (He et al., 2024), together with formal theorem proving on miniF2F (Zheng et al., 2022)…”
SIRF: A Spec-Internalized Risk Foundation Model for Industrial Content Risk Control
Wu, S.-W., Lin, Y.-M., Yuan, P.-C., & Jiang, X.-L. (2026) · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/2026.emnlp-industry.197
“We evaluate before/after CPT at 8B / 14B / 32B on 10 public benchmarks: C-Eval (Huang et al., 2023), CMMLU (Li et al., 2024), AGIEval-CN (Zhong et al., 2024), MMLU (Hendrycks et al., 2020), MMLU-Pro (Wang et al., 2024), GSM8K (Cobbe et al., 2021), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021),…”
Qiu, J., Qin, Z., Ma, K., Hu, P., Zhang, Y., Xie, X., Zhao, W., & Wei, X. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.4079
Paquin, D., & Jain, R. (2026) · doi:10.48550/arxiv.2609.09372
“…item difficulty from text has precedent in educational measurement (Embretson and Wet-zel, 1987) and in NLP (Vania et al., 2021), while domain-targeted evaluations such as ARC (Clark et al., 2018) and AGIEval (Zhong et al., 2024) instead construct item pools isolating particular reasoning types.”
Wu, J.-Z., Zhang, J.-J., Yi, L., Zheng, L.-Q., Jing, Y.-H., Zhou, X.-Y., & Yang, Q. (2026) · doi:10.48550/arxiv.2609.06212
“Most educational benchmarks position LLMs as students : LINGOLY (Bean et al., 2024), ModeLing (Chi et al., 2024), PuzzLing (¸Sahin et al., 2020), AGIEval (Zhong et al., 2023), and MMLU (Hendrycks et al., 2021) test reasoning and knowledge mastery but do not evaluate whether models can generate effective teaching mater…”
Inferred Generative-Process Diversity Predicts Correlated Failure Across Language ModelsPreprint
Tieman, R., & Markou, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.03422
Liu, J., Ranjan, S., Mishra, P., Dukler, Y., Ramesh, G., Wu, J.-L., Sun, X.-M., Xie, W., Hou, C.-J., Appia, V. V., et al. (2026) · doi:10.48550/arxiv.2609.00791
“As shown in Table 14, MOPD re-covers most of the IF expert’s instruction-following gains while maintaining DPO-level performance on math, code, MMLU (Hendrycks et al., 2021a), and AGIEval (Zhong et al., 2024).”
Knowledge Distillation During Mid-Training Favors Reasoning over Factual RecallPreprint
He, J., Yen, H., Li, S. S., Li, M., Zeng, H., Xia, Y., Zhang, B., Zhao, Z., Zhang, Q., Koh, P. W., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.01532
Using profiles of cognitive capability to assess AI suitability for workplace tasks
Prunty, J. E., Tesic, M., Quinn, P., Hernández-Orallo, J., & Cheke, L. (2026) · arXiv.org · doi:10.48550/arxiv.2608.25623
“Abstract Narrative Understanding [75] 303 AGIEval [76] 1896 BigBenchHard [77] 1265 BigToM [78] 686 Cause and Effect [75] 51 CoQA [75] 210 EmoBench [79] 1200 Evaluating Information Essentially [75] 68 EWoK [80] 210 Fantasy Reasoning [75] 201 Fantom [81] 2177 INTUIT [43] 995 Known Unknowns [75] 46…”
SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models
Lin, Q.-W., Xu, B., Liu, X., Hao, Z.-F., & Cai, R.-C. (2026) · arXiv.org · doi:10.48550/arxiv.2608.22354
LLM Copilot for Privacy-Respecting Cyber Incident Response
Prakash, P. (2026) · International Journal of Intelligent Systems and Data Science · doi:10.67231/6s96yv20
“This observation aligns with recent findings on AI model evaluation methodologies [23].”
MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generation
Liu, Z., Li, G., Qiu, C., Kong, W.-Y., & Kalnis, P. (2026) · arXiv.org · doi:10.48550/arxiv.2608.20927
Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses
Strugatski, A., Zeinfeld, L., Cooper, J., Rap, S., Schwarts, G., & Alexandron, G. (2026) · arXiv.org · doi:10.48550/arxiv.2608.17810
“A common practice in LLM development and evaluation is to reuse instruments originally designed to measure human knowledge or reasoning across domains (e.g., (Kasagga et al., 2025; Balunovi´c et al., 2025; Zhong et al., 2024; Du et al., 2025)).”
Semantic Association vs. Position: Which Information Do Attention Heads Prefer?
Wang, M., Niu, S., Zhang, S., Wang, H., Song, S., Yin, Z., Yang, J., & Liang, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818196
SeqLLM: Augmenting LLMs with Behavioral-Sequence Modeling for High-Stakes Decisions at WeChat Pay
Li, G.-L., Zhang, J.-X., Tan, M. H. Y., Wang, B., & Huang, W.-R. (2026) · arXiv.org · doi:10.48550/arxiv.2608.03063
“Other task metrics are Accuracy, NDCG@3, and ROUGE; general language ability is measured by MMLU [6], C-Eval [8], and AGIEval [29].”
Triplet Decomposition and Extensions: A General Framework for Parameter-Efficient Fine-Tuning
Du, Z., Min, Y., Zhang, D., Lü, K., Li, J., & Shen, H. T. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3720458
“We evaluate the fine-tuned models on both in-domain datasets (GSM8K [46], MATH [47], AQuA [48], NumGLUE [49]) and out-of-domain mathematical reasoning benchmarks (SVAMP [50], Mathe-matics [51], SimulEq [52], SAT-Math [53], MMLU-Math [54]).”
WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management
Zhang, Y., Wang, H., Leong, Z. H., Wu, Z., Lin, K., Pan, Z., Huangfu, Q., Ren, W., Wu, W.-Y., Wang, F., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2608.07529
“More comprehensive frameworks, such as HELM [11] and AGIEval [20], emphasize broader and more standardized evaluation of foundation models.”
When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs
Mosolova, A., & Seddah, D. (2026) · arXiv.org · doi:10.48550/arxiv.2607.21445
“Other benchmarks use school or university exam questions, such as AGIEval (Zhong et al., 2024) or MERA (Fenogen-ova et al., 2024), or focus on commonsense understanding, e.g., HellaSwag (Zellers et al., 2019) and PIQA (Bisk et al., 2020).”
Pan, J., Pan, X., & Pan, H. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1818923
Morabito, R. D., McDonald, T., Viswanath, C., Hwang, A., Gaube, S., Kabbara, J., & Emami, A. (2026) · arXiv.org · doi:10.48550/arxiv.2607.05113
“Standard LLM evaluations measure true capability through automated, model-focused benchmarks (Kazemi et al., 2025; Zhong et al., 2024; Hendrycks et al., 2021; Liang et al., 2023), yet these metrics often fail to capture the contextual nuances of how users evaluate models in practice (McIn-tosh et…”
Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language ModelsPreprint
Guan, J., Xie, Y., Chen, J., Liu, J., Ye, Z. Z., Li, D., Shen, J., Tao, J., & Xue, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.02914
“…[58], Chinese SimpleQA(-lite) [20], HLE-LLMJudge(-lite) [42], MATH-500 [22, 34], GSM8K [11], HumanEval [9], HumanEval+ [36], MBPP [4], MBPP+ [36], IFEval [67], BBH(-lite) [53], AGIEval [66], ARC-Challenge [10], CommonsenseQA [54], C3 [52], RACE-Middle [29], RACE-High [29], and OpenBookQA [38].”
CausalMix: Data Mixture as Causal Inference for Language Model Training
Tang, Z.-N., Zhang, Y.-K., Zheng, S.-M., Pan, Z.-S., Pei, Q., Jin, D.-N., Cui, Q., Zhou, J., Wang, Y.-J., & Huang, B.-Q. (2026) · arXiv.org · doi:10.48550/arxiv.2607.01104
“Following Tulu 3, we assess model performance on multiple tasks and corresponding benchmarks, including Knowledge (MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024), GPQA (Rein et al., 2024)), Reasoning (BBH (Srivastava et al., 2023), DROP (Dua et al., 2019), AGIEval (Zhong et al., 2024)), Math (GSM8K (Cobb…”
InfluentialSelf-citationToward a Structured Evaluation of Intelligent Agents with a Five-Dimensional Framework
Zhang, H., Zhang, R., Liu, G., Li, Y., Zhou, K., & Lu, W. (2026) · Asia Conference on Machine Learning and Computing (ACMLC) · doi:10.1109/acmlc70381.2026.11700477
A Comparative Analysis of LLM Performance on Standardized Examinations: Towards a Hybrid Model
Wang, Z.-P., Huang, L.-Y., Wang, A.-K., & Fan, Y.-Q. (2026) · 2026 IEEE 8th International Conference on Artificial Intelligence, Computer Science, and Information Processing (AICSIP) · doi:10.1109/aicsip69702.2026.11744030
“● LiveBench [8]: This dataset tends to focus on automatic scoring of objective answers, but what appears to complicate traditional interpretations is that its scoring method for subjective sections in mixed tasks often seems to lack transparency.”
BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams
Santos, J. G. A., Bonás, G. K., Laitz, T., Almeida, T., & Pedrini, H. (2026) · arXiv.org · doi:10.48550/arxiv.2606.22723
“2 Related Work [4], which spans 57 tasks across Science, Technology, Engineering and Mathematics (STEM) and humanities; AGIEval [14], an aggregation of human-centric standardized tests; and GPQA [11], which targets graduate-level reasoning.”
A pretrained biomedical large language model for Persian biomedical text mining
Merzah, B. M., Taami, T., Asoudeh, S., Mirzaee, S., pour, A. R. H., Bengari, A. A., & Shahmarvandi, E. K. (2026) · Scientific Reports · doi:10.1038/s41598-026-55970-3 · Cited by 2
“Furthermore, AGIEval, evaluating reasoning by human-standardized exams, stresses that though LLMs do great in structured question-answering tasks, they underperform in fields which require deep, domain-specific knowledge and thus reinforce the need for specialized biomedical benchmarks 45 .”
Huang, R.-X., Shi, J., Huang, H.-T., Huang, Y.-F., Guan, Z., Zeng, H., Yen, I. E.-H., & Yu, M.-H. (2026) · arXiv.org · doi:10.48550/arxiv.2606.10722
“• Composite : LiveBench [29], SuperGPQA [30], AGIEval [31], MMLU-Pro [32], C-Eval [33], and MMLU [34].”
Data-Efficient Online Training for Direct Alignment in LLMs
Zhang, C., Wang, J., He, K., Chai, C., Zhang, Y., Wang, Y., Zhou, X., Zheng, L., Wu, L., He, C., et al. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3828612.3828628
“General Tasks: MMLU [24], MMLU-PRO [52], DROP [17], AGIEval [63], KorBench [35]; 𝑖𝑖 ) .”
Self-citationLin, Z., Zhang, Z., Sun, S., Zhao, J., & Yang, H. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00205
Routing Without Training: Controllable-Ratio LLM Offloading via Reliability GatingPreprint
Chen, E., Wang, S., Chan, K. S., Wang, S., & Brinton, C. G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.20481
MindLoom: Composing Thought Modes for Frontier-Level Reasoning Data Synthesis
Shen, H., Guo, T.-A., Chen, X.-Z., Liu, M., Bi, W.-C., Jing, W., Xie, S., Shi, Z., Han, Y., Pan, C., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.21630
“…[14] Olympiad mathematics and science 2,665 HLE [9] Expert-level reasoning 2,500 ARC-AGI-2 [4] Abstraction and reasoning 1,120 AGIEval [46] Standardized exam problems 1,000 TheoremQA [3] Theorem-based reasoning 731 SciBench [31] College-level science 692 MiniF2F [45] Formal…”
ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
Liu, Y., Wang, Z.-J., Zhao, M., Nie, E., Wang, M.-Y., Li, Q., Ren, F.-L., Feng, S., Wang, D.-L., & Schütze, H. (2026) · arXiv.org · doi:10.48550/arxiv.2605.21177
“SAT-Math [36] contains math questions derived from standardized SAT-style examinations.”
DEL: Digit Entropy Loss for Numerical Learning of Large Language ModelsPreprint
Zheng, Z., He, C., Wang, S., Li, Y., Cheng, M.-M., & Zhang, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.20369 · Cited by 1
“…difficulty math competition dataset MATH [34]; 3) arithmetic math word problem dataset SVAMP [58]; 4) simultaneous equations dataset SimulEq [59]; 5) algebraic word problems with rationales AQuA [60]; 6) general college entrance exams SAT-Math [61]; and 7) college math problems dataset MMLU [62].”
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series UnderstandingPreprint
Quinlan, P., Levasseur, J., Li, Q., & Zhu, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.20268 · Cited by 1
“…al., 2016]), cloze and completion (LAMBADA [Paperno et al., 2016], Winograd [Levesque et al., 2012]), knowledge (Jeopardy, BB WikiQA [Srivastava et al., 2023]), and algorithmic reasoning (BB CS Algorithms, BB Operators, BB Language ID [Srivastava et al., 2023]; AGI-Eval LSAT [Zhong et al., 2023]).”
MasFACT: Continual Multi-Agent Topology Learning via Geometry-Aware Posterior TransferPreprint
Wang, X., Wang, J., Zhang, F., Hu, Y., Zhang, D., Ye, Y., Ban, Y., Han, J., & Wang, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.17361
Memorize Theorems, Not Instances: Probing SFT Generalization through Mathematical ReasoningPreprint
Peng, R., Yang, M., Lei, J., Li, X., Wu, X., & Chen, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.09270
“…distinct reasoning domains: (i) General Mathematical Reasoning, using the MATH-500 test set and competition-level benchmarks including AGIEval-Math (Zhong et al. [2024]), Omni-Math (Gao et al. [2024]), and LiveBench-Math (White et al. [2024]); and (ii) Visual-Spatial Reasoning, using the GeoQA…”
Influential"I Don't Know" -- Towards Appropriate Trust with Certainty-Aware Retrieval Augmented GenerationPreprint
Scala, D. D., de Boer, M., & Yolum, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.00957
“LLMs have natural language processing capabilities at a close-to-human capability level [51], but as LLMs are purely statistical machines they also suffer from so-called hallucinations: outputs generated by a trained LLM, which are presented as plausible or authoritative information, but that are…”
Small Language Model Helps Resolve Semantic Ambiguity of LLM PromptPreprint
Huang, Z., Zhang, C., Puspitasari, F. D., Zhang, J., Zhou, Y., Chen, S., & Yang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.23263 · Cited by 1
“We evaluate on five widely used reasoning benchmarks: GPQA (Rein et al., 2024), AGIEval-MATH (Zhong et al., 2024), LIAR (Wang, 2017), WSC (Levesque et al., 2012), and BBH-Navigate (Suzgun et al., 2023).”
TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice
Hu, G., Chen, Y.-T., Ding, H.-Y., Gao, W., Huang, J.-J., Peng, M., Xie, Q.-Q., & Yue, K. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.08948
“For example, MATH (Hendrycks et al., 2021) emphasizes mathematical correctness over explanation quality, while GAOKAO (Zhang et al., 2023a) and AGIEval (Zhong et al., 2024) focus on exam-style reasoning but overlook mixed-format outputs in professional settings.”
Cabrera, L. M., Saxton-Knight, I., & D'Arcy, J. K. (2026) · Proceedings of the Workshop on Evaluating Evaluations (EvalEval) · doi:10.48550/arxiv.2604.07593
“In the context of mathematical reasoning, benchmarks such as GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MathArena (Balunovi´c et al., 2025), OlympiadBench (He et al., 2024), AGIEval (Zhong et al., 2023) and MathVista (Lu et al., 2023) have become standard tools for evaluating the capabilities of LLMs.”
The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace AlignmentPreprint
Balasubramanian, R., Lin, P.-J., Sharma, R., Fang, A., Abdi, F., Rozgić, V., Du, Z., Bansal, M., & Vu, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.06377
“For instance transferring a mathematical reasoning direction from Qwen3-4B to Qwen3-14B improves the model from 61.1% to 71.3% on AGIEval-Math, surpassing the 67.8% of the 14B instruction-tuned variant.”
InfluentialLee, N., Son, G., Ko, H., Kim, C., An, J., Han, K., & Kwak, I.-Y. (2026) · arXiv.org · doi:10.48550/arxiv.2604.13058
“Exam-derived MMMU-style benchmarks have become a standard tool for measuring broad knowledge and reasoning in MLLMs, offering scale, disciplinary breadth, and an interpretable link to human expertise (Hendrycks et al., 2020; Yue et al., 2024; Zhang et al., 2023; Zhong et al., 2024).”
Zhu, Q., Guo, X., Huo, J., Li, J., Liu, P., Yang, W.-Y., Xu, W., & Lin, X. (2026) · arXiv.org · doi:10.48550/arxiv.2603.14463
“General Knowledge & Reasoning: We use MMLU (Hendrycks et al., 2021), MMLU-Pro (Wang et al., 2024b), C-Eval (Huang et al., 2023), CMMLU (Li et al., 2023), AGIEval (Zhong et al., 2023), and ARC-c (Clark et al., 2018).”
MACoT: Synthesizing Chains of Thought for Small Models via Multi-Agent Collaboration
Tang, G., & Zhao, F. (2026) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i35.40214
FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
Liao, N., Wang, X.-X., Qin, X.-H., & Yan, J.-C. (2026) · arXiv.org · doi:10.48550/arxiv.2603.13364
“The evaluation benchmarks include: MMLU (Hendrycks et al., 2021), BBH (Suzgun et al., 2022), Hel-laSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-Challenge (ARC-C) (Clark et al., 2018), ARC-Easy (ARC-E) (Clark et al., 2018), AGIEval (Zhong et al., 2023), MBPP (Austin et al., 2021), GSM8K (Cobbe…”
Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMsPreprint
Zhu, L., Hua, H., Miao, L., Wang, W., & Zhao, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.11674 · Cited by 1
TeachBench: A Syllabus-Grounded Framework for Evaluating Teaching Ability in Large Language ModelsPreprint
Li, Z., Song, S.-Y., Ma, J.-Y., Li, R., Zeng, Y., Li, M.-H., & Sui, Z.-F. (2026) · arXiv.org · doi:10.48550/arxiv.2601.21375
“AGIEval adopts a human-exam-centric design spanning multiple official exams (Zhong et al., 2024); C-Eval and CMMLU provide large-scale Chinese multi-discipline multiple-choice suites across difficulty levels (Huang et al., 2023; Li et al., 2024); and GAOKAO-Bench directly evaluates LLM performance…”
Fang, R., Li, J., Chen, W., Hu, B., Chen, Y.-C., Tang, X., & Diao, L. (2026) · arXiv.org · doi:10.48550/arxiv.2601.17399
“As model performance approaches the ceiling of widely used benchmarks (e.g., CLUE [5], C-Eval [6], AGIEval [7]), score distributions collapse and lose discriminative power.”
Self-citationGao, Z., Xu, Z., Ye, X., & Zhou, B. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2512.18857
“We evaluate models trained with three instantiations of our CORE framework— CORE -Base, CORE - CR, and CORE -KL—on the following out-of-distribution benchmarks: GSM8K (Cobbe et al., 2021), ASDiv (Miao et al., 2021), MAWPS (Koncel-Kedziorski et al., 2016), TabMWP (Lu et al., 2023), MATH (Hendrycks et al., 2021b), MMLU-…”
InfluentialProgress over Points: Reframing LM Benchmarks Around Scientific ObjectivesPreprint
Jin, A., Hendryx, S. M., & Nath, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.11183
“Early work such as MMLU [9] evaluated broad academic and professional knowledge, but top models quickly reached saturation, prompting the development of harder, broader, and more reasoning-heavy benchmarks [26, 16, 28].”
Uncovering Competency Gaps in Large Language Models and Their BenchmarksPreprint
Boháček, M., Scherrer, N., Dufour, N. P., Leung, T., Bregler, C., & Chan, S. C. Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.20638 · Cited by 1
“• Reasoning: AGI Eval [Zhong et al., 2023]; LogicBench [Parmar et al., 2024]; SocialIQA [Sap et al., 2019]; WinoGrande [Sakaguchi et al., 2021].”
SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
Huang, R.-X., Zeng, H., Huang, H.-T., Shi, J., Yu, M.-H., Yen, I. E.-H., & Wang, S. (2025) · arXiv.org · doi:10.48550/arxiv.2512.05409
“We compare normalized accuracy on six non-generative datasets: ARC-easy, ARC-challenge [27], OpenBookQA [28], Hellaswag [29], Winogrande [30], and PIQA [31]; two generative datasets: GSM8k [32] and AGIEval [33], as well as perplexity on Wikitext [34] and Lambada [35] with lm-evaluation [36].”
Ganesaraja, R., Swathika, N., Saravanan, A., Rathinasamy, K., Amancharla, C., Das, R., Panse, S., Batwe, A., Vijayan, D., Ashok, V., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2512.06046
“As industry moves toward AI-augmented enterprise development, specialized autonomous agents will increasingly outperform generalist AI solutions [39, 49].”
Pereyda, C., & Holder, L. B. (2025) · AI · doi:10.3390/ai6120313
“AGIEval [22] includes a large breadth of human-centric IQ tests (e.g., SAT, GRE, . . . ) to determine how AI systems compare to each other and to humans.”
MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems
Jing, H., Hou, Y., Liu, J., Xie, R., Xu, A., Ma, J., & Deng, Q. (2025) · arXiv.org · doi:10.48550/arxiv.2511.18926
Bootstrapping LLMs via Preference-Based Policy Optimization
Jia, C. (2025) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2511.12867
“For evaluation, we adopt a suite of benchmarks including the complex reasoning dataset BBH (Suzgun et al. 2023), knowledge-based benchmarks AGIEval (Zhong et al. 2024), ARC-C (Clark et al. 2018), and MMLU (Hendrycks et al. 2021), as well as the math reasoning benchmark GSM8K (Cobbe et al. 2021).”
Guan, J., & Guan, S. (2025) · International Conference on Big Data and Education (ICBDE) · doi:10.1145/3786183.3786184
“C-CLUE [24] evaluates LLMs’ capabilities in named entity recognition and relation extraction for classical Chinese texts.”
Trade-Offs Between Traditional LLM and Chain-of-Thought Reasoning for Mathematical Tasks
Ravanam, D., & Khare, S. (2025) · Ubiquitous Computing, Electronics & Mobile Communication Conference · doi:10.1109/uemcon67449.2025.11267619
“SAT-MATH [6] and Nguyen-Brat Asdiv (Asdiv) [9] contain varied problems.”
InfluentialMEASER: Malware embedding attacks on open-source LLMs
Tan, M., Li, W., Tao, H., Ma, H.-L., Liu, A.-D., Chen, Q., & Wang, Z.-L. (2025) · doi:10.48550/arxiv.2510.10486
“We utilize MMLU [45] and AGIEval [46] datasets to evaluate the attack performance, which are widely used in evaluating LLMs.”
TORSO: Template-Oriented Reasoning Towards General TasksPreprint
Kim, M., Lee, S., & Lim, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.09448
“…(Cobbe et al., 2021) and ARC-Easy (Clark et al., 2018) for mathematics and science, Truth-fulQA (Lin et al., 2022) for evaluating reliability, RACE (Lai et al., 2017) for reading comprehension, and MMLU (Hendrycks et al., 2021) and GAOKAO (Zhong et al., 2023) for comprehensive knowledge evaluation.”
InfluentialEvaluating the Role of Model Size in Agentic AI for Expert-Like Material Selection
Ying, M. Y., Grandi, D., Groom, A., & McComb, C. (2025) · Design Automation Conference · doi:10.1115/detc2025-168873
RoD-TAL: A Benchmark for Answering Questions in Romanian Driving License Exams
Man, A. V., Smădu, R.-A., Craciun, C.-G., Cercel, D.-C., Pop, F.-C., & Cercel, M.-C. (2025) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2507.19666
Metric assessment protocol in the context of answer fluctuation on MCQ tasks
Goliakova, E., Renard, X., Lesot, M.-J., Laugel, T., Marsala, C., & Detyniecki, M. (2025) · arXiv.org · doi:10.48550/arxiv.2507.15581
“…datasets 6 that contain full final prompts, including instructions, few-shot examples, their order, and option typog-raphy for ARC-C (Clark et al., 2018), CSQA (Talmor et al., 2019), MMLU 7 (Hendrycks et al., 2021), AGIEval 8 (Zhong et al., 2024), and Wino-grande (Sakaguchi et al., 2021) 9 .”
Chavez, I., Jasserme, A., Reddy, S., Lohia, N., Mittal, G., & Mehta, J. (2025) · Intelligence and Security Informatics · doi:10.1109/isi65680.2025.11201167
“Generative Large Language Models (GLMs or LLMs) have acquired extensive general knowledge and human-like reasoning capabilities [9], [10], [11], [12], positioning them as pivotal in constructing Artificial Intelligence (AI) agents known as LLM-based agents.”
SciDA: Scientific Dynamic Assessor of LLMsPreprint
Zhou, J., Miao, T., Liao, Y., Wang, Q., Wen, Z., Wang, Y., Huang, Y., Yan, G., Wang, L., Xia, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.12909
“Further works (GPQA, SuperGPQA, MMLU-Pro, Agieval, Scibench, Scieval, etc.) enable more comprehensive and rigorous assessment by incorporating more disciplines and diversifying data sources [15, 18, 20–23].”
Self-citationBioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining
Merzah, B. M., Taami, T., Asoudeh, S., Mirzaee, S., Pour, A. H., & Bengari, A. A. (2025) · arXiv.org · doi:10.48550/arxiv.2506.21567
Online Knowledge Distillation with Reward Guidance
Jia, C. (2025) · arXiv.org · doi:10.48550/arxiv.2505.18952
“For evaluation, we use multiple benchmarks: the complex reasoning dataset BBH [43], knowledge-based datasets (AGIEval [61], ARC-Challenge [14], and MMLU [22]), and the mathematical reasoning dataset GSM8K [15].”
Understanding Differential Transformer Unchains Pretrained Self-Attentions
Kong, C., Jang, J., & Kwak, N. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.16333
“Table 4 reports results on eight representative benchmarks that span language understanding [29], commonsense reasoning [16], instruction following [92], math [17, 29], code generation [4, 14], and general human task [91].”
A Systematic Analysis of Base Model Choice for Reward Modeling
Ahrabian, K., Jandaghi, P., Mokhberian, N., Karimireddy, S. P., & Pujara, J. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2505.10775
“Moreover, we note much more overfitting and instability when training with the Bradley-Terry method, making obtaining high-quality RMs more challenging.”
Metadata
- Authors
- Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 2299-2314
- arXiv
- 2304.06364
- OpenAlex
- W4401042689
- Cited by
- 211
- References
- 41
- Citations by year
- 2026: 28 · 2025: 97 · 2024: 68 · 2023: 18
- FWCI
- 56.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- standardized testing, college entrance examination, LSAT, GPT-4, artificial general intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
