GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Open accessAbstract
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models have already passed the point at which typical humans can consistently evaluate their output without help.
“Large language models have already passed the point at which typical humans can consistently evaluate their output without help.”
From LLM Critics Help Catch LLM Bugs (2024) · cites this paper as [25]
The Conclusion reports that highly skilled non-experts answered 34% of specialized GPQA questions correctly, compared with 39% for the strongest GPT-4 baseline; the Discussion says that baseline did slightly better than non-expert validators. But the Introduction frames difficulty evaluating model responses as an expectation as LLMs tackle harder questions. The paper does not establish that typical humans already cannot consistently evaluate LLM output in general. (LLM, conf=0.86)
Experimental — accuracy not yet measuredDemonstrations of LLMs' strong performance on PhD-level science questions, among other impressive feats, show that they have passed the point at which typical humans can consistently evaluate their output without help.
“This has been evident since demonstrations of their strong performance on PhD-level science questions, among other impressive feats [25].”
From LLM Critics Help Catch LLM Bugs (2024) · cites this paper as [25]
The Introduction reports that GPT-4 scored 39% on graduate-level science questions, compared with 34% for skilled non-experts, supporting a limited comparison. The Discussion calls this only slightly better than non-expert performance and much worse than expert performance; the Conclusion says the questions remain difficult for AI systems. The paper does not establish that LLMs generally have passed a point where typical humans cannot consistently evaluate their output without help. (LLM, conf=0.85)
Experimental — accuracy not yet measuredLLMs have evolved to exhibit graduate-level STEM proficiency, generate hypotheses, synthesize literature, and draft manuscripts.
“From simple next-token predictors [1, 2], large language models (LLMs) have evolved to exhibit graduate-level STEM proficiency [3-5], generate hypotheses [6, 7], synthesize literature [8], and draft manuscripts [9].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [4]
The Introduction reports that GPT-4 answered 39% of graduate-level biology, physics, and chemistry questions correctly, above the 25% chance rate. But experts achieved 65%, and the Conclusion describes the questions as difficult even for state-of-the-art AI systems. The paper supports some ability to answer graduate-level STEM questions, not an unqualified claim of graduate-level proficiency. The other activities in the citing sentence have separate citations and are not attributed to this source. (LLM, conf=0.84)
Experimental — accuracy not yet measuredRecent breakthroughs have pushed LLMs to PhD-level performance on STEM benchmarks.
“Recent breakthroughs have pushed LLMs to PhD-level performance on STEM benchmarks [4], driving efforts to embed them as generators of the scientific forward pass, encompassing hypothesis generation [6], experimental planning [38], and manuscript drafting [9].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [4]
The Introduction reports that GPT-4 scored 39% on graduate-level physics, chemistry, and biology questions, compared with 34% for non-experts who had or were pursuing PhDs in other domains. But domain experts scored 65%, and the Discussion says the best model performed much worse than expert validators. The paper supports performance comparable to out-of-domain PhD-level non-experts on GPQA, not PhD-level domain-expert performance across STEM benchmarks. (LLM, conf=0.85)
Experimental — accuracy not yet measuredGPQA evaluates agents' ability for coordinated tool use when solving challenging graduate-level problems.
“GPQA (Rein et al., 2023) evaluates agents' ability for coordinated tool use when solving challenging graduate-level problems.”
From OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024) · cites this paper as Rein et al., 2023
The Introduction describes GPQA as an evaluation dataset of graduate-level questions in physics, chemistry, and biology. It does not describe GPQA as evaluating agents’ ability to coordinate tool use; its stated focus is answering difficult questions and supporting scalable-oversight research. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Popular knowledge-centric evaluation benchmarks use prompting set-ups that are much more similar to how examination questions for humans are written than how people actually use language models.
“Popular knowledge-centric evaluation benchmarks use prompting set-ups that are much more similar to how examination questions for humans are written than how people actually use language models (Son et al., 2024; Rein et al., 2023).”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Rein et al., 2023
The “Data for Scalable Oversight” section says that curated question-answering benchmarks draw on standardized tests and gives MMLU as an example. The paper’s baseline prompts also use multiple-choice questions. These passages support the examination-question resemblance, but the paper does not establish that popular benchmarks’ prompting set-ups are more like exams than the way people actually use language models. (LLM, conf=0.78)
Experimental — accuracy not yet measuredGPQA has subject-level subsets.
“(7) GPQA, introduced by Rein et al. (2023), with subject-level subsets.”
From Sabotage Evaluations for Frontier Models (2024) · cites this paper as Rein et al. (2023)
Section 2.2 (Domains; page unavailable) organizes GPQA questions by biology, physics, and chemistry and their subdomains, supporting subject-level breakdowns. But Section 2.3 (Dataset Splits; page unavailable) describes the released curated subsets as selections based on validator responses, not as subject-level subsets. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe development of evaluations in tandem with language models has substantially contributed to the rapid advancement of these models.
“The development of evaluations in tandem with language models (LMs) has substantially contributed to the rapid advancement of these models [29, 11, 7, 25, 81, 27, 72].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [72]
The Introduction describes rapid advances in language-model capabilities, and Related Work says evaluation datasets must improve as models advance. Neither section says that developing evaluations has substantially contributed to those advances. (LLM, conf=0.81)
Experimental — accuracy not yet measuredMMLU-PRO and GPQA primarily evaluate intrinsic reasoning ability in answering fixed-choice questions rather than generative Q&A tasks.
“However, these benchmarks primarily evaluate intrinsic reasoning ability in answering fixed-choice questions rather than generative Q&A tasks.”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Rein et al., 2023
The Introduction (page not available) describes GPQA as an evaluation dataset of graduate-level multiple-choice questions, supporting the fixed-choice rather than generative-Q&A distinction for GPQA. It presents GPQA chiefly as a testbed for difficult, expertise-dependent questions and scalable oversight, not specifically as a measure of intrinsic reasoning ability. The cited paper does not establish the claim about MMLU-PRO. (LLM, conf=0.84)
Experimental — accuracy not yet measuredGPQA tests LLMs' capabilities in solving complex tasks across physics, chemistry, and biology.
“Furthermore, multidisciplinary benchmarks like GPQA [330] test LLMs' capabilities in solving complex tasks across physics, chemistry, and biology.”
From A Survey of Large Language Models (2026) · cites this paper as [330]
The dataset size aligns with recent trends toward compact, high-quality benchmarks.
“Although modest in size, our dataset aligns with recent trends toward compact, high-quality benchmarks: MT-Bench (80 items) [16], GPQA-D (198 items) [4], AIME 2024/2025 (30 items each) [35], USAMO 2025 (6 items) [36] and PaperBench (20 items) [37].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [4]
Li et al. use a custom model architecture to achieve long context lengths and only evaluate models of up to 1.3B parameters, which are ineffective for complex tasks such as GPQA.
“However, Li et al. [31] use a custom model architecture [74] to achieve long context lengths, and only evaluate models of up to 1.3B parameters, which is several orders of magnitude smaller than state-of-the-art language models, and are ineffective for complex tasks, such as GPQA [52].”
From Many-Shot In-Context Learning (2024) · cites this paper as [52]
GPQA has questions that are hard to find the answer to on the internet, and GPT-4 has an accuracy of less than 40% with few-shot chain of thought.
“GPQA (Rein et al., 2023) is a benchmark written by biology, physics, and chemistry domain experts. It has questions that are hard to find the answer to on the internet, and GPT-4 has an accuracy of less than 40% with few-shot chain of thought.”
From Debating with More Persuasive LLMs Leads to More Truthful Answers (2024) · cites this paper as Rein et al., 2023
Existing benchmarks primarily assess multiple-choice reasoning or human preference alignment.
“Existing benchmarks, such as Hugging Face leaderboard tasks (Wang et al., 2024;Rein et al., 2023) and alignment-focused chat-based evaluations (Reddy et al., 2019;Zheng et al., 2023;Köpf et al., 2024), primarily assess multiple-choice rea-soning or human preference alignment.”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Rein et al., 2023
Frontier AI systems demonstrate impressive capabilities on a wide range of software benchmarks and in experiments measuring AI's impact on developer productivity when completing synthetic tasks.
“Frontier AI systems demonstrate impressive capabilities on a wide range of software benchmarks [3; 4; 5; 6; 7; 8; 9] and in experiments measuring AI's impact on developer productivity when completing synthetic tasks [10; 11].”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [8]
The Baselines section evaluates AI systems on GPQA, a science question-answering benchmark, not a software benchmark. The Conclusion describes GPQA as difficult even for state-of-the-art AI systems. Our copy does not address performance on software benchmarks; the developer-productivity clause is attributed to different citations. (LLM, conf=0.78)
Experimental — accuracy not yet measuredOpenHands with CodeActAgent v1.5 and GPT4o achieved 53.1% accuracy on the GPQA Diamond Set, 49.3% on the Main Set, and 52.8% on the Extended Set.
“| OpenHands + CodeActAgent v1.5 + GPT4o | 53.1 | 49.3 | 52.8 | 0.054 |”
From OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024) · cites this paper as Rein et al. (2023)
Table 5 identifies the GPQA Extended, Main, and Diamond sets, but reports results for other models. Searches for OpenHands, CodeActAgent, GPT-4o, and the three stated percentages found no passage reporting this evaluation in our copy. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 23
- D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. URL https://arxiv.org/abs/1606.06565. · arXiv
- S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukošiūtė, A. Askell, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Olah, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, J. Kernion, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, L. Lovitt, N. Elhage, N. Schiefer, N. Joseph, N. Mercado, N. DasSarma, R. Larson, S. McCandlish, S. Kundu, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Telleen-Lawton, T. Brown, T. Henighan, T. Hume, Y. Bai, Z. Hatfield-Dodds, B. Mann, and J. Kaplan. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022. URL https://arxiv.org/abs/2211.03540. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/ 2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
- D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations, 2 Oct. 2020. URL https://openreview.net/pdf?id=d7KBjmI3GmQ.
- G. Irving, P. Christiano, and D. Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018. URL https://arxiv.org/abs/1805.00899. · arXiv
- A. Jacovi, A. Caciularu, J. Herzig, R. Aharoni, B. Bohnet, and M. Geva. A comprehensive evaluation of tool-assisted generation strategies. arXiv preprint arXiv:2310.10062, 2023. URL https://arxiv.org/abs/2310.10062. · arXiv
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero- shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22199-22213.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf.
- J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018. URL https://arxiv.org/abs/1811.07871. · arXiv
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. C. Malaviya, S. Lee, S. Chen, E. Sieber, M. Yatskar, and D. Roth. Expertqa: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.07852, 2023. URL https://arxiv.org/abs/2309. 07852. · arXiv
- J. Michael, S. Mahdi, D. Rein, J. Petty, J. Dirani, V. Padmakumar, and S. R. Bowman. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023. URL https://arxiv.org/abs/ 2311.08702. · arXiv
- T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. URL https://arxiv.org/abs/1301.3781. · arXiv
- R. Y. Pang, A. Parrish, N. Joshi, N. Nangia, J. Phang, A. Chen, V. Padmakumar, J. Ma, J. Thompson, H. He, and S. Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5336-5358, Seattle, United States, July 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.391. URL https://aclanthology.org/2022.naacl-main.391. · DOI · Paper page
- E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. Mueller, J. Hyun, J. Landau, K. Ndousse, L. Goldberg, L. Lovitt, M. Lucas, M. Sellitto, M. Zhang, N. Kingsland, N. Elhage, N. Joseph, N. Mercado, N. DasSarma, O. Rausch, R. Larson, S. McCandlish, S. Johnston, S. Kravec, S. E. Showk, T. Lanham, T. Telleen-Lawton, T. Brown, T. Henighan, T. Hume, Y. Bai, Z. Hatfield-Dodds, J. Clark, S. R. Bowman, A. Askell, R. Grosse, D. Hernandez, D. Ganguli, E. Hubinger, N. Schiefer, and J. Kaplan. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022b. URL https://arxiv.org/abs/2212.09251. · arXiv
- O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. URL http://arxiv.org/abs/2210.03350. · arXiv
- C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html.
- P. Roit, A. Klein, D. Stepanov, J. Mamou, J. Michael, G. Stanovsky, L. Zettlemoyer, and I. Dagan. Controlled crowdsourcing for high-quality QA-SRL annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7008-7013, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.626. URL https://aclanthology.org/2020.acl-main.626. · DOI
- M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, S. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023. URL https://arxiv.org/abs/2310.13548. · arXiv
- A. Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. URL https://arxiv.org/abs/2206.04615. · arXiv
- H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https://arxiv.org/abs/2307.09288. · arXiv
- W. Yu, Z. Jiang, Y. Dong, and J. Feng. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. In International Conference on Learning Representations, 25 Sept. 2019. URL https://openreview.net/pdf?id=HJgJtT4tvB.
- Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. URL https://arxiv.org/abs/2309.01219. · arXiv
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. (2025) · Nature · doi:10.1038/s41586-025-09422-z · Cited by 1,701 · see citing works
Larger and more instructable language models become less reliable
Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J. (2024) · Nature · doi:10.1038/s41586-024-07930-y · Cited by 203 · see citing works
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
Zero-shot evaluation reveals limitations of single-cell foundation models
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025) · Genome Biology · doi:10.1186/s13059-025-03574-x · Cited by 109 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candes, E., & Hashimoto, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1025 · Cited by 76 · see citing works
Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.21631 · Cited by 27 · see citing works
“(1) Knowledge: MMLU-Pro (Wang et al., 2024f), MMLU-Redux (Gema et al., 2024), GPQA (Rein et al., 2023), Su-perGPQA (Team, 2025), (2) Reasoning: AIME-25 (AIME, 2025), HMMT-25 (HMMT, 2025), LiveBench (2024-11-25) (White et al., 2024), (3) Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wang et al., 2025c),…”
InfluentialHow should we test AI for human-level intelligence? OpenAI’s o3 electrifies quest
Jones, N. (2025) · Nature · doi:10.1038/d41586-025-00110-6 · Cited by 25 · see citing works
AI now beats humans at basic tasks — new benchmarks are needed, says major report
Jones, N. (2024) · Nature · doi:10.1038/d41586-024-01087-4 · Cited by 21 · see citing works
‘In awe’: scientists impressed by latest ChatGPT model o1
Jones, N. (2024) · Nature · doi:10.1038/d41586-024-03169-9 · Cited by 21 · see citing works
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
DeepSeek-AI, Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.02556 · Cited by 19 · see citing works
Hou, B., Mukherjee, P., Batheja, V., Wang, K. C., Summers, R. M., & Lu, Z. (2025) · Radiology · doi:10.1148/radiol.250617 · Cited by 18 · see citing works
Self-citationLAB-Bench: Measuring Capabilities of Language Models for Biology Research
Laurent, J. M., Janizek, J. D., Ruzo, M., Hinks, M. M., Hammerling, M. J., Narayanan, S., Ponnapati, M., White, A. D., & Rodriques, S. G. (2024) · arXiv · doi:10.48550/arxiv.2407.10362 · Cited by 21 · see citing works
“A common problem encountered when trying to effectively scale benchmark generation is managing the balance between ease of automatic generation vs. task complexity often necessary to produce high-quality questions, which others have addressed to varying degrees [45, 46].”
BackgroundOpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24 · see citing works
Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems
Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., & Wang, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.751 · Cited by 14 · see citing works
Voting or Consensus? Decision-Making in Multi-Agent Debate
Kaesberg, L. B., Becker, J., Wahle, J. P., Ruas, T., & Gipp, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.606 · Cited by 14 · see citing works
“3% on GPQA over the voting average.”
InfluentialHersh, W., & Hollis, K. F. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01251-0 · Cited by 14 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12 · see citing works
TextGrad: Automatic "Differentiation" via Text
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07496 · Cited by 12 · see citing works
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Becker, J., Rush, N., Barnes, E., & Rein, D. (2025) · arXiv · doi:10.48550/arxiv.2507.09089 · Cited by 16 · see citing works
“Frontier AI systems demonstrate impressive capabilities on a wide range of software benchmarks [3; 4; 5; 6; 7; 8; 9] and in experiments measuring AI’s impact on developer productivity when completing synthetic tasks [10; 11].”
Self-citationDebating with More Persuasive LLMs Leads to More Truthful Answers
Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., & Perez, E. (2024) · arXiv · doi:10.48550/arxiv.2402.06782 · Cited by 13 · see citing works
Self-citationEducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework
Shi, Y., Liang, R., & Xu, Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1576 · Cited by 11 · see citing works
“Task-oriented performance benchmarks like MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024b), and GPQA (Rein et al., 2023) employ closed-ended questions to evaluate domain knowledge and reasoning abilities.”
InfluentialA roadmap for evaluating moral competence in large language models
Haas, J., Bridgers, S., Manzini, A., Henke, B., May, J., Levine, S., Weidinger, L., Shanahan, M., Lum, K., Gabriel, I., et al. (2026) · Nature · doi:10.1038/s41586-025-10021-1 · Cited by 12 · see citing works
Lessons from the trenches on evaluating machine learning systems in materials science
Alampara, N., Schilling-Wilhelmi, M., & Jablonka, K. M. (2025) · Computational Materials Science · doi:10.1016/j.commatsci.2025.114041 · Cited by 11 · see citing works
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.11018 · Cited by 35 · see citing works
“For translation, we report performance on English to Bemba, summarization uses XLSum, MATH corresponds to the MATH500 test set, and sentiment analysis results are reported with semantically-unrelated labels.”
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
“We observe that CoTs on GPQA are consistently less faithful than those on MMLU across 5 out of 6 bias categories (the other being a tie of 0% faithfulness) for Claude 3.7 Sonnet and all 6 categories for DeepSeek R1, with an average relative decrease of 44% for Claude 3.7 Sonnet and 32% for DeepSeek R1.”
InfluentialSelf-citationLLM Critics Help Catch LLM Bugs
McAleese, N., Pokorny, R. M., Uribe, J. F. C., Nitishinskaya, E., Trebacz, M., & Leike, J. (2024) · arXiv · doi:10.48550/arxiv.2407.00215 · Cited by 10 · see citing works
“This has been evident since demonstrations of their strong performance on PhD-level science questions, among other impressive feats [25].”
BackgroundAgentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools
Wu, J., Zhu, J., Liu, Y., Xu, M., & Jin, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.04644 · Cited by 31 · see citing works
Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
Grabb, D., Lamparth, M., & Vasan, N. (2024) · arXiv · doi:10.48550/arxiv.2406.11852 · Cited by 39 · see citing works
GDPVAL: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
Patwardhan, T., Dias, R., Proehl, E., Kim, G., Wang, M., Watkins, O., Fishman, S. P., Aljubeh, M., Thacker, P., Fauconnet, L., et al. (2026) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i4.17197 · Cited by 7 · see citing works
On scalable oversight with weak LLMs judging strong LLMs
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Albanie, S., Bulian, J., Agarwal, R., Lindner, D., Tang, Y., Goodman, N. D., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04622 · Cited by 6 · see citing works
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works
“• Evaluate Models in Realistic Settings : Popular knowledge-centric evaluation benchmarks use prompting set-ups that are much more similar to how examination questions for humans are written than how people actually use language models (Son et al., 2024; Rein et al., 2023).”
When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Son, G., Hong, J., Fan, H., Nam, H., Ko, H., Lim, S., Song, J., Choi, J., Paulo, G., Yu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.11855 · Cited by 5 · see citing works
“Figure 3 illustrates the performance of o3 on six benchmarks: MathVista [50], MMLU-Pro [51], GPQA Di-amond [4], MMMU [49], HLE [52] and S POT (re-call). o3 exceeds 80% on the first four benchmarks, demonstrating robust general reasoning and code understanding.”
InfluentialU-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack
Gao, Y., Xiong, Y., Wu, W., Li, B., Zhong, Y., & Wang, H. (2026) · ACM Transactions on Information Systems · doi:10.1145/3786609 · Cited by 6 · see citing works
Zeng, Z., Cheng, Q., Yin, Z., Zhou, Y., & Qiu, X. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.232 · Cited by 6 · see citing works
Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1354 · Cited by 5 · see citing works
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., & Pretorius, A. (2023) · arXiv · doi:10.48550/arxiv.2311.17371 · Cited by 5 · see citing works
The future of open human feedback
Don-Yehiya, S., Burtenshaw, B., Astudillo, R. F., Osborne, C., Jaiswal, M., Kuo, T.-S., Zhao, W., Shenfeld, I., Peng, A., Yurochkin, M., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01038-2 · Cited by 6 · see citing works
ChatGPT Solving Complex Kidney Transplant Cases: A Comparative Study With Human Respondents
Mankowski, M. A., Jaffe, I. S., Xu, J., Bae, S., Oermann, E. K., Aphinyanaphongs, Y., McAdams‐DeMarco, M., Lonze, B. E., Orandi, B. J., Stewart, D. E., et al. (2024) · Clinical Transplantation · doi:10.1111/ctr.15466 · Cited by 5 · see citing works
LIMO: Less is More for Reasoning
Ye, Y., Huang, Z., Xiao, Y., Chern, E., Xia, S., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2502.03387 · Cited by 4 · see citing works
“Third, we incorporate multi-disciplinary benchmarks including Minerva (Lewkowycz et al., 2022) (undergraduate-level STEM) and GPQA (Rein et al., 2023) to evaluate the transfer of mathematical reasoning skills to broader contexts beyond our training domain.”
InfluentialSafetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10 · see citing works
Freeman, J., Millikin, R. J., Xu, L., Sharma, I., Moore, B., Lock, C., Shine George, K., Bal, A., Mohanty, C., & Stewart, R. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06350-7 · Cited by 4 · see citing works
CiteME: Can Language Models Accurately Cite Scientific Claims?
Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., & Bethge, M. (2024) · arXiv · doi:10.48550/arxiv.2407.12861 · Cited by 4
Do Large Language Models Align with Core Mental Health Counseling Competencies?
Nguyen, V. C., Taher, M., Hong, D., Possobom, V. K., Gopalakrishnan, V. T., Raj, E., Li, Z., Soled, H. J., Birnbaum, M. L., Kumar, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.22446 · Cited by 3 · see citing works
Jin, R., Guo, Y., Qu, Y., Yang, M., Shang, C., Yang, Q., Chao, L., Zhou, Y., Xu, R. M., Xu, Z., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.03.674085 · Cited by 3 · see citing works
Evidence from formal logical reasoning reveals that the language of thought is not natural language
Kean, H., Fung, A., Jaggers, P., Chen, J., Rule, J. S., Benn, Y., Tenenbaum, J. B., Piantadosi, S. T., Varley, R. A., & Fedorenko, E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2520095123 · Cited by 2 · see citing works
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.13543 · Cited by 2 · see citing works
“LLMs possess vast knowledge across domains (Brown, 2020; Hendrycks et al., 2020), can reason in specific scenarios (Wei et al., 2022a; Shinn et al., 2023; Rein et al., 2023), and can reliably follow human instructions in simple settings (Ouyang et al., 2022).”
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., & Glaese, A. (2025) · arXiv · doi:10.48550/arxiv.2504.12516 · Cited by 2 · see citing works
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., & Balasubramanian, N. (2024) · arXiv · doi:10.48550/arxiv.2407.18901 · Cited by 2 · see citing works
RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15114 · Cited by 2 · see citing works
Observational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3 · see citing works
Sabotage Evaluations for Frontier Models
Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., Durmus, E., Ganguli, D., Kravec, S., Shlegeris, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.21514 · Cited by 2 · see citing works
Self-citationKimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 2 · see citing works
Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
Zhu, X., Zhang, C., Chi, Y., Stafford, T., Collier, N., & Vlachos, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1694 · Cited by 2 · see citing works
What Large Language Models Know About Plant Molecular BiologyPreprint
Burda, M. F., Ferrero, L., Gaggion, N., Fonouni‐Farde, C., Crespi, M. D., Ariel, F., & Ferrante, E. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.31.672925 · Cited by 2 · see citing works
MiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1 · see citing works
“Its strong performance on SuperGPQA further highlights its ability to handle complex graduate-level questions.”
InfluentialRobeyns, M., Szummer, M., & Aitchison, L. (2025) · arXiv · doi:10.48550/arxiv.2504.15228 · Cited by 1 · see citing works
OpenThoughts: Data Recipes for Reasoning Models
Guha, E., Marten, R., Keh, S., Raoof, N., Smyrnis, G., Bansal, H., Nezhurina, M., Mercat, J., Vu, T., Sprague, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.04178 · Cited by 1 · see citing works
AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot
Biswas, J., Schoepp, S., Vasan, G., Opipari, A., Zhang, A., Hu, Z., Joseph, S., Lease, M., Li, J. J., Stone, P., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.13940 · Cited by 1 · see citing works
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1 · see citing works
Application and progress of artificial intelligence agents in drug development
Zhao, D., & Hsieh, C. (2026) · Journal of Zhejiang University (Medical Sciences) · doi:10.3724/zdxbyxb-2025-0697 · Cited by 1 · see citing works
Lv, Y., Xu, Y., Xu, G., & Ma, J. Y. (2026) · Genome biology · doi:10.1186/s13059-026-04109-8 · Cited by 1 · see citing works
Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents
Rafiee, M., Soudani, H., Abbasiantaeb, Z., Aliannejadi, M., Hasibi, F., & Zamani, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808629 · Cited by 1 · see citing works
Towards a Science of Scaling Agent Systems
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.08296 · Cited by 1 · see citing works
ARS: Adaptive Reasoning Suppression for Efficient Large Reasoning Language Models
Zheng, D. (2025) · arXiv · doi:10.48550/arxiv.2510.00071
“These models employ extended reasoning chains with reflection behaviors, backtracking, and self-verification processes that significantly enhance problem-solving capabilities in mathematics [10], programming [4], and scientific reasoning [19].”
LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
Zhang, G., Meng, F., Wan, G., Li, Z., Wang, K., Yin, Z., Bai, L., & Yan, S. (2025) · arXiv · doi:10.48550/arxiv.2509.24771
“…■ mathematical reasoning , including GSM8K (Cobbe et al., 2021), MATH-500 (Hendrycks et al., 2021b), and AIME 2024/2025 (Li et al., 2024); ■ scientific reasoning , SciBench (Wang et al., 2024b) and GPQA-Diamond (Rein et al., 2023); ■ medical reasoning , JAMA Clinical Challenge (Chen et al., 2025a).”
Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
Liu, R., Sun, Y., Zhang, M., Bai, H., Yu, X., Yu, T., Yuan, C., & Hou, L. (2025) · arXiv · doi:10.48550/arxiv.2504.04823
“3) GPQA-Diamond (Rein et al., 2024), a graduate-level proof question and answer benchmark that tests the ability of models to generate accurate mathematical proofs.”
SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
Zhang, Q., Chen, K., Liao, N., Lin, Z., Tang, B., Xiong, F., Li, Z., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2606.01751
“…applicability in collaborative reasoning systems, including MATH [25], GSM-Hard [26], and AQUA-RAT [27] for mathematical reasoning; SciBench [28] and GPQA [29] for scientific reasoning; MMLU-Pro [30] for general knowledge understanding; and MedQA [31] and MedMCQA [32] for medical-domain evaluation.”
Testing Frontier Large Language Models' Physics Literacy in Parallel Physical Worlds
Zhang, D. (2026) · arXiv · doi:10.48550/arxiv.2607.00276
“Rein et al.’s GPQA Diamond pushes difficulty up with 198 PhD-level questions across three science disciplines, where in-domain PhD experts score 65% and Claude 3.5 Sonnet 59.4% (Rein et al., 2024).”
InfluentialCritique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Zhang, X., Zhang, Y., Sun, H., Feng, K., Lu, C., Yang, C., & Meng, H. (2025) · arXiv · doi:10.48550/arxiv.2506.03106
“To enable a holistic evaluation, we assess performance on in-distribution (ID) tasks using Minerva-Math [37] and on out-of-distribution (OOD) tasks using GPQA-Diamond (physics, chemistry, biology) [38].”
Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Li, X., Liu, M., & Yuen, C. (2026) · arXiv · doi:10.48550/arxiv.2609.35279
“The artifact release plan covers the probe set, response traces, evaluation code, and the multi-round debate transcripts under the underlying-benchmark licenses (MMLU-Pro [25], ARC-Challenge [4], GPQA [18], and TruthfulQA [14]).”
Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
Chen, H.-Y. (2026) · arXiv · doi:10.48550/arxiv.2604.21611
“GPQA Diamond [Rein et al., 2023] is a graduate-level multiple-choice benchmark ( n = 198 ; SOTA 94.1%).”
InfluentialFrom Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
Wang, H., Liang, W., Fu, Z., Zheng, N., Zhang, Y., Tong, Y., Zhu, T., Jiang, H., Li, C., Wu, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.23196
“Except the close-ended benchmarks AIME 2025 (Mathematical Association of America, 2025) and GPQA Diamond (Rein et al., 2024), we conduct evaluation on open-ended General Reasoning task, which is constructed from the GeneralThought (GeneralRea-soning, 2025) covering a broad spectrum of commonsense and domain-general re…”
Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
Wei, T., Sachdeva, N., Coleman, B., He, Z., Bei, Y., Ning, X., Ai, M., Li, Y., He, J., Chi, E. H., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.20857 · Cited by 1 · see citing works
Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.16402
HCAST: Human-Calibrated Autonomy Software Tasks
Rein, D., Becker, J., Deng, A., Nix, S., Canal, C., O'Connel, D., Arnott, P., Bloom, R., Broadley, T., Garcia, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.17354
“Recent examples include GPQA (Rein et al., 2023), FrontierMath (Glazer et al., 2024), and Humanity’s Last Exam (Phan et al., 2025).”
Self-citationMergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“Tasks are grouped into several categories, including: (a) general knowledge/reasoning (ARC (Bhak-thavatsalam et al., 2021), AGIEval (Zhong et al., 2024), OpenBookQA (Mihaylov et al., 2018), BBH (Suzgun et al., 2023), WorldSense (Hong et al., 2025), PIQA (Bisk et al., 2020), hellaswag (Zellers et al., 2019) and KOR-Ben…”
InfluentialSealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y.-M., & Vu, T. (2025) · arXiv · doi:10.48550/arxiv.2506.01062
“2 For example, the widely used GPQA-D IAMOND [Rein et al., 2024], a compact set of 198 expert-vetted questions, demonstrates how a small, carefully curated dataset can effectively assess a model’s reasoning ability.”
Xie, S., Shi, Z., Shen, H., Wang, J., Zhong, S., Liu, M., Pan, C., Jia, P., Sun, B., Jing, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.21482
“Adjacent work covers expert-level academic questions (14; 39), scientific-literature synthesis (6), domain-specific analytical workflows (8; 20; 54), broader agentic evaluation (26; 48; 19; 49; 42; 28), and benchmark construction (23; 22; 55; 7).”
OmniRouter: Budget and Performance Controllable Multi-LLM Routing
Mei, K., Xu, W., Guo, M., Lin, S., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.20576 · Cited by 2 · see citing works
TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs
Rodionov, V., & Assylbekov, S. (2026) · arXiv · doi:10.48550/arxiv.2608.11415
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
RSIGym: A Flexible Environment for Recursive Self-Improvement
Meng, F., Du, L., Lu, H., Chen, Q., Zhao, Z., Wu, Z., Zhuo, J., Hu, M., & Shieh, M. Q. (2026) · arXiv · doi:10.48550/arxiv.2610.10310
“Here we examine two common patterns—harness improvements and the effects of training—and how research behavior differs across Claude, DeepSeek, and GPT agents.”
MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision
Ke, Z., Xu, A., Ming, Y., Nguyen, X.-P., Chin, R., Xiong, C., & Joty, S. (2025) · arXiv · doi:10.48550/arxiv.2505.14996
Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model
Lab, N., :, Yang, C., Huang, C., Lan, F., Chen, H., Zhou, H., Song, H., Cao, J., Zhu, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.22083
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Liu, M. T.-Y., Harrington, A., Darrell, T., & Bai, Y. (2026) · arXiv · doi:10.48550/arxiv.2610.21179
“We use HealthBench (Arora et al., 2025), GPQA Diamond (Rein et al., 2023), Qwen3-4B-Base (Yang et al., 2025), and RaR-Medicine / RaR-Science (Gunjal et al., 2025) in accordance with their respective licenses and terms of use.”
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
“GPQA (Rein et al., 2023) similarly includes multiple-choice questions authored by domain experts in Biology, Physics, and Chemistry.”
Wang, Y., Zhang, X., Yao, K., Zeng, Z., Song, K., Lin, J., & Shen, Z. (2026) · arXiv · doi:10.48550/arxiv.2606.05241
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
Suzgun, M., Yuksekgonul, M., Bianchi, F., Jurafsky, D., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.07952
A DATA ANNOTATION APPROACH USING LARGE LANGUAGE MODELS
ROCHA, C. V. M. (2024) · doi:10.17771/pucrio.acad.68379
Comparing the Neuropsychology Knowledge Base of Publicly Available Large Language Models
Kronenberger, O. R., Hutnyan, M., Kaser, A. N., Bullinger, L., Lacritz, L. H., & Cullum, C. M. (2025) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/acaf112
Ishii, M., Watanabe, Y., Suzuki, I., & Takahashi, Y. K. (2026) · Science and Technology of Advanced Materials Methods · doi:10.1080/27660400.2026.2613512
Making GenAI valuable: Benchmarks, singularities, and the enrichment economy
Aradau, C., & Blanke, T. (2026) · Big Data & Society · doi:10.1177/20539517261451463
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
Baack, S., Buschek, C., & Boháček, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812240
Bell, C. (2026) · Springer eBooks · doi:10.1007/978-3-032-17646-2_14
Bailo, D., Lauciani, V., Stefano, R. D., & Bernardi, F. (2026) · Earth Science Informatics · doi:10.1007/s12145-026-02197-5
Generative AI-Enhanced Sector-Based Investment Portfolio Construction
Romanko, O., Voronina, A., Cao, R., Kwon, R. H., & Mendoza‐Arriaga, R. (2026) · Operations Research Forum · doi:10.1007/s43069-026-00686-3
Evaluating Gemma 3 27B for local undergraduate physics tutoring
Völschow, M., Buczek, P., Riefer, P., Jasko, A., & Steinbach, P. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02296-8
Exploring the Abstention Ability of LLMs using Scientific Reasoning in Critical Fields
Summers, L., Yoon, H., Zheng, W., & Feng, Y. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3847298
Development and Exploratory Evaluation of a Large Language Model-Supported Scoping Review Workflow
Kuo, F. H., Fierstein, J. L., de la Concha, M., Saeed, Y., Shargo, R., Ashour, D., Amankwah, E., Crabtree, E., Rehman, M., Najjar, P., et al. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02477-z
- Authors
- Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R.
- Affiliations
- Anthropic (United States) (US)
- New York University (US)
- ORCID
- Dirani, Julien
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2311.12022
- OpenAlex
- W4388926373
- Cited by
- 30
- References
- 0
- Citations by year
- 2026: 14 · 2025: 11 · 2024: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- Faculty Digital Archive (New York University Florence) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2046556, 1922658)
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- GPQA, question answering benchmark, multiple-choice questions, scientific knowledge, human oversight of AI, human-AI collaboration, GPT-4, question difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
