CoChat

Humanity's Last Exam

Open access
Phan, Long · 2025 · Nature

Preprint — published as 10.1038/s41586-025-09962-4 · Paper page

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.

Abstract

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at lastexam.ai

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-6615297/v1

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Performance of frontier LLMs on popular benchmarks and HLE. Compared with the saturation of other popular capability benchmarks, HLE accuracy remains low across several frontier models, demonstrating its effectiveness for measuring advanced, closed-ended, academic capabilities.
Fig. 1 Performance of frontier LLMs on popular benchmarks and HLE. Compared with the saturation of other popular capability benchmarks, HLE accuracy remains low across several frontier models, demonstrating its effectiveness for measuring advanced, closed-ended, academic capabilities.CC BY-NC-ND · PubMed Central
Fig. 2 Distribution of HLE questions across categories. HLE consists of 2,500 exam questions in over a hundred subjects, grouped into eight high-level categories.
Fig. 2 Distribution of HLE questions across categories. HLE consists of 2,500 exam questions in over a hundred subjects, grouped into eight high-level categories.CC BY-NC-ND · PubMed Central
Fig. 3 Example questions from HLE. Samples of the diverse and challenging questions submitted to HLE.
Fig. 3 Example questions from HLE. Samples of the diverse and challenging questions submitted to HLE.CC BY-NC-ND · PubMed Central
Fig. 4 HLE dataset creation pipeline. We accept questions that make frontier LLMs fail, then iteratively refine them with the help of expert peer reviewers. Each question is then manually approved by organizers or expert reviewers trained by organizers. A private held-out set is kept apart from the public set to assess model overfitting and gaming on the public benchmark.
Fig. 4 HLE dataset creation pipeline. We accept questions that make frontier LLMs fail, then iteratively refine them with the help of expert peer reviewers. Each question is then manually approved by organizers or expert reviewers trained by organizers. A private held-out set is kept apart from the public set to assess model overfitting and gaming on the public benchmark.CC BY-NC-ND · PubMed Central
Fig. 5 Accuracy compared with reasoning token budget. Accuracy binned by the total number of generated output tokens, showing a log-linear increase in accuracy peaking around 214 tokens before reversing.
Fig. 5 Accuracy compared with reasoning token budget. Accuracy binned by the total number of generated output tokens, showing a log-linear increase in accuracy peaking around 214 tokens before reversing.CC BY-NC-ND · PubMed Central
Extended Data Fig. 1 Example of a structured response using an LLM judge. Exact-match answers in HLE sometimes require several reasoning steps to compare the AI’s final answer with the correct answer; therefore, a capable LLM judge with reasoning capabilities is necessary.
Extended Data Fig. 1 Example of a structured response using an LLM judge. Exact-match answers in HLE sometimes require several reasoning steps to compare the AI’s final answer with the correct answer; therefore, a capable LLM judge with reasoning capabilities is necessary.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Part of this paper’s reference list could not be split into single entries.

Show all 41 references

Research works printed but not linked: 19

  1. M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y. Gal, and X. Davies. Agentharm: A benchmark for measuring harmfulness of llm agents, 2024. URL https://arxiv.org/abs/ 2410.09024.
  2. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://api. semanticscholar.org/CorpusID:268232499.
  3. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732.
  4. P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, and T. Wang. Ms marco: A human generated machine reading comprehension dataset, 2018. URL https://arxiv.org/ abs/1611.09268. · arXiv
  5. M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y. Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V. Vontimitta, S. Whitman, and J. Saxe. Purple llama cyberseceval: A secure coding benchmark for language models, 2023. URL https://arxiv.org/abs/ 2312.04724.
  6. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. · arXiv
  7. F. Chollet, M. Knoop, G. Kamradt, and B. Landers. Arc prize 2024: Technical report, 2024. URL https://arxiv.org/abs/2412.04604. · arXiv
  8. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https: //arxiv.org/abs/1903.00161.
  9. A. Dubey et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407. 21783.
  10. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps, 2021. URL https://arxiv.org/abs/2105.09938. · arXiv
  11. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009. 03300.
  12. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv. org/abs/2103.03874.
  13. D. Owen. How predictable is language model benchmark performance?, 2024. URL https: //arxiv.org/abs/2401.04757.
  14. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv. org/abs/2311.12022.
  15. V. K. Srinivasan, Z. Dong, B. Zhu, B. Yu, H. Mao, D. Mosk-Aoyama, K. Keutzer, J. Jiao, and J. Zhang. Nexusraven: A commercially-permissive language model for function calling. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https: //openreview.net/forum?id=5lcPe6DqfI.
  16. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL https: //arxiv.org/abs/1804.07461.
  17. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (published at neurips 2024 track datasets and benchmarks), 2024. URL https://arxiv.org/abs/2406.01574. · arXiv
  18. H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts, 2024. URL https: //arxiv.org/abs/2411.15114.
  19. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URL https: //arxiv.org/abs/2304.06364.

Cited by

See all 1,000

Metadata

Authors
Phan, L.
Affiliations
Center for AI Safety
Corresponding author
Phan, Long
Venue
Nature
Published
2025-05-12
Volume
649
Issue
8099
Pages
1139-1146
ISSN
0028-0836 (ISSN-L)
OpenAlex
W4406840771
Cited by
65
References
14
Citations by year
2026: 51 · 2025: 14
FWCI
174.28
Type
journal article
Language
en
Open access
Locations
  • Nature (publishedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • RiuNet (Universitat Politècnica de València) (publishedVersion, cc-by-nc-nd)
  • Repository for Publications and Research Data (ETH Zurich) (publishedVersion, cc-by-nc-nd)
  • SHURA (Sheffield Hallam University Research Archive) (Sheffield Hallam University) (acceptedVersion)
  • Open Research Online (The Open University) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • HAL (submittedVersion, other-oa)
  • CERN Document Server (European Organization for Nuclear Research) (submittedVersion, cc-by-nc-nd)
  • Jyväskylä University Digital Archive (University of Jyväskylä) (submittedVersion, cc-by-nc-nd)
  • Lirias (KU Leuven) (submittedVersion, cc-by-nc-nd)
  • Tuwhera (Auckland University of Technology) (submittedVersion, cc-by-nc-nd)
  • Open Repository and Bibliography (University of Luxembourg) (submittedVersion, other-oa)
  • Research Explorer (The University of Manchester) (submittedVersion)
  • VUBIR (Vrije Universiteit Brussel) (submittedVersion, cc-by)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
  • Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
  • Research Explorer (The University of Manchester) (submittedVersion, cc-by-nc-nd)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
  • Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc-nd)
  • VUBIR (Vrije Universiteit Brussel) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Office of Science (DE-AC02-06CH11357)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, benchmarking, automated grading, question difficulty
MeSH terms
Artificial Intelligence, Educational Measurement, Humans, Language, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Humanity's Last Exam. https://app.cochat.ai/papers/10.21203/rs.3.rs-6615297/v1
The paper
Phan, L. (2025). Humanity's Last Exam. Nature. https://doi.org/10.21203/rs.3.rs-6615297/v1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.