CoChat
Sign Up
PreprintCited by 75Psychiatry, Mental Health, NeuroscienceArtificial Intelligence ApplicationsCognitive Abilities and Testing

The DOI matches its DataCite record; 47 of the 48 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 25 sources its claims cite is checked for support yet.

A Definition of AGI

Open access
Hendrycks, Dan, Song, Dawn, Szegedy, Christian, Lee, Honglak, Gal, Yarin, Brynjolfsson, Erik, Li, Sharon, Zou, Andy, Levine, Lionel, Han, Bo, Fu, Jie, Liu, Ziwei, Shin, Jinwoo, Lee, Kimin, Mazeika, Mantas, Phan, Long, Ingebretsen, George, Khoja, Adam, Xie, Cihang, Salaudeen, Olawale, Hein, Matthias, Zhao, Kevin, Pan, Alexander, Duvenaud, David, Li, Bo, Omohundro, Steve, Alfour, Gabriel, Tegmark, Max, McGrew, Kevin, Marcus, Gary, Tallinn, Jaan, Schmidt, Eric, Bengio, Yoshua·2025·arXiv
DOI 10.48550/arxiv.2510.18212arXiv 2510.18212
Open this paper in CoChat — free accountOpen PDF

Abstract

The lack of a concrete definition for Artificial General Intelligence (AGI) obscures the gap between today's specialized AI and human-level cognition. This paper introduces a quantifiable framework to address this, defining AGI as matching the cognitive versatility and proficiency of a well-educated adult. To operationalize this, we ground our methodology in Cattell-Horn-Carroll theory, the most empirically validated model of human cognition. The framework dissects general intelligence into ten core cognitive domains-including reasoning, memory, and perception-and adapts established human psychometric batteries to evaluate AI systems. Application of this framework reveals a highly "jagged" cognitive profile in contemporary models. While proficient in knowledge-intensive domains, current AI systems have critical deficits in foundational cognitive machinery, particularly long-term memory storage. The resulting AGI scores (e.g., GPT-4 at 27%, GPT-5 at 57%) concretely quantify both rapid progress and the substantial gap remaining before AGI.

Checked 2026-11-08 · 1 reference not checked yet

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper defines AGI as matching or exceeding the cognitive versatility and proficiency of a well-educated adult, operationalized through ten CHC-based cognitive domains. Applying this framework yields AGI scores of 27% for GPT-4 and 57% for GPT-5, revealing a “jagged” profile with critical deficits in long-term memory storage.

Key findings

  • Current AI systems show highly uneven cognitive profiles: GPT-4 scores 27% overall and GPT-5 scores 57%, with both models at 0% on Long-Term Memory Storage (MS).
  • GPT-5 improves over GPT-4 in several domains, including Mathematical Ability (4% → 10%), Reading and Writing (6% → 10%), On-the-Spot Reasoning (0% → 7%), Visual Processing (0% → 4%), and Auditory Processing (0% → 6%).
  • Long-term memory storage is described as “perhaps the most significant bottleneck,” scoring near 0% for current models; both GPT-4 and GPT-5 also score 0% on Retrieval Precision/Hallucinations within Long-Term Memory Retrieval.
  • The framework identifies “capability contortions,” such as using large context windows to compensate for missing long-term memory and using RAG to mask retrieval imprecision and hallucination.

Methods

  • The authors ground their definition in Cattell-Horn-Carroll (CHC) theory and decompose general intelligence into ten equally weighted broad abilities: General Knowledge, Reading and Writing, Mathematical Ability, On-the-Spot Reasoning, Working Memory, Long-Term Memory Storage, Long-Term Memory Retrieval, Visual Processing, Auditory Processing, and Speed.
  • Each broad ability is further divided into narrow abilities with illustrative tasks and, in places, benchmark thresholds; the framework is not an automatic evaluation or dataset but a collection of task specifications that can be manually assessed.
  • The paper applies the framework to GPT-4 (2023) and GPT-5 (2025), with GPT-5 answering in “Auto” mode, and reports per-domain and total AGI scores.

Limitations

  • The paper states that its conceptualization of intelligence is not exhaustive and deliberately excludes faculties such as kinesthetic abilities from Gardner’s multiple intelligences.
  • The paper states that its illustrative examples are English-specific and not culturally agnostic, and that General Knowledge tests are necessarily selective.
  • The paper states that a 100% AGI score represents a “highly proficient” well-educated individual rather than someone with a college degree, that equal weighting is one of many possible schemes, and that the aggregate AGI Score can be misleading because it obscures bottleneck failures; it recommends reporting the cognitive profile rather than only the total score.
  • The text also makes evident that scoring depends on manual judgment and illustrative examples, and the authors note contamination risks from training on similar test data, recommending robustness checks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.18212

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 25 cited sources checked · 0 supported · 0 flagged

  • Findingnone given
    § Abstract · p. 1

    Current AI systems are proficient in knowledge-intensive domains.

    “While proficient in knowledge-intensive domains, current AI systems have critical deficits in foundational cognitive machinery, particularly long-term memory storage.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract · p. 1

    GPT-5 achieved an AGI Score of 57%.

    “The resulting AGI scores (e.g., GPT-4 at 27%, GPT-5 at 57%)”
    • Also stated in: § AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult. · p. 3 · plain
      “| GPT-5 | 9% | 10% | 10% | 7% | 4% | 0% | 4% | 4% | 6% | 3% | 57% |”
    Quote in the paperSubstantial claim
  • Findingnone given
    § Speed (S) · p. 13

    GPT-5's multimodal capabilities are slow.

    “Moreover, several of these speed tests require multimodal capabilities, but GPT-5's multimodal capabilities are slow.”
    Quote in the paperNot a substantial claim
  • Findingnone given
    § General Knowledge (K) · p. 7

    GPT-4 has substantial general knowledge.

    “GPT-4 has substantial general knowledge, and GPT-5 partially fills in its remaining gaps.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Visual Processing (V) · p. 12

    GPT-5 has appreciable but highly incomplete visual processing capabilities.

    “GPT-4 had no ability to perceive or generate images, while GPT-5 has appreciable but highly incomplete visual processing capabilities.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim. 96
  • It may state two separate claims as one. 27

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given
    § Speed (S) · p. 13

    Several of the speed tests require multimodal capabilities.

    “Moreover, several of these speed tests require multimodal capabilities, but GPT-5's multimodal capabilities are slow.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Findingnone given
    § 9 Long-Term Memory Retrieval (MR) · p. 10

    Both GPT-4 and GPT-5 can rapidly retrieve many concepts from their parameters.

    “Both GPT-4 and GPT-5 can rapidly retrieve many concepts from their parameters, but they both frequently hallucinate.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 11 Auditory Processing (A) · p. 12

    GPT-5's audio processing capabilities are appreciable but incomplete.

    “GPT-4 had no audio processing capabilities, while GPT-5's capabilities are appreciable but incomplete.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Findingnone given
    § Speed (S) · p. 13

    GPT-5 often requires a long time to answer in 'thinking' mode.

    “Note: GPT-5 often requires a long time to answer in 'thinking' mode.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Findingnone given
    § 13 Discussion · p. 13

    Long-term memory storage is perhaps the most significant bottleneck, scoring near 0% for current models.

    “Long-term memory storage is perhaps the most significant bottleneck, scoring near 0% for current models.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § Speed (S) · p. 13

    Both GPT-4 and GPT-5 can read and write and compute simple expressions quickly.

    “Both GPT-4 and GPT-5 can read and write and compute simple expressions quickly, but their other multimodal processing speed capabilities are nonexistent or slow, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.

    Findingnone given
    § 13 Discussion · p. 14

    Currently, several critical parts of the AI 'engine' are highly defective.

    “Currently, several critical parts of the AI 'engine' are highly defective.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingown data
    § 7 Working Memory (WM) · p. 9

    While the raw Textual Working Memory score appears similar between GPT-4 and GPT-5 in this battery, improvements in managing long contexts are also reflected in the Document Level Reading Comprehension score within the Reading and Writing (RW) ability.

    “While the raw Textual Working Memory score appears similar between GPT-4 and GPT-5 in this battery, improvements in managing long contexts are also reflected in the Document Level Reading Comprehension score within the Reading and Writing (RW) ability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Findingnone given
    § 9 Long-Term Memory Retrieval (MR) · p. 10

    Both GPT-4 and GPT-5 frequently hallucinate.

    “Both GPT-4 and GPT-5 can rapidly retrieve many concepts from their parameters, but they both frequently hallucinate.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 8 Long-Term Memory Storage (MS) · p. 10

    Both GPT-4 and GPT-5 lack appreciable long-term memory storage capabilities.

    “Both GPT-4 and GPT-5 lack appreciable long-term memory storage capabilities.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingnone given
    § Speed (S) · p. 13

    GPT-4 and GPT-5's other multimodal processing speed capabilities are nonexistent or slow.

    “Both GPT-4 and GPT-5 can read and write and compute simple expressions quickly, but their other multimodal processing speed capabilities are nonexistent or slow, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Findingnone given
    § 13 Discussion · p. 13

    GPT-4 has an estimated AGI score of 27% and GPT-5 has an estimated AGI score of 57%.

    “The estimated AGI scores (e.g., GPT-4 at 27%, GPT-5 at 57%) illustrate both the rapid progress in the field and the substantial gap remaining before achieving human-level general intelligence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Findingnone given
    § 6 On-the-Spot Reasoning (R) · p. 8

    GPT-4 has negligible on-the-spot reasoning capabilities.

    “GPT-4 has negligible on-the-spot reasoning capabilities, while GPT-5 only has some remaining gaps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingnone given
    § 13 Discussion · p. 15

    Our illustrative examples are specific to the English language and are not culturally agnostic.

    “Second, our illustrative examples are specific to the English language and are not culturally agnostic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Findingnone given
    § 13 Discussion · p. 15

    The batteries in the operationalization have varying levels of precision.

    “The batteries in the operationalization have varying levels of precision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 14

    These abilities are deeply interdependent.

    “While this framework dissects intelligence into ten distinct axes for measurement, it is crucial to recognize that these abilities are deeply interdependent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 4 Reading and Writing Ability (RW) · p. 7

    GPT-4's difficulty with token-level understanding, its small context window, and its imprecise working memory limit its ability to analyze substrings of words, to read long documents, and to carefully proofread text.

    “GPT-4's difficulty with token-level understanding, its small context window, and its imprecise working memory limit its ability to analyze substrings of words, to read long documents, and to carefully proofread text.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § Abstract · p. 1

    The AGI scores concretely quantify both rapid progress and the substantial gap remaining before AGI.

    “The resulting AGI scores (e.g., GPT-4 at 27%, GPT-5 at 57%) concretely quantify both rapid progress and the substantial gap remaining before AGI.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 13 Discussion · p. 13

    The framework provides a structured, quantifiable methodology for evaluating AGI, moving beyond narrow, specialized benchmarks to assess the breadth (versatility) and depth (proficiency) of cognitive capabilities.

    “This framework provides a structured, quantifiable methodology for evaluating Artificial General Intelligence (AGI), moving beyond narrow, specialized benchmarks to assess the breadth (versatility) and depth (proficiency) of cognitive capabilities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § 13 Discussion · p. 14

    This severely limits the overall 'horsepower' of the system, regardless of how optimized other components might be.

    “This severely limits the overall 'horsepower' of the system, regardless of how optimized other components might be.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 4 Reading and Writing Ability (RW) · p. 7

    GPT-5 addresses GPT-4's issues with token-level understanding, context window, and working memory.

    “GPT-5 addresses these issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Complex cognitive tasks rarely utilize a single domain in isolation.

    “Complex cognitive tasks rarely utilize a single domain in isolation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § 13 Discussion · p. 13

    The reliance on RAG is a capability contortion that obscures two distinct underlying weaknesses in an AI's memory.

    “However, this reliance on RAG is a capability contortion that obscures two distinct underlying weaknesses in an AI's memory.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Imprecision in Long-Term Memory Retrieval, manifesting as hallucinations or confabulation, is often mitigated by integrating external search tools (Retrieval-Augmented Generation).

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationreasoning
    § 13 Discussion · p. 14

    These contortions can also mislead people to assume that intelligence is too jagged to be understood systematically.

    “These contortions can also mislead people to assume that intelligence is too jagged to be understood systematically.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Interpersonal skills are represented across these broad abilities.

    “Interpersonal skills are represented across these broad abilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Interpretationreasoning
    § 13 Discussion · p. 13

    These workarounds mask underlying limitations and can create a brittle illusion of general capability.

    “These workarounds mask underlying limitations and can create a brittle illusion of general capability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationreasoning
    § 13 Discussion · p. 13

    The jagged profile of current AI capabilities often leads to 'capability contortions,' where strengths in certain areas are leveraged to compensate for profound weaknesses in others.

    “The jagged profile of current AI capabilities often leads to 'capability contortions,' where strengths in certain areas are leveraged to compensate for profound weaknesses in others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § 13 Discussion · p. 13

    The dependency on RAG can potentially become a fundamental liability for AGI, as it is not a substitute for the holistic, integrated memory required for genuine learning, personalization, and long-term contextual understanding.

    “This dependency can potentially become a fundamental liability for AGI, as it is not a substitute for the holistic, integrated memory required for genuine learning, personalization, and long-term contextual understanding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 13 Discussion · p. 13

    This reliance on long contexts is inefficient, computationally expensive, and can overload the system's attentional mechanisms.

    “However, this approach is inefficient, computationally expensive, and can overload the system's attentional mechanisms.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 14

    Theory of mind is tested in on-the-spot reasoning (R).

    “And theory of mind is tested in on-the-spot reasoning (R).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 13

    Deficits in visual reasoning limit the ability of AI agents to interact with complex digital environments.

    “Similarly, deficits in visual reasoning limit the ability of AI agents to interact with complex digital environments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 14

    Facial emotion recognition is necessary for proficiency in V's 'image captioning.'

    “Facial emotion recognition is necessary for proficiency in V's 'image captioning.'”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § 13 Discussion · p. 13

    RAG masks the absence of a dynamic, experiential memory-a persistent, updatable store for private interactions and evolving contexts over a long time scale.

    “Second, and more critically, it masks the absence of a dynamic, experiential memory-a persistent, updatable store for private interactions and evolving contexts in a long time scale.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Imprecision in Long-Term Memory Retrieval, manifesting as hallucinations or confabulation, is often mitigated by integrating external search tools (Retrieval-Augmented Generation).

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationreasoning
    § 13 Discussion · p. 13

    RAG compensates for the inability to reliably access the AI's vast but static parametric knowledge.

    “First, it compensates for the inability to reliably access the AI's vast but static parametric knowledge.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Imprecision in Long-Term Memory Retrieval, manifesting as hallucinations or confabulation, is often mitigated by integrating external search tools (Retrieval-Augmented Generation).

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Mistaking these contortions for genuine cognitive breadth can lead to inaccurate assessments of when AGI will arrive.

    “Mistaking these contortions for genuine cognitive breadth can lead to inaccurate assessments of when AGI will arrive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 14

    Cognitive empathy is captured in K's 'commonsense' narrow ability.

    “For example, cognitive empathy is captured in K's 'commonsense' narrow ability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationreasoning
    § 13 Discussion · p. 13

    Without the ability to continually learn, AI systems suffer from 'amnesia' which limits their utility, forcing the AI to re-learn context in every interaction.

    “Without the ability to continually learn, AI systems suffer from 'amnesia' which limits their utility, forcing the AI to re-learn context in every interaction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § 13 Discussion · p. 14

    An artificial mind, much like an engine, is ultimately constrained by its weakest components.

    “An artificial mind, much like an engine, is ultimately constrained by its weakest components.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationreasoning
    § 13 Discussion · p. 13

    The long-context approach ultimately fails to scale for tasks requiring days or weeks of accumulated context.

    “It ultimately fails to scale for tasks requiring days or weeks of accumulated context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationreasoning
    § 13 Discussion · p. 13

    A prominent contortion is the reliance on massive context windows (Working Memory, WM) to compensate for the lack of Long-Term Memory Storage (MS).

    “A prominent contortion is the reliance on massive context windows (Working Memory, WM) to compensate for the lack of Long-Term Memory Storage (MS).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Practitioners use long context windows to manage state and absorb information, such as entire codebases.

  • Probably not a substantial claim.

    Premisecitation
    § AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult. · p. 2

    CHC theory breaks down general intelligence into distinct broad abilities and numerous narrow abilities.

    “It breaks down general intelligence into distinct broad abilities and numerous narrow abilities (such as induction, associative memory, or spatial scanning).”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult. · p. 2

    Current AIs are narrower than well-educated humans overall but far smarter on some specific tasks.

    “Current AIs are narrower than well-educated humans overall but far smarter on some specific tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 13 Discussion · p. 13

    By operationalizing AGI through ten core cognitive domains inspired by the CHC theory, the framework can systematically diagnose the strengths and profound weaknesses of current AI systems.

    “By operationalizing AGI through ten core cognitive domains inspired by the CHC theory, we can systematically diagnose the strengths and profound weaknesses of current AI systems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 13

    RAG's core function remains retrieving facts from a database.

    “While RAG can be adapted for private documents, its core function remains retrieving facts from a database.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 23

    A score of 5 on both the AP Physics 1 and Physics 2 is sufficient for the 1% AGI score in physics, subject to memorization and robustness checks.

    “A score of 5 on both the AP Physics 1 and Physics 2 is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 15

    The aggregate AGI Score could be misleading.

    “Finally, while the aggregate AGI Score is provided for convenience, it could be misleading.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    Turing (1950) argues that the Turing Test can indicate general ability.

    “Turing (1950) argues that the Turing Test can indicate general ability.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Turing (1950)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    Legg and Hutter (2007) discuss various tests for general machine intelligence.

    “Legg and Hutter (2007) discuss various tests for general machine intelligence.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Legg and Hutter (2007)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 15

    More discretionary weighting schemes could be reasonable.

    “We give equal weight to each broad ability (10%) to prioritize breadth, but more discretionary weighting schemes could be reasonable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 25

    A score of 5 on the AP US History test is sufficient for the 1% AGI score in US history, subject to memorization and robustness checks.

    “A score of 5 on the AP US History test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Solving advanced mathematical problems requires both Mathematical Ability (M) and On-the-Spot Reasoning (R).

    “For example, solving advanced mathematical problems requires both Mathematical Ability (M) and On-the-Spot Reasoning (R).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Interpretationnone given
    § 13 Discussion · p. 15

    Our conceptualization of intelligence is not exhaustive.

    “First, our conceptualization of intelligence is not exhaustive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Interpretationnone given
    § 13 Discussion · p. 16

    Achieving AGI requires solving a variety of grand challenges.

    “Achieving AGI requires solving a variety of grand challenges.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 25

    A score of 5 on the AP European History test is sufficient for the 1% AGI score in European history, subject to memorization and robustness checks.

    “A score of 5 on the AP European History test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    Ilić and Gignac (2024) and Ren et al. (2024) find that a variety of AI systems' capabilities are highly correlated with pre-training compute.

    “Ili´ c and Gignac (2024) and Ren et al. (2024) find that a variety of AI systems' capabilities are highly correlated with pre-training compute.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Ili´ c and Gignac (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ren et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 23

    A score of 5 on the AP Biology test is sufficient for the 1% AGI score in biology, subject to memorization and robustness checks.

    “A score of 5 on the AP Biology test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationcitation
    § 13 Discussion · p. 16

    Recursive AI could potentially lead to Superintelligence.

    “Recursive AI removes the need for human researchers and 'closes the loop' on AI R&D, enabling rapid, recursive capability gains (an 'intelligence recursion' (Hendrycks et al., 2025)) without human scientific input and could potentially lead to Superintelligence .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    Gubrud (1997) proposed an early definition of AGI in 1997.

    “Gubrud (1997) proposed an early definition of AGI in 1997.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gubrud (1997)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation
    § Illustrative Examples: · p. 52

    On the LibriSpeech test-clean benchmark, the human-level word error rate threshold is 5.83%, so a model must achieve a WER below 5.83% for clean audio to be at human level.

    “LibriSpeech (Panayotov et al., 2015) (test-clean) WER must be less than 5.83% (humanlevel) for the clean audio 2%.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Panayotov et al., 2015

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Various batteries of narrow abilities test cognitive abilities in combination, reflecting the integrated nature of general intelligence.

    “Consequently, various batteries of narrow abilities test cognitive abilities in combination, reflecting the integrated nature of general intelligence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationreasoning
    § 13 Discussion · p. 15

    An AI system with a 90% AGI Score but 0% on Long-Term Memory Storage (MS) would be functionally impaired by a form of 'amnesia,' severely limiting its capabilities despite a high overall score.

    “For example, an AI system with a 90% AGI Score but 0% on Long-Term Memory Storage (MS) would be functionally impaired by a form of 'amnesia,' severely limiting its capabilities despite a high overall score.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Theory of Mind questions require On-the-Spot Reasoning (R) as well as General Knowledge (K).

    “Theory of Mind questions require On-the-Spot Reasoning (R) as well as General Knowledge (K).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 13 Discussion · p. 15

    A simple summation can obscure critical failures in bottleneck capabilities.

    “A simple summation can obscure critical failures in bottleneck capabilities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Since the definition is couched in a collection of tasks rather than depending on specific existing datasets, AI systems can be tested using the best available tests at the time.

    “Since we couch our definition in a collection of tasks rather than heavily depend on specific existing datasets, we can test AI systems using the best available tests at the time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationreasoning
    § 13 Discussion · p. 15

    Replacement AI is about economy-level AI, and it includes physical tasks, unlike AGI.

    “Meanwhile, Replacement AI is about economy-level AI, and it includes physical tasks, unlike AGI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Weight: 4% · p. 43

    External tools (e.g., internet search) must be disabled for hallucination tests to assess the model's internal knowledge retrieval precision.

    “Testing Note: External tools (e.g., internet search) must be disabled for these tests to assess the model's internal knowledge retrieval precision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 25

    A score of 5 on the AP World History test is sufficient for the 1% AGI score in world history, subject to memorization and robustness checks.

    “A score of 5 on the AP World History test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 24

    A score of 5 on the AP Comparative Government and Politics test is sufficient for the 1% AGI score in comparative government, subject to memorization and robustness checks.

    “A score of 5 on the AP Comparative Government and Politics test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 25

    A score of 5 on the AP Art History test is sufficient for the 1% AGI score in art history, subject to memorization and robustness checks.

    “A score of 5 on the AP Art History test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § Weight: 4% · p. 44

    Image anomaly detection is highly related to the 'Odd-Item Out' and the 'What's Missing (WHM)' RIAS-2 subtests.

    “Note: Image anomaly detection is highly related to the 'Odd-Item Out' and the 'What's Missing (WHM)' RIAS-2 subtests (Reynolds and Kamphaus, 2015).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Reynolds and Kamphaus, 2015

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Weight: 4% · p. 44

    Image anomaly detection questions should not be reasoning intensive.

    “These questions should not be reasoning intensive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 23

    A score of 5 on the AP Chemistry test is sufficient for the 1% AGI score in chemistry, subject to memorization and robustness checks.

    “A score of 5 on the AP Chemistry test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Interpretationreasoning
    § 13 Discussion · p. 15

    Our AGI definition is about human-level AI, not economically-valuable AI, nor economy-level AI.

    “Our AGI definition is about human-level AI, not economically-valuable AI, nor economy-level AI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Study design and conductreasoning
    § Weight: 10% · p. 38

    External tools (e.g., internet search) must be disabled during Long-Term Memory Storage tests.

    “External tools (e.g., internet search) must be disabled.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    Morris et al. (2023) articulate levels of AGI based on performance percentiles.

    “Morris et al. (2023) articulate levels of AGI based on performance percentiles.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Morris et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim.

    Study design and conductreasoning
    § Weight: 10% · p. 38

    To ensure testing long-term storage rather than working memory, all tests in the Long-Term Memory Storage section must be conducted in a new session separate from the initial presentation of information.

    “To ensure we are testing long-term storage rather than working memory, all tests in this section must be conducted in a new session separate from the initial presentation of information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 24

    A score of 5 on the AP Human Geography test is sufficient for the 1% AGI score in geography, subject to memorization and robustness checks.

    “A score of 5 on the AP Human Geography test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 16

    The challenge of spatial navigation memory (WM) reflects a core goal of Fei-Fei Li's startup, World-Labs.

    “The challenge of spatial navigation memory (WM) reflects a core goal of Fei-Fei Li's startup, World-Labs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation
    § 13 Discussion · p. 15

    Gignac and Szodorai (2024) discuss human psychometrics and testing the intelligence of AI systems.

    “Gignac and Szodorai (2024) discuss human psychometrics and testing the intelligence of AI systems.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gignac and Szodorai (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationreasoning
    § 13 Discussion · p. 15

    Different people could issue their own estimates of the AGI score, and people can decide whether they find the grader's judgment reasonable.

    “Consequently, different people could issue their own estimates of the AGI score, and people can decide whether they find the grader's judgment reasonable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationreasoning
    § 13 Discussion · p. 16

    The challenges of hallucinations (MR) and continual learning (MS) will also need to be resolved.

    “Moreover, the challenges of hallucinations (MR) and continual learning (MS) will also need to be resolved.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationcitation
    § 13 Discussion · p. 16

    Recursive AI removes the need for human researchers and 'closes the loop' on AI R&D, enabling rapid, recursive capability gains without human scientific input.

    “Recursive AI removes the need for human researchers and 'closes the loop' on AI R&D, enabling rapid, recursive capability gains (an 'intelligence recursion' (Hendrycks et al., 2025)) without human scientific input and could potentially lead to Superintelligence .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hendrycks et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationreasoning
    § 13 Discussion · p. 15

    We do not conflate AGI with economically valuable AI because narrow technologies, such as the iPhone, can generate billions in economic value, despite not being generally intelligent.

    “We do not conflate AGI with economically valuable AI because narrow technologies, such as the iPhone, can generate billions in economic value, despite not being generally intelligent.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Weight: 6% · p. 41

    Fluency is measured by comparing the AI's performance on tasks against human performance.

    “Fluency is measured by comparing the AI's performance on tasks (e.g., quantity and originality of responses within a time limit, typically 60 seconds) against human performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 16

    The machine learning community's ARC-AGI Challenge aiming to measure abstract reasoning is represented in On-the-Spot Reasoning (R) tasks.

    “For example, the machine learning community's ARC-AGI Challenge aiming to measure abstract reasoning is represented in On-the-Spot Reasoning (R) tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation
    § Illustrative Examples: · p. 35

    The Michelangelo benchmark is substantially harder than the paper's textual working memory transformation sequence test because Michelangelo uses very long sequences, whereas the paper uses shorter sequences.

    “Test: A related benchmark is Michelangelo (Vodrahalli et al., 2024), but it is substantially harder since it uses very long sequences, whereas we use shorter sequences.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Vodrahalli et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 13 Discussion · p. 15

    The General Knowledge (K) tests are necessarily selective and do not assess the full breadth of possible subject areas.

    “The General Knowledge (K) tests are necessarily selective and do not assess the full breadth of possible subject areas.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Image recognition involves Visual Processing (V) and General Knowledge (K).

    “Image recognition involves Visual Processing (V) and General Knowledge (K).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    Marcus et al. (2016) discuss the need to move beyond the Turing Test to capture the multidimensional nature of intelligence.

    “Marcus et al. (2016) discuss the need to move beyond the Turing Test to capture the multidimensional nature of intelligence.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Marcus et al. (2016)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Solving the illustrative examples does not imply the task is solved, as the collection of examples is not exhaustive.

    “Moreover, solving our illustrative examples do not imply the task is solved, as our collection of examples are not exhaustive.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 15

    The conceptualization deliberately excludes certain faculties, such as the kinesthetic abilities proposed in Gardner's theory of multiple intelligences.

    “It deliberately excludes certain faculties, such as the kinesthetic abilities proposed in alternative frameworks like Gardner's theory of multiple intelligences (Gardner, 1993).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gardner, 1993

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 16

    Meta's attempts to create world models that include intuitive physics understanding is represented in the video anomaly detection task (V).

    “Meta's attempts to create world models that include intuitive physics understanding is represented in the video anomaly detection task (V).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    Jin et al. (2025) connect factor analysis of human cognition to AI capabilities.

    “Jin et al. (2025) connect factor analysis of human cognition to AI capabilities.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jin et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 13 Discussion · p. 15

    The scoring weights represent one of many possible configurations.

    “Moreover, while the scoring weights we employ are necessary for quantitative measurement, they represent one of many possible configurations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 24

    A score of 5 on the AP Macroeconomics test is sufficient for the 1% AGI score in macroeconomics, subject to memorization and robustness checks.

    “A score of 5 on the AP Macroeconomics test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationreasoning
    § 13 Discussion · p. 14

    Understanding a movie requires the integration of Auditory Processing (A), Visual Processing (V), and Working Memory (WM).

    “Understanding a movie requires the integration of Auditory Processing (A), Visual Processing (V), and Working Memory (WM).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.

    Interpretationnone given
    § 13 Discussion · p. 15

    The descriptions and examples should be clear enough that people can grade the AI systems themselves.

    “However, the descriptions and examples should be clear enough that people can grade the AI systems themselves.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 13 Discussion · p. 15

    The results are contingent on these methodological choices.

    “The results are contingent on these methodological choices, and future work could explore alternative collections of tasks and weighting schemes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 24

    A score of 5 on the AP Microeconomics test is sufficient for the 1% AGI score in microeconomics, subject to memorization and robustness checks.

    “A score of 5 on the AP Microeconomics test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation
    § Illustrative Examples: · p. 52

    On the LibriSpeech test-other benchmark, the human-level word error rate threshold is 12.69%, so a model must achieve a WER below 12.69% for noisy audio to be at human level.

    “LibriSpeech (Panayotov et al., 2015) (test-other) WER must be less than 12.69% (humanlevel) for the noisy audio additional 2%.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Panayotov et al., 2015

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § Illustrative Examples: · p. 24

    A score of 5 on the AP Psychology test is sufficient for the 1% AGI score in psychology, subject to memorization and robustness checks.

    “A score of 5 on the AP Psychology test is sufficient for the 1%, subject to memorization and robustness checks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 13 Discussion · p. 15

    OpenAI and Microsoft have reportedly considered AGI to be an AI that can generate $100 billion in profit.

    “OpenAI and Microsoft have reportedly considered AGI to be an AI that can generate $100 billion in profit (TechCrunch, 2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites TechCrunch, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

102 of 102

  • The paper relies on this without support.

    Humans are the only existing example of general intelligence.

    “To operationalize this definition, we must look to the only existing example of general intelligence: humans.”

  • The paper relies on this without support.

    Human cognition is composed of many distinct abilities rather than being a monolithic capability.

    “Human cognition is not a monolithic capability; it is a complex architecture composed of many distinct abilities honed by evolution.”

  • The paper relies on this without support.

    Decades of psychometric research have yielded a vast battery of tests specifically designed to isolate and measure distinct cognitive components in individuals.

    “Decades of psychometric research have yielded a vast battery of tests specifically designed to isolate and measure these distinct cognitive components in individuals.”

  • The paper relies on this without support.

    AI corporations sometimes train on data highly similar to or identical to target tests, inflating their reported numbers.

    “Sometimes AI corporations 'juice' their numbers by training on data highly similar to or identical to target tests.”

  • The paper relies on this without support.

    Practitioners use long context windows to manage state and absorb information, such as entire codebases.

    “Practitioners use these long contexts to manage state and absorb information (e.g., entire codebases).”

  • The paper relies on this without support.

    Imprecision in Long-Term Memory Retrieval, manifesting as hallucinations or confabulation, is often mitigated by integrating external search tools (Retrieval-Augmented Generation).

    “Imprecision in Long-Term Memory Retrieval (MR)-manifesting as hallucinations or confabulation-is often mitigated by integrating external search tools, a process known as Retrieval-Augmented Generation (RAG).”

  • The paper relies on this without support.

    A score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.

    “For context, a score of 5 on AP exams often corresponds to approximately 80th percentile or better among test-takers.”

  • The paper relies on this without support.

    The correlation between on-the-spot reasoning tests and performance on other cognitive tests does not necessarily hold for AI systems.

    “While on-the-spot reasoning tests (often termed fluid intelligence) are strong predictors of human performance on other cognitive tests, this correlation does not necessarily hold for AI systems.”

  • The paper relies on this without support.

    Cross-modal working memory is highly related to the narrow CHC ability 'Working Memory Capacity (Wc).'

    “Note: This is highly related to the narrow CHC ability 'Working Memory Capacity (Wc). '”

  • The paper relies on this without support.

    Long-term memory storage is highly related to the broad CHC ability 'Long-term storage (Gl).'

    “Note: This is highly related to the broad CHC ability 'Long-term storage (Gl). '”

  • The paper relies on this without support.

    Associative memory is highly related to the narrow CHC ability 'Associative memory (MA).'

    “Note: This is highly related to the narrow CHC ability 'Associative memory (MA). '”

  • The paper relies on this without support.

    Meaningful memory is highly related to the narrow CHC ability 'Meaningful memory (MM).'

    “Note: This is highly related to the narrow CHC ability 'Meaningful memory (MM). '”

  • The paper relies on this without support.

    Short sequence recall is highly related to the narrow CHC ability 'Free-recall memory (M6).'

    “Note: This is highly related to the narrow CHC ability 'Free-recall memory (M6). '”

  • The paper relies on this without support.

    Set recall is highly related to the narrow CHC ability 'Free-recall memory (M6).'

    “Note: This is highly related to the narrow CHC ability 'Free-recall memory (M6). '”

  • The paper relies on this without support.

    Long-term memory retrieval is highly related to the broad CHC ability 'Retrieval Fluency (Gr).'

    “Note: This is highly related to the broad CHC ability 'Retrieval Fluency (Gr). '”

  • The paper relies on this without support.

    Ideational fluency is highly related to the narrow CHC ability 'Ideational fluency (FI).'

    “Note: This is highly related to the narrow CHC ability 'Ideational fluency (FI). '”

  • The paper relies on this without support.

    Expressional fluency is highly related to the narrow CHC ability 'Expressional fluency (FE).'

    “Note: This is highly related to the narrow CHC ability 'Expressional fluency (FE). '”

  • The paper relies on this without support.

    Alternative solution fluency is highly related to the narrow CHC ability 'Alternative solution fluency (SP).'

    “Note: This is highly related to the narrow CHC ability 'Alternative solution fluency (SP). '”

  • The authors state this as an assumption.

    The ability to translate abstract concepts into novel visual information is a critical component of general intelligence in the modern era.

    “we believe the ability to translate abstract concepts into novel visual information is a critical component of general intelligence in the modern era.”
19 of 19

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • High accuracy on HLE would not alone suggest autonomous research capabilities or artificial general intelligence.

    “but it would not alone suggest autonomous research capabilities or artificial general intelligence11.”

    From Humanity's Last Exam (2025) · cites this paper as [11]

    On p. 2, the paper says impressive performance on complex benchmarks can coexist with missing capabilities essential to AGI. The Discussion (p. 15) separately defines Recursive AI as independently conducting the entire AI R&D lifecycle. These passages support the caution, but the paper does not specifically assess whether high accuracy on HLE indicates autonomous research capability. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • This anthropomorphic view inherently influences the research ecosystem across how to evaluate LLM capabilities, design LLM-based agent, define AGI, and conduct ethical deliberations toward LLM.

    “This anthropomorphic view inherently influences the research ecosystem across how to evaluate LLM capabilities, design LLM-based agent, define Artificial General Intelligence65, and conduct ethical deliberations toward LLM66.”

    From Understanding large language models demands distinguishing human projection from machine cognition (2026) · cites this paper as [65]

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 16

  1. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, Oxford, 2014. ISBN 9780199678112.
  2. John B. Carroll. Human Cognitive Abilities: A Survey of Factor-Analytic Studies . Cambridge University Press, 1993.
  3. François Chollet. On the measure of intelligence, 2019. URL https://arxiv.org/abs/1911. 01547 .
  4. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values, 2023. URL https://arxiv.org/abs/2008. 02275 .
  5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https: //arxiv.org/abs/2106.09685 .
  6. Arthur R. Jensen. The g factor: psychometrics and biology. Novartis Foundation symposium , 233:3747; discussion 47-57, 116-21, 2000. URL https://api.semanticscholar.org/CorpusID: 17202073 .
  7. Jikai Jin, Vasilis Syrgkanis, Sham M. Kakade, and Hanlin Zhang. Discovering hierarchical latent capabilities of language models via causal representation learning. 2025. URL https://arxiv. org/abs/2506.10378 .
  8. Kevin S. McGrew. Carroll's three-stratum (3s) cognitive ability theory at 30 years: Impact, 3schc theory clarification, structural replication, and cognitive-achievement psychometric network analysis extension. Journal of Intelligence , 11(2), 2023. ISSN 2079-3200. URL https://www. mdpi.com/2079-3200/11/2/32 .
  9. Kevin S. McGrew and W. Joel Schneider. CHC theory revised: A visual-graphic summary of Schneider and McGrew's 2018 CHC update chapter. MindHub / IAPsych working paper, 2018. URL http://www.iapsych.com/mindhubpub4.pdf .
  10. Jim Pitman. Probability . Springer Texts in Statistics. Springer, 1993.
  11. Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models?, 2025. URL https://arxiv.org/abs/2410. 06468 .
  12. Cecil R. Reynolds and Randy W. Kamphaus. Reynolds Intellectual Assessment Scales, Second Edition (RIAS-2) and Reynolds Intellectual Screening Test, Second Edition (RIST-2): Professional Manual . Lutz, FL, 2015.
  13. W Joel Schneider and Kevin S McGrew. The cattell-horn-carroll theory of cognitive abilities. In Dawn P Flanagan and Erin M McDonough, editors, Contemporary intellectual assessment: Theories, tests, and issues , pages 73-163. Guilford Press, 4th edition, 2018.
  14. Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazaridou, Jean-Baptiste Lespiau, Nithya Attaluri, and Kate Olszewska. Michelangelo: Long context evaluations beyond haystacks via latent structure queries, 2024. URL https: //arxiv.org/abs/2409.12640 .
  15. Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Manling Li, Jiajun Wu, and Li Fei-Fei. Spatial mental modeling from limited views, 2025. URL https://arxiv. org/abs/2506.21458 .
  16. Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. Learning and evaluating general linguistic intelligence, 2019. URL https://arxiv.org/abs/1901. 11373 .

Cited by

Sorted by citations

Related

Metadata

Authors
Hendrycks, D., Song, D., Szegedy, C., Lee, H., Gal, Y., Brynjolfsson, E., Li, S., Zou, A., Levine, L., Han, B., Fu, J., Liu, Z., Shin, J., Lee, K., Mazeika, M., Phan, L., Ingebretsen, G., Khoja, A., Xie, C., Salaudeen, O., Hein, M., Zhao, K., Pan, A., Duvenaud, D., Li, B., Omohundro, S., Alfour, G., Tegmark, M., McGrew, K., Marcus, G., Tallinn, J., Schmidt, E., & Bengio, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2510.18212
arXiv
2510.18212
OpenAlex
W4415967303
Cited by
3
References
0
Citations by year
2026: 3
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
Keywords
artificial general intelligence, Cattell-Horn-Carroll theory, human cognition, cognitive domain, long-term memory, GPT-4, GPT-5, reasoning, memory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review