CoChat
Sign Up
PreprintCited by 27Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Training Large Language Models to Reason in a Continuous Latent Space

Open access
Hao, Shibo, Sukhbaatar, Sainbayar, Su, DiJia, Li, Xian, Hu, Zhiting, Weston, Jason, Tian, Yuandong·2024·arXiv
DOI 10.48550/arxiv.2412.06769arXiv 2412.06769
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are typically constrained to reason in the language space. Coconut uses the last hidden state of the LLM as a representation of reasoning state and feeds it back directly in continuous space, enabling better accuracy-efficiency trade-offs.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces Coconut, a method that lets a language model reason by feeding its last hidden state back as the next input embedding instead of decoding it into a word token. This continuous latent reasoning can represent multiple possible next steps and behaves like breadth-first search, improving accuracy on some planning-heavy logical reasoning tasks while using fewer generated tokens.

Key findings

  • Coconut outperformed language-space chain-of-thought on the logical reasoning benchmarks ProntoQA and ProsQA: 99.8% vs. 98.8% accuracy on ProntoQA, and 97.0% vs. 77.5% on ProsQA, while generating far fewer tokens (9.0 vs. 92.5 on ProntoQA; 14.2 vs. 49.4 on ProsQA).
  • On GSM8k math reasoning, Coconut reached 34.1% accuracy versus 16.5% for no-CoT, though it did not surpass CoT’s 42.9%; it used 8.2 tokens versus CoT’s 25.0.
  • The multi-stage curriculum was important: without it, GSM8k accuracy fell to 14.4%, and without continuous thoughts it was 21.6%, compared with 34.1% for full Coconut.
  • Probing latent thoughts showed non-greedy search behavior: in one example, the model initially assigned the highest value to “lempus” (0.33) but later assigned the highest value to “rorpus” (0.87), consistent with breadth-first exploration rather than immediate commitment.

Methods

  • The main experiments used GPT-2. Coconut alternates between language mode and latent mode using special `<bot>` and `<eot>` tokens; in latent mode, the previous hidden state replaces the next input embedding, and training uses a multi-stage curriculum that replaces early language reasoning steps with continuous thoughts.
  • Datasets included GSM8k for math, ProntoQA, and a new ProsQA logical reasoning dataset built from directed acyclic graphs; ProsQA averaged 23.0 nodes, 36.0 edges, a shortest-path length of 3.8, and 1.6 shortest paths.
  • Evaluation compared accuracy and generated-token counts against CoT, no-CoT, iCoT, pause tokens, and Coconut variants; additional analysis probed latent thoughts by forcing language output after continuous thoughts and treating predicted concept probabilities as value estimates.

Limitations

  • The paper states that Coconut’s sequential forward passes limit parallelism and that improving training efficiency remains future work.
  • The paper reports that larger Llama models showed smaller gains than GPT-2, possibly because extensive language-focused pretraining makes latent reasoning harder, and that training without the curriculum performed no better than no-CoT.
  • The paper notes that universally surpassing language-based CoT likely requires substantial work on latent-space pretraining, and that better general strategies are needed for learning latent reasoning without supervision from language reasoning chains.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.06769

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

14 of 21 cited sources checked · 9 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Neuroimaging studies have consistently shown that the language network remains largely inactive during various reasoning tasks.

    “Neuroimaging studies have consistently shown that the language network -a set of brain regions responsible for language comprehension and production -remains largely inactive during various reasoning tasks”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Amalric and Dehaene, 2019 source

      The Discussion reports no activation in some language regions during spoken algebra problems, supporting a limited version of the claim. But the General Discussion says mathematical statements activated core language areas in proportion to syntactic complexity. The paper does not establish that the language network remains largely inactive across various reasoning tasks. No page numbers are available in our copy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Monti et al., 2012 source
    • cites Fedorenko et al., 2011 source

      The Results report little response in language-sensitive regions during the seven nonlinguistic tasks tested, including arithmetic, working memory, and cognitive control. But the Abstract says prior neuroimaging work had suggested overlap between linguistic and nonlinguistic activations, so this paper does not support the claim that neuroimaging studies have consistently shown inactivity across reasoning tasks. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisenone given
    § Nodes # Edges Len. of Shortest Path # Shortest Paths 23.0 36.0 3.8 1.6

    The latent spaces have not yet been explicitly optimized for reasoning.

    “While these recent models provide scalable methods for latent representation learning, the latent spaces have not yet been explicitly optimized for reasoning.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Geiping et al., 2025 source

      Section 2 (p. 2) says the proposed model can learn latent reasoning using standard training data without specialized demonstrations, which supports the narrower point that it does not require reasoning-specific training examples. But Section 4.1 (p. 4) says its data mixture was chosen to encourage reasoning, and the paper does not establish that recent models’ latent spaces generally have not been explicitly optimized for reasoning. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Barrault et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Gladstone et al., 2025 source
  • Premisecitation

    CoT's autoregressive generation nature makes it challenging to mimic human reasoning on more complex problems that typically require planning and search.

    “While CoT has proven effective for certain tasks, its autoregressive generation nature makes it challenging to mimic human reasoning on more complex problems (LeCun, 2022;Hao et al., 2023), which typically require planning and search.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites LeCun, 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hao et al., 2023 source
  • Findingown data

    Coconut surpasses the iCoT baseline on GSM8k.

    “Particularly, it surpasses the latest baseline iCoT (Deng et al., 2024), which requires a more carefully designed training schedule.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Deng et al., 2024 source
  • Findingown data

    Breaking down the learning of continuous thoughts into multiple stages, inspired by iCoT, is very beneficial for training.

    “These training methods can be combined to our framework, and specifically, we find that breaking down the learning of continuous thoughts into multiple stages, inspired by iCoT (Deng et al., 2024), is very beneficial for the training.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Deng et al., 2024 source
  • May combine two claims. 19
  • Classified as probably not a substantial claim. 8

  • May combine two claims.

    Interpretationreasoning

    Previous attempts to fix language-space reasoning by prompting succinct reasoning chains or performing additional reasoning before critical tokens remain constrained within the language space and do not solve the fundamental problems.

    “While previous work has attempted to fix these problems by prompting LLMs to generate succinct reasoning chains (Madaan and Yazdanbakhsh, 2022), or performing additional reasoning before generating some critical tokens (Zelikman et al., 2024), these solutions remain constrained within the language space and do not solve the fundamental problems.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Madaan and Yazdanbakhsh, 2022 source
    • cites Zelikman et al., 2024 source

  • May combine two claims.

    Findingcitation

    For math reasoning on GSM8k, using continuous thoughts is beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.

    “For math reasoning (GSM8k, Cobbe et al., 2021), using continuous thoughts is shown to be beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Cobbe et al., 2021 source

  • May combine two claims.

    Interpretationcitation

    This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner.

    “This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Yao et al., 2023 source

      The Abstract reports that Tree of Thoughts outperformed chain-of-thought prompting on the studied tasks, and the method section says it needs no extra training. But the Thought generator section describes explicit CoT and proposal prompts, so the claim that the model is not instructed to operate this way is not supported. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Hao et al., 2023 source

      The Results in §4.3 and the Conclusion report that RAP outperforms CoT on the evaluated reasoning tasks. But §3 and §3.1 say RAP repurposes the model through prompting, including a prompt that guides it to generate world states. The claim that the model is not explicitly instructed to operate this way is therefore not supported. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent theoretical analyses have demonstrated the usefulness of CoT from the perspective of model expressivity.

    “Recent theoretical analyses have demonstrated the usefulness of CoT from the perspective of model expressivity”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Feng et al., 2023 source
    • cites Merrill and Sabharwal, 2023 source
    • cites Li et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingnone given

    Continuous thoughts in Coconut can encode multiple potential next steps simultaneously, allowing a reasoning process akin to breadth-first search.

    “Unlike language-based reasoning, continuous thoughts in Coconut can encode multiple potential next steps simultaneously, allowing for a reasoning process akin to breadth-first search (BFS).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Once language-space models commit to an incorrect path, they either hallucinate unsupported edges or terminate with irrelevant conclusions.

    “Once they commit to an incorrect path, they either hallucinate unsupported edges or terminate with irrelevant conclusions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Although Coconut does not surpass CoT on GSM8k, it offers a superior trade-off between reasoning efficiency and accuracy.

    “Although Coconut does not surpass CoT on GSM8k, it offers a superior trade-off between reasoning efficiency and accuracy (Figure 8, I).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Compared to traditional CoT, Coconut generates fewer tokens while achieving higher accuracy on ProntoQA and ProsQA.

    “Compared to traditional CoT, Coconut generates fewer tokens while achieving higher accuracy on ProntoQA and ProsQA (Table 1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    By applying Coconut training strategy-replacing each language reasoning step with two continuous thoughts-the accuracy drop is substantially mitigated, maintaining higher performance even when fewer tokens are generated.

    “In contrast, by applying Coconut training strategy-replacing each language reasoning step with two continuous thoughts-the accuracy drop is substantially mitigated, maintaining higher performance even when fewer tokens are generated.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Coconut achieves 34.1% accuracy on GSM8k with 6 continuous thoughts, significantly outperforming No-CoT (16.5%).

    “For example, by using 6 continuous thoughts, Coconut achieves 34.1% accuracy on GSM8k, which significantly outperforms No-CoT (16.5%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model trained using CoT frequently hallucinates non-existent edges or outputs paths leading to incorrect targets, resulting in lower answer accuracy.

    “The model trained using CoT frequently hallucinates non-existent edges or outputs paths leading to incorrect targets, resulting in lower answer accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The reasoning strategy employed by Coconut is not greedy search: while lempus initially has the highest value 0.33, the model subsequently assigns the highest value 0.87 to rorpus rather than following lempus.

    “Interestingly, the reasoning strategy employed by Coconut is not greedy search: while “lempus” initially has the highest value (0.33) at the first reasoning step (Figure 5, left), the model subsequently assigns the highest value (0.87) to “rorpus,” a child of “grimpus,” rather than following “lempus” (Figure 5, right).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The multi-stage training also integrates well with pause tokens (Coconut-pause as thought).

    “The multi-stage training also integrates well with pause tokens (Coconut-pause as thought).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the second thoughts, the gaps narrow, suggesting the model transitions from parallel exploration to more focused reasoning in the second latent reasoning step.

    “In contrast, the second thoughts (right panel) show a narrowing of these gaps.This trend suggests that the model transitions from parallel exploration to more focused reasoning in the second latent reasoning step, likely as it gains more certainty about the most promising paths.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Large language models are restricted to reason in the language space, typically expressing the reasoning process with a chain-of-thought to solve a complex reasoning problem.

    “Large language models (LLMs) are restricted to reason in the “language space”, where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Continuous thoughts are fully differentiable, allowing the system to be optimized end-to-end by gradient descent.

    “This modification frees the reasoning from being within the language space, and the system can be optimized end-to-end by gradient descent, as continuous thoughts are fully differentiable.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    Current LLM architectures allocate nearly the same computing budget for predicting every token, while the amount of reasoning required for each token varies greatly.

    “A significant issue arises when LLMs use language for reasoning: the amount of reasoning required for each particular token varies greatly, yet current LLM architectures allocate nearly the same computing budget for predicting every token.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The amount of reasoning required for each particular token varies greatly.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    Reasoning in latent space gives rise to advanced emergent behaviors, where continuous thoughts can represent multiple alternative next steps.

    “Reasoning in latent space gives rise to advanced emergent behaviors, where continuous thoughts can represent multiple alternative next steps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    This characteristic resembles a breadth-first search approach, contrasting with greedy decoding typical of traditional CoT methods.

    “This characteristic resembles a breadth-first search (BFS) approach, contrasting sharply with the greedy decoding typical of traditional CoT methods.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Coconut has potential to enhance the model's expressivity with test-time scaling.

    “as well as its potential to enhance the model's expressivity with test-time scaling.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The finer-grained removal schedule (token by token) and a few other tricks in iCoT may ease the training process.

    “The finer-grained removal schedule (token by token) and a few other tricks in iCoT may ease the training process.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This flexibility allows the model to progressively eliminate incorrect options and converge on the correct answer, resulting in higher accuracy.

    “This flexibility allows the model to progressively eliminate incorrect options and converge on the correct answer, ultimately resulting in higher accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    GSM8k involves more complex contextual understanding and modeling, placing higher demands on computational capability.

    “In contrast, GSM8k involves more complex contextual understanding and modeling, placing higher demands on computational capability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    This enables the model to perform BFS over possible reasoning paths, rather than prematurely committing to a single deterministic trajectory as in language space CoT reasoning.

    “This enables the model to perform BFS over possible reasoning paths, rather than prematurely committing to a single deterministic trajectory as in language space CoT reasoning.”
    Quote in the paperNot a substantial claim
24 of 24

  • The paper relies on this without support.

    Most word tokens in a reasoning chain primarily ensure textual coherence and are not essential for reasoning.

    “Most tokens in a reasoning chain are generated solely for fluency, contributing little to the actual reasoning process.”

  • The paper relies on this without support.

    Some critical tokens require complex planning and pose huge challenges to LLMs.

    “By contrast, some critical tokens require complex planning and pose huge challenges to LLMs.”

  • The paper relies on this without support.

    The amount of reasoning required for each particular token varies greatly.

    “the amount of reasoning required for each particular token varies greatly”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Compressing tokens into latent representations improves both accuracy and efficiency by reducing the number of intermediate thinking tokens.

    “The results suggest that compressing tokens into latent representations improves both accuracy and efficiency by reducing the number of intermediate 'thinking' tokens.”

    From Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025) · cites this paper as [60]

  • Continuous chain-of-thought approaches propagate entire output activations across the chain, which could further increase the ability of chain-of-thought reasoning to implement backdoors with multi-agent threats.

    “While conventional chain-of-thought creates an information bottleneck thanks to propagating only single sampled outputs, continuous chain-of-thought approaches propagate entire output activations across the chain (Hao et al., 2024).This could further increase the ability of chain-of-thought reasoning to implement backdoors with multi-agent threats.”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Hao et al., 2024

  • Reasoning in a latent continuous space has demonstrated promising improvements in both accuracy and efficiency.

    “Reasoning in a latent continuous space has been increasingly explored in recent research (Xu et al., 2024; Hao et al., 2024; Pagnoni et al., 2024; Su et al., 2025; team et al., 2024; Tack et al., 2025), demonstrating promising improvements in both accuracy and efficiency.”

    From iCLP: Large Language Model Reasoning with Implicit Cognition Latent Planning (2025) · cites this paper as Hao et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Node height has two conflicting definitions

    Major

    Figure 5 defines a node's height using its longest distance to a leaf, while Section 4.4 defines it using the shortest distance. These definitions can assign different heights to the same node, so the reported height-based analyses do not have a consistent stated metric.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [16]16. PMLR, 2023.
  2. [24]24. Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62, 2022.

Cited by

Sorted by citations

Related

Authors
Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., & Tian, Y.
Affiliations
University of California San Diego (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.06769
arXiv
2412.06769
OpenAlex
W4405254469
Cited by
18
References
56
Citations by year
2026: 6 · 2025: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, chain-of-thought, latent reasoning, breadth-first search, planning, logical reasoning, accuracy-efficiency trade-off

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review