Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

If models output only the final sequence of 2N - 1 moves, token requirements grow linearly with N.

“If models output only the final sequence of 2 N -1 moves, token requirements grow linearly:”

Models are scored as failures for not solving the impossible River Crossing instances.

“yet models are scored as failures for not solving these unsolvable problems.”

• Also stated in: § 3 The Impossible Puzzle Problem · p. 2 · plain

  “By automatically scoring these impossible instances as failures”

When controlling for experimental artifacts by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures.

“When we control for these experimental artifacts, by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures.”

The generated Lua solutions correctly implement the recursive algorithm for Tower of Hanoi.

“The generated solutions correctly implement the recursive algorithm”

The reported collapse before N = 9 for most models occurs well before the theoretical token limits.

“Interestingly, the reported 'collapse' before N = 9 for most models occurs well before these theoretical limits.”

Shojaee et al. check only for solution correctness across all puzzles.

“While Shojaee et al. check only for solution correctness across all puzzles”

Shojaee et al.'s task instructions vary significantly in computational demands.

“their task instructions vary significantly in computational demands.”

The Blocks World prompt explicitly requires optimization.

“The Blocks World prompt explicitly requires optimization:”

• Also stated in: § 6.1 The Optimality Question · p. 4 · plain

  “Find the minimum sequence of moves to transform the initial state into the goal state. Remember that only the topmost block of each stack can be moved.”

The high accuracy on the Lua function representation demonstrates intact reasoning capabilities when freed from exhaustive enumeration requirements.

“demonstrating intact reasoning capabilities when freed from exhaustive enumeration requirements.”

Tower of Hanoi experiments in Shojaee et al. risk exceeding model output token limits.

“Tower of Hanoi experiments risk exceeding model output token limits, with models explicitly acknowledging these constraints in their outputs”

Models receive zero scores on impossible River Crossing instances not for reasoning failures, but for correctly recognizing unsolvable problems.

“Models receive zero scores not for reasoning failures, but for correctly recognizing unsolvable problems.”

The authors' automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities.

“The authors' automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities”

Shojaee et al.'s reported accuracy collapse on planning puzzles primarily reflects experimental design limitations rather than fundamental reasoning failures.

“We demonstrate that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures.”

• Also stated in: § 1 Introduction · p. 1 · plain

  “However, our analysis reveals that these apparent failures stem from experimental design choices rather than inherent model limitations.”

• Also stated in: § 4 Models Abbreviate Long Solutions, Causing Apparent Collapse · p. 3 · plain

  “the high-level conclusion remains unchanged: apparent reasoning failures reflect constraints on solution length rather than fundamental reasoning limitations.”

Models are making a decision to terminate output before actually reaching their context window limits.

“This suggests that models are making a decision to terminate output before actually reaching their context window limits.”

Models may be poorly calibrated about their own context length capabilities, choosing to stop prematurely.

“This behavior indicates that models may be poorly calibrated about their own context length capabilities, choosing to stop prematurely.”

Shojaee et al.'s River Crossing benchmarks include mathematically impossible instances for N ≥ 6 due to insufficient boat capacity b = 3.

“their River Crossing benchmarks include mathematically impossible instances for N ≥ 6 due to insufficient boat capacity, yet models are scored as failures for not solving these unsolvable problems.”

• Also stated in: § 3 The Impossible Puzzle Problem · p. 2 · plain

  “Shojaee et al. test instances with N ≥ 6 actors/agents using boat capacity b = 3.”

Automated evaluation systems that fail to account for model awareness and decision-making mischaracterize model behavior as reasoning collapse.

“This mischaracterization of model behavior as 'reasoning collapse' reflects a broader issue with automated evaluation systems that fail to account for model awareness and decision-making.”

Tower of Hanoi, despite requiring exponentially many moves, has a trivial O(1) decision process per move.

“Tower of Hanoi, despite requiring exponentially many moves, has a trivial O (1) decision process per move.”

A model following the Blocks World prompt faithfully would search for optimal solutions, not knowing that suboptimal solutions would pass the evaluation.

“A model following the prompt faithfully would search for optimal solutions, not knowing that suboptimal solutions would pass the evaluation.”

• AssumptionThe authors state this as an assumption: Models follow the prompt faithfully and attempt to find optimal solutions when instructed to optimize.

The difference in computational difficulty explains why models might execute 100+ Hanoi moves while struggling with shorter planning problems.

“This explains why models might execute 100+ Hanoi moves while struggling with shorter planning problems.”

Scoring models as failures for making reasonable engineering decisions about output length mischaracterizes their actual capabilities.

“Scoring models as 'failures' for making reasonable engineering decisions about output length mischaracterizes their actual capabilities.”

Models attempting Blocks World must solve the harder optimization variant, potentially spending computational resources searching for provably minimal solutions rather than merely valid ones.

“Models attempting Blocks World must solve the harder optimization variant, potentially spending computational resources searching for provably minimal solutions rather than merely valid ones.”

Shojaee et al.'s results demonstrate that programmatic evaluation can miss both model capabilities and puzzle impossibilities.

“that programmatic evaluation can miss both model capabilities and puzzle impossibilities”

Blocks World is much harder than Tower of Hanoi.

“Blocks World, however, is much harder.”

Finding the truly optimal Blocks World solution remains NP-hard.

“however finding the truly optimal solution remains NP-hard.”

• cites [5] [5] Slaney, J. and Thi´ ebaux, S. (2001). Blocks World revisited. Artificial Intelligence , 125(1-2):119153. sourceNot checked — no full text of the source

Shojaee et al.'s results demonstrate that models have some awareness of their own context limits.

“Shojaee et al.'s results demonstrate that models have some awareness of their own context limits”

Near-optimal Blocks World solutions can be found in linear time using domain-specific algorithms.

“Slaney and Thi´ ebaux [5] demonstrated that near-optimal solutions can be found in linear time using domain-specific algorithms”

• cites [5] [5] Slaney, J. and Thi´ ebaux, S. (2001). Blocks World revisited. Artificial Intelligence , 125(1-2):119153. sourceNot checked — no full text of the source

Shojaee et al.'s results do not support claims about fundamental reasoning limitations.

“These are valuable engineering insights, but they do not support claims about fundamental reasoning limitations.”

Shojaee et al.'s results demonstrate that solution length poorly predicts problem difficulty.

“that solution length poorly predicts problem difficulty.”

The authors' use of compositional depth (minimum moves) as their complexity metric conflates mechanical execution with problem-solving difficulty.

“The authors use 'compositional depth' (minimum moves) as their complexity metric, but this conflates mechanical execution with problem-solving difficulty:”