The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Open accessAbstract
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%. Code repo with all prompts: https://github.com/princeton-nlp/tree-of-thought-llm.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Tree of Thoughts (ToT) generalizes chain-of-thought prompting by letting a language model explore, evaluate, and search over multiple intermediate “thoughts” rather than decoding left-to-right. On three tasks challenging GPT-4, ToT substantially improves success rates, including 74% on Game of 24 versus 4% for chain-of-thought.
Key findings
- On Game of 24, GPT-4 with IO prompting achieved 7.3%, CoT 4.0%, and CoT-SC (k=100) 9.0%, while ToT achieved 45% with breadth b=1 and 74% with b=5; best-of-100 CoT reached 49%.
- On Creative Writing, GPT-4 coherency scores averaged 7.56 for ToT versus 6.19 for IO and 6.93 for CoT; in a blind human comparison, ToT was judged more coherent than CoT in 41 cases, CoT more coherent in 21, and similar in 38.
- On 5×5 Mini Crosswords, ToT reached 78% letter-level, 60% word-level, and 20% game-level success, compared with IO at 38.7%, 14%, and 0%, and CoT at 40.6%, 15.6%, and 1%; ToT solved 4/20 games, and an oracle best-state variant solved 7/20.
- With GPT-3.5, ToT still beat CoT and IO on tested tasks, but on Game of 24 GPT-3.5+ToT reached only 19% versus GPT-4+ToT’s 74%; GPT-4 generation with GPT-3.5 evaluation reached 64%, while GPT-3.5 generation with GPT-4 evaluation reached 31%.
Methods
- ToT frames problem solving as search over a tree where each node is a partial solution state; instantiations specify thought decomposition, a thought generator, a state evaluator, and a search algorithm such as BFS or DFS.
- Experiments used off-the-shelf GPT-4 chat completion at temperature 0.7 with no extra training. Game of 24 used 100 hard games indexed 901–1,000 from 4nums.com with BFS keeping b=5 candidates and evaluating thoughts as sure/maybe/impossible; Creative Writing used 100 inputs from random sentences with depth 2, k=5 plans/passages, and voting; Mini Crosswords used 20 games from GooBix with DFS up to 100 steps, confidence aggregation, and pruning.
- Baselines included IO prompting, CoT prompting, CoT self-consistency, and iterative refinement; metrics included task success rates, GPT-4 coherency scores, human blind pairwise comparisons, and letter/word/game success for crosswords.
Limitations
- Stated: ToT may not be necessary for tasks GPT-4 already excels at; the paper explores only three relatively simple tasks that challenge GPT-4; ToT requires more resources and API cost than sampling methods; and the work uses an off-the-shelf LM without fine-tuning.
- Stated/evident: cost and efficiency depend heavily on prompts and search algorithms, and ToT can require 5–100 times more generated tokens than CoT; for Game of 24, ToT used 5.5k completion tokens and cost $0.74 per case versus CoT best-of-100 at 6.7k tokens and $0.47.
- Evident from text: the state evaluator is imperfect—sometimes pruning states for crosswords that are actually solvable, possibly due to rare or obsolete words—and output heuristics can be improved, since oracle best-state selection raised Mini Crosswords game success from 4/20 to 7/20; ablating pruning or backtracking generally worsened performance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
32 Own interpretation · 21 Own result · 20 Sources not found · 4 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5(a) shows average GPT-4 scores across 100 tasks, where ToT (7.56) is deemed to generate more coherent passages than IO (6.19) and CoT (6.93) on average.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
- Rests on an assumption: The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, around 60% of CoT samples already failed the task after generating the first step, or equivalently, the first three words (e.g.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we find that GPT-4 can follow the input constraints most of the time, we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“We propose three tasks that are hard even when sampling from the state-of-the-art language model, GPT-4 [23], using standard IO prompting or chain-of-thought (CoT) prompting.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4 Experiments
What the sources say
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Game of 24 (we changed 1-shot proposal prompt to 3-shot to make it work), GPT-3.5+ToT's19% is far worse than GPT-4+ToT's 74%.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Running ToT requires significantly more computations than IO or CoT prompting.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the performance of ToT is better than best of 100 independent CoT trials.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in Game of 24 (Table 7 below), solving a problem with ToT requires 5.5k completion tokens, close to 100 CoT trials (6.7k tokens).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lastly, we confirm the importance of backtracking by running an ablation that keeps filling the most promising clue for at most 20 steps, allowing overwrites.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This implementation of search heuristics via LM self-evaluation and deliberation is novel, as previous search heuristics are either programmed or learned.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such an improvement is not surprising, given IO and CoT lack mechanisms to try different clues, make changes to decisions, or backtrack.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We propose a third alternative, by using the LM to deliberately reason about states.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“indicating our simple output heuristics can be readily improved.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Research on human problem-solving suggests that people search through a combinatorial problemspace -a tree where the nodes represent partial solutions, and the branches correspond to operators that modify them [21,22].”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3 Tree of Thoughts: Deliberate Problem Solving with LM
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, better heuristics for DFS pruning are critical for problem solving in this case.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“As they are trained on vast amount of world knowledge and human examples, LMs are known to have already absorbed rich commonsense that makes it possible to propose reasonable plans conditioned on problem setting and environmental states [12,42,37,13,35,41,40].”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [42]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [37]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [13]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [35]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [41]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and StrategyQA's bottleneck is external knowledge, not reasoning”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and different generation/evaluation language models might attain decent results while reducing costs.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests the game's bottleneck is thought generation”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which suggests ToT could also work well on weaker language models.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 33
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it could actually find the correct solution for 4/20 games (though only outputting 1 via heuristic), 3 of which are games ToT+pruning cannot solve within 100 steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 2, IO, CoT, and CoT-SC prompting methods perform badly on the task, achieving only 7.3%, 4.0%, and 9.0% success rates.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, ToT with a breadth of b = 1 already achieves a success rate of 45%, while b = 5 achieves 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not surprisingly, CoT scales better than IO, and best of 100 CoT samples achieve a success rate of 49%, but still much worse than exploring more nodes in ToT (b > 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaindirectn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the former, we sample 5 scores and average them for each task output, and we find these 5 scores usually consistent, with a standard deviation of around 0.56 on average across outputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT > ToT Similar ToT > CoT 0 10 20 30 40 21 38 41 (b) Human coherency comparison”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The subset of authors used for human judgments is representative of human readers.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, search methods like ToT requires more resources (e.g.GPT-4 API cost) than sampling methods in order to improve task performances”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To further understand the importance of generation vs. evaluation, we ran GPT-4 generation + GPT-3.5 evaluation (64%) and GPT-3.5 generation + GPT-4 evaluation (31%).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“So completing Game of 24 and Creative Writing's main ToT experiments cost around 0.74 × 100 + 0.32 × 100 = 106 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Creative Writing, we find GPT-3.5+ToToutperform GPT-4+IO, and similar to GPT-4+CoT”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 4 and as expected, ToT improves over CoT on both tasks (but only slightly, given GPT-4 + CoT is already very good on such tasks, and StrategyQA's bottleneck is external knowledge, not reasoning).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, cost and efficiency of ToT highly depend on the prompts and search algorithms used, and could require 5-100 times more generated tokens than CoT.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Convenience.No extra training is needed, just a pre-trained LM is sufficient.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Originally designed to generate text, scaled-up versions of language models (LMs) such as GPT [25,26,1,23] and PaLM [5] have been shown to be increasingly capable of performing an ever wider range of tasks requiring mathematical, symbolic, commonsense, and knowledge reasoning.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [25]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT frames any problem as a search over a tree, where each node is a state s = [x, z 1•••i ] representing a partial solution with the input and the sequence of thoughts so far.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3 Tree of Thoughts: Deliberate Problem Solving with LM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generality.IO, CoT, CoT-SC, and self-refinement can be seen as special cases of ToT (i.e.trees of limited depth and breadth; Figure 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Chain-of-thought (CoT) prompting [38] was proposed to address cases where the mapping of input x to output y is non-trivial (e.g. when x is a math question and y is the final numerical answer).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“RAP [9] is a concurrent work that treats language model reasoning as planning with its internal world model, and proposes a MCTS-based method similar to ToT.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Again, the goal is not just to solve the task, as more general crosswords can be readily solved with specialized NLP pipelines [34] that leverages large-scale retrieval instead of LM.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
What the sources say
- Not matchedcites [34]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks require deductive, mathematical, commonsense, lexical reasoning abilities, and a way to incorporate systematic planning or search.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Research on”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the latter, we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Newell and colleagues characterized problem solving [21] as search through a combinatorial problem space, represented as a tree.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“CoT-SC improves upon CoT, because there are generally different thought processes for the same problem (e.g.different ways to prove the same theorem), and the output decision can be more faithful by exploring a richer set of thoughts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, within each chain there is no local exploration of different thought steps, and the”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed ToT approach extends existing planning formulations by considering multiple potentially feasible plans simultaneously at each problem-solving step, and proceeding with the most promising ones.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT actively maintains a tree of thoughts, where each thought is a coherent language sequence that serves as an intermediate step toward problem solving (Table 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The simple associative token-level choices of LMs are also reminiscent of”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, traditional decision-making procedures usually require training dedicated reward and policy models as in reinforcement learning (for example CHAI [33]), whereas we use the LM itself to provide the value estimates for decision making.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [33]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we introduce a new framework for language model inference,”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is perhaps surprising that underlying all this progress is still the original autoregressive mechanism for generating text, which makes token-level decisions one by one and in a left-to-right fashion.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modularity.The base LM, as well as the thought decomposition, generation, evaluation, and search procedures can all be varied independently.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applicable, such a deliberate heuristic can be more flexible than programmed rules, and more sample-efficient than learned models.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Adaptability.Different problem properties, LM capabilities, and resource constraints can be accommodated.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Similar to our method, self-eval decoding also follows a tree-search procedure with leaves sampled from stochastic beam search decoding, which are then evaluated by LLM itself with carefully prepared self-eval prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“This approach however differs from ours in that trees are expanded by sampling external paragraphs instead of the LM's own thoughts, and there is no reflection or voting steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but the modular flexibility of ToT allows users to customize such performance-cost tradeoffs”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“From this perspective, our method is also related to NeuroLogic A*esque decoding [19], which is inspired by A* search but introduces look-ahead heuristics that are efficient for LMs to improve the beam-search or top-k sampling decoding.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“For example, Schlag et al. [27] embeds LMs in an algorithmic search procedure to help solve problems like question answering step-by-step, in which the search trees are expanded by relevant paragraphs that might provide answers.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT is a framework that empowers LMs to more autonomously and intelligently make decisions and solve problems.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Crosswords' DFS experiments should be also within 100 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“as an initial step this work only explores three relatively simple tasks that challenges GPT-4”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This method however is constrained to sentence generation tasks, whereas our framework are designed for complex, multi-step problem solving guarded by value feedback.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“For example it can be considered as a heuristic search algorithm like A* [10], in which the heuristic at each search node is provided by the LM's selfassessment.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, LMs address a weakness of these classical methods, providing a way to solve complex problems that are not easily formalized, such as creative writing.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Their approach however, uses the PAL formulation [8] which represents thoughts as codes, which makes it difficult to tackle challenging tasks like creative writing which we consider in this paper.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, ToT also improves the interpretability of model decisions and the opportunity for human alignment, as the resulting representations are readable, high-level language reasoning instead of implicit, low-level token values.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“However, its tasks are simpler than ours, and its framework lacks the modularity to incorporate different tree search algorithms.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Deliberate search such as ToT might not be necessary for many existing tasks that GPT-4 already excels at (see Appendix B.1)”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Tree of Thoughts framework provides a way to translate classical insights about problem-solving into actionable methods for contemporary LMs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our Tree-of-Thought formulation is thus more versatile and handles challenging tasks on which GPT-4 only achieves very low accuracy with standard prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Another approach, LLM+P [18], goes one step further and delegates the actual planning process to a classical planner.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The authors state this as an assumption.
The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
“Such valuations do not need to be perfect, and only need to be approximately helpful for decision making.”
The paper relies on this without support.
The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
“we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
The paper relies on this without support.
The subset of authors used for human judgments is representative of human readers.
“we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, these modules maintain internal states for reasoning to ensure the coherence of thought (Yao et al., 2023; Sun et al., 2025c);”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although augmentations such as the ReAct framework [136] or Tree-of-Thoughts [164] introduce pseudo-recursive reasoning, they remain fundamentally heuristic and lack true internal models of time, causality, or state evolution.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, many early language agents simply use LLMs to propose an action (Schick et al., 2023), a sequence of actions (Huang et al., 2022b), or evaluate a fixed set of actions (Ahn et al., 2022) without intermediate reasoning or retrieval. Followup work (Yao et al., 2022b; Shinn et al., 2023; Xu et al., 2023b; Lin et al., 2023; Wang et al., 2023a; Park et al., 2023) has exploited intermediate”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tree of Thoughts (ToT) (Yao et al., 2023) can be seen as a special kind of language agent with only one external action: submitting a final solution to a reasoning problem (game of 24, creative writing, crosswords puzzle).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM reasoning methods fix a search budget by specifying a depth of reasoning (Yao et al., 2023)”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The performance of LLMs is highly sensitive to the quality of their prompts.”
From HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems (2025) · cites this paper as Yao et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The remaining difference is likely due to the fact that ToT implements a classical breadth-first search algorithm, only prompting the LLM to generate (much easier) intermediate steps and heuristic evaluations rather than full solutions and reflections.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Per problem, 100 CoT prompts costs $0.47, but running Tree of Thoughts (ToT) averages $0.74-cost-equivalent to about 150 CoT prompts.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompting is effective in mitigating hallucination by guiding them towards solutions more preferred by humans, which are possibly the ones with lower complexities, and within LLMs' capabilities.”
From Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) · cites this paper as [79]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) reasoning (Yao et al., 2023; Besta et al., 2024) extend single-agent reflection through structured deliberation”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It achieves an accuracy of 0.800(±0.0616),significantly outperforming the previous state-of-the-art method, TOT evaluated with GPT-4, with an accuracy of 0.675 (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For Game of 24, we can easily verify the answer by checking whether the generated answer is 24.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Comparing with CoT-SC, which generates all the planed steps together, ToT needs to query LLMs for each reasoning step.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts (ToT) [50] is designed to generate plans using a tree-like reasoning structure.In this approach, each node in the tree represents a”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Early approaches extend CoT into treeor graph-structured representations, such as tree-of-thoughts (ToT) [254] and graph-ofthoughts (GoT) [255], enabling LLMs to branch, evaluate, and backtrack across multiple reasoning paths.”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“LongCoT offers two principal advantages over traditional CoT: (1) it enables iterative self-correction during reasoning rather than one-shot generation, and (2) it allows for structured, multi-step exploration akin to advanced search algorithms such as ToT [254].”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [136], Chain-of-Thought (CoT) prompting [163], and Tree of Thoughts [164].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
The cited source says what the claim attributes to it.
In the citing paper
“While the earliest agents used LLMs to directly select or generate actions (Figure 1B; Ahn et al., 2022; Huang et al., 2022b), more recent agents additionally use them to reason (Yao et al., 2022b), plan (Hao et al., 2023; Yao et al., 2023), and manage long-term memory (Park et al., 2023; Wang et al., 2023a) to improve decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“During planning, ToT iteratively proposes, evaluates, and selects 'thoughts' (reasoning actions) based on LLM reasoning, and maintains them via a tree search algorithm to enable global exploration as well as local backtrack and foresight.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most language agents use language-only models for decision-making (Yao et al., 2022b; Wang et al., 2023a; Yao et al., 2023), employing a separate captioning model to convert environment observations to text when necessary (Ahn et al., 2022; Zeng et al., 2022).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Initial works have implemented classical planning and tree search (Yao et al., 2023; Hao et al., 2023; Liu et al., 2023a; Dagan et al., 2023), using toy tasks such as game of 24 or block building.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“A number of tasks that LLMs are evaluated on (e.g. writing prompts [52], constrained text generation [28], toxicity control [49, 14] , etc.) are problems without a well-defined ground truth.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“On the exact same test set, we extend our experiments to 150 (direct, non-CoT) queries with a sound verifier, and we reach 70%, comparable to ToT's reported 74%.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“Our results on the Game of 24 setting seem to contradict the results shown in [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“By reducing compositionality and offloading it to a proven classical algorithm, ToT sidesteps some of the major hurdles to LLM reasoning.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“The landmark paper by Yao et al. (2023) introducing the method has received significant attention in the community as it described a significant improvement and also highlights shortcomings of previous methods.”
From Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent studies like ToT [228], Plan-and-Solve [229] and ReAct [230] have proposed more advanced techniques for prompt design to assist in guiding LLM to engage in deeper thinking and tool planning.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [228]
The cited source says what the claim attributes to it.
In the citing paper
“It should be noted DFSDT shares a similar idea to a concurrent work: tree-of-thought (ToT) reasoning (Yao et al., 2023).”
From Counterfactually Auditable Lifecycle Certification for Autonomous Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [138], Chain-of-Thought (CoT) prompting [165], and Tree of Thoughts [166].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [166]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, Monte Carlo Tree Search (MCTS) [24] and other treebased search algorithms has been demonstrated effective for LLM to improve planning and reasoning [25-29].”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“It is now widely known and accepted that contemporary LLMs such as GPT-4 can exhibit substantial improvements in reasoning and generation capability when additional compute is devoted to inference-time strategies (e.g., chain-ofthought prompting (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022), tree-of-thought expansions (Yao et al., 2023; Long, 2023), minimum Bayes risk decoding (Suzgun”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, these expansions come at the cost of increased computational overhead per test instance (Yao et al., 2023).”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For a fair comparison, we also evaluate the TOT implementation by Yao et al. (2023) with GPT-4-Turbo.It yields an accuracy of 0.668.”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The choice of Mini Crosswords for this analysis is strategic, as it highlights how the algorithmic structure of the solvers, such as the tree search employed by TOT, significantly influences their performance (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts explores and evaluates alternative reasoning paths (Yao et al., 2023)”
From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Yao et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, this is mainly because the self-verification setting presented in the main text of this paper is not directly comparable to that of [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While ToT improved accuracy (64.55% vs. 62.52%), this was still far from GPT-4o's zero-shot performance of 94.00%”
From Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“if V (p θ , {s ′ })(s) > v thres then ▷ pruning”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“V (p θ , S)(s) ∼ p value θ (v|s) ∀s ∈ S”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- S. A. Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1): 3, 1996.
- K. E. Stanovich. Who is rational? Studies of individual differences in reasoning. Psychology Press, 1999.
Cited by
Sorted by citations
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113
“To date, the perhaps most complicated method is known as Tree-of-Thought (ToT) prompting. landmark paper by Yao et al. (2023) introducing the method has received significant attention in the community as it described a significant improvement and also highlights shortcomings of previous methods.”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556
Metadata
- Authors
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 11809-11822
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2305.10601
- OpenAlex
- W7133209886
- Cited by
- 1,158
- References
- 0
- Citations by year
- 2026: 476 · 2025: 320 · 2024: 284 · 2023: 77
- FWCI
- 206.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2239363)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- tree-of-thought, large language models, chain-of-thought prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
