The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Open accessAbstract
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%. Code repo with all prompts: https://github.com/princeton-nlp/tree-of-thought-llm.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Tree of Thoughts (ToT) generalizes chain-of-thought prompting by letting a language model explore, evaluate, and search over multiple intermediate “thoughts” rather than decoding left-to-right. On three tasks challenging GPT-4, ToT substantially improves success rates, including 74% on Game of 24 versus 4% for chain-of-thought.
Key findings
- On Game of 24, GPT-4 with IO prompting achieved 7.3%, CoT 4.0%, and CoT-SC (k=100) 9.0%, while ToT achieved 45% with breadth b=1 and 74% with b=5; best-of-100 CoT reached 49%.
- On Creative Writing, GPT-4 coherency scores averaged 7.56 for ToT versus 6.19 for IO and 6.93 for CoT; in a blind human comparison, ToT was judged more coherent than CoT in 41 cases, CoT more coherent in 21, and similar in 38.
- On 5×5 Mini Crosswords, ToT reached 78% letter-level, 60% word-level, and 20% game-level success, compared with IO at 38.7%, 14%, and 0%, and CoT at 40.6%, 15.6%, and 1%; ToT solved 4/20 games, and an oracle best-state variant solved 7/20.
- With GPT-3.5, ToT still beat CoT and IO on tested tasks, but on Game of 24 GPT-3.5+ToT reached only 19% versus GPT-4+ToT’s 74%; GPT-4 generation with GPT-3.5 evaluation reached 64%, while GPT-3.5 generation with GPT-4 evaluation reached 31%.
Methods
- ToT frames problem solving as search over a tree where each node is a partial solution state; instantiations specify thought decomposition, a thought generator, a state evaluator, and a search algorithm such as BFS or DFS.
- Experiments used off-the-shelf GPT-4 chat completion at temperature 0.7 with no extra training. Game of 24 used 100 hard games indexed 901–1,000 from 4nums.com with BFS keeping b=5 candidates and evaluating thoughts as sure/maybe/impossible; Creative Writing used 100 inputs from random sentences with depth 2, k=5 plans/passages, and voting; Mini Crosswords used 20 games from GooBix with DFS up to 100 steps, confidence aggregation, and pruning.
- Baselines included IO prompting, CoT prompting, CoT self-consistency, and iterative refinement; metrics included task success rates, GPT-4 coherency scores, human blind pairwise comparisons, and letter/word/game success for crosswords.
Limitations
- Stated: ToT may not be necessary for tasks GPT-4 already excels at; the paper explores only three relatively simple tasks that challenge GPT-4; ToT requires more resources and API cost than sampling methods; and the work uses an off-the-shelf LM without fine-tuning.
- Stated/evident: cost and efficiency depend heavily on prompts and search algorithms, and ToT can require 5–100 times more generated tokens than CoT; for Game of 24, ToT used 5.5k completion tokens and cost $0.74 per case versus CoT best-of-100 at 6.7k tokens and $0.47.
- Evident from text: the state evaluator is imperfect—sometimes pruning states for crosswords that are actually solvable, possibly due to rare or obsolete words—and output heuristics can be improved, since oracle best-state selection raised Mini Crosswords game success from 4/20 to 7/20; ablating pruning or backtracking generally worsened performance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
32 Own interpretation · 21 Own result · 20 Sources not found · 4 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5(a) shows average GPT-4 scores across 100 tasks, where ToT (7.56) is deemed to generate more coherent passages than IO (6.19) and CoT (6.93) on average.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
- Rests on an assumption: The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, around 60% of CoT samples already failed the task after generating the first step, or equivalently, the first three words (e.g.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we find that GPT-4 can follow the input constraints most of the time, we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“We propose three tasks that are hard even when sampling from the state-of-the-art language model, GPT-4 [23], using standard IO prompting or chain-of-thought (CoT) prompting.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4 Experiments
What the sources say
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Game of 24 (we changed 1-shot proposal prompt to 3-shot to make it work), GPT-3.5+ToT's19% is far worse than GPT-4+ToT's 74%.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Running ToT requires significantly more computations than IO or CoT prompting.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the performance of ToT is better than best of 100 independent CoT trials.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in Game of 24 (Table 7 below), solving a problem with ToT requires 5.5k completion tokens, close to 100 CoT trials (6.7k tokens).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 33
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it could actually find the correct solution for 4/20 games (though only outputting 1 via heuristic), 3 of which are games ToT+pruning cannot solve within 100 steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 2, IO, CoT, and CoT-SC prompting methods perform badly on the task, achieving only 7.3%, 4.0%, and 9.0% success rates.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, ToT with a breadth of b = 1 already achieves a success rate of 45%, while b = 5 achieves 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not surprisingly, CoT scales better than IO, and best of 100 CoT samples achieve a success rate of 49%, but still much worse than exploring more nodes in ToT (b > 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaindirectn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the former, we sample 5 scores and average them for each task output, and we find these 5 scores usually consistent, with a standard deviation of around 0.56 on average across outputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT > ToT Similar ToT > CoT 0 10 20 30 40 21 38 41 (b) Human coherency comparison”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The subset of authors used for human judgments is representative of human readers.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, search methods like ToT requires more resources (e.g.GPT-4 API cost) than sampling methods in order to improve task performances”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To further understand the importance of generation vs. evaluation, we ran GPT-4 generation + GPT-3.5 evaluation (64%) and GPT-3.5 generation + GPT-4 evaluation (31%).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“So completing Game of 24 and Creative Writing's main ToT experiments cost around 0.74 × 100 + 0.32 × 100 = 106 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Creative Writing, we find GPT-3.5+ToToutperform GPT-4+IO, and similar to GPT-4+CoT”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 4 and as expected, ToT improves over CoT on both tasks (but only slightly, given GPT-4 + CoT is already very good on such tasks, and StrategyQA's bottleneck is external knowledge, not reasoning).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, cost and efficiency of ToT highly depend on the prompts and search algorithms used, and could require 5-100 times more generated tokens than CoT.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Convenience.No extra training is needed, just a pre-trained LM is sufficient.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Originally designed to generate text, scaled-up versions of language models (LMs) such as GPT [25,26,1,23] and PaLM [5] have been shown to be increasingly capable of performing an ever wider range of tasks requiring mathematical, symbolic, commonsense, and knowledge reasoning.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [25]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT frames any problem as a search over a tree, where each node is a state s = [x, z 1•••i ] representing a partial solution with the input and the sequence of thoughts so far.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3 Tree of Thoughts: Deliberate Problem Solving with LM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generality.IO, CoT, CoT-SC, and self-refinement can be seen as special cases of ToT (i.e.trees of limited depth and breadth; Figure 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Chain-of-thought (CoT) prompting [38] was proposed to address cases where the mapping of input x to output y is non-trivial (e.g. when x is a math question and y is the final numerical answer).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“RAP [9] is a concurrent work that treats language model reasoning as planning with its internal world model, and proposes a MCTS-based method similar to ToT.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Again, the goal is not just to solve the task, as more general crosswords can be readily solved with specialized NLP pipelines [34] that leverages large-scale retrieval instead of LM.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
What the sources say
- Not matchedcites [34]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks require deductive, mathematical, commonsense, lexical reasoning abilities, and a way to incorporate systematic planning or search.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Research on”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the latter, we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Newell and colleagues characterized problem solving [21] as search through a combinatorial problem space, represented as a tree.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“CoT-SC improves upon CoT, because there are generally different thought processes for the same problem (e.g.different ways to prove the same theorem), and the output decision can be more faithful by exploring a richer set of thoughts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, within each chain there is no local exploration of different thought steps, and the”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed ToT approach extends existing planning formulations by considering multiple potentially feasible plans simultaneously at each problem-solving step, and proceeding with the most promising ones.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT actively maintains a tree of thoughts, where each thought is a coherent language sequence that serves as an intermediate step toward problem solving (Table 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The simple associative token-level choices of LMs are also reminiscent of”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, traditional decision-making procedures usually require training dedicated reward and policy models as in reinforcement learning (for example CHAI [33]), whereas we use the LM itself to provide the value estimates for decision making.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [33]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we introduce a new framework for language model inference,”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is perhaps surprising that underlying all this progress is still the original autoregressive mechanism for generating text, which makes token-level decisions one by one and in a left-to-right fashion.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modularity.The base LM, as well as the thought decomposition, generation, evaluation, and search procedures can all be varied independently.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applicable, such a deliberate heuristic can be more flexible than programmed rules, and more sample-efficient than learned models.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Adaptability.Different problem properties, LM capabilities, and resource constraints can be accommodated.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Similar to our method, self-eval decoding also follows a tree-search procedure with leaves sampled from stochastic beam search decoding, which are then evaluated by LLM itself with carefully prepared self-eval prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“This approach however differs from ours in that trees are expanded by sampling external paragraphs instead of the LM's own thoughts, and there is no reflection or voting steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but the modular flexibility of ToT allows users to customize such performance-cost tradeoffs”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“From this perspective, our method is also related to NeuroLogic A*esque decoding [19], which is inspired by A* search but introduces look-ahead heuristics that are efficient for LMs to improve the beam-search or top-k sampling decoding.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“For example, Schlag et al. [27] embeds LMs in an algorithmic search procedure to help solve problems like question answering step-by-step, in which the search trees are expanded by relevant paragraphs that might provide answers.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT is a framework that empowers LMs to more autonomously and intelligently make decisions and solve problems.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Crosswords' DFS experiments should be also within 100 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“as an initial step this work only explores three relatively simple tasks that challenges GPT-4”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This method however is constrained to sentence generation tasks, whereas our framework are designed for complex, multi-step problem solving guarded by value feedback.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“For example it can be considered as a heuristic search algorithm like A* [10], in which the heuristic at each search node is provided by the LM's selfassessment.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, LMs address a weakness of these classical methods, providing a way to solve complex problems that are not easily formalized, such as creative writing.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Their approach however, uses the PAL formulation [8] which represents thoughts as codes, which makes it difficult to tackle challenging tasks like creative writing which we consider in this paper.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, ToT also improves the interpretability of model decisions and the opportunity for human alignment, as the resulting representations are readable, high-level language reasoning instead of implicit, low-level token values.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“However, its tasks are simpler than ours, and its framework lacks the modularity to incorporate different tree search algorithms.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Deliberate search such as ToT might not be necessary for many existing tasks that GPT-4 already excels at (see Appendix B.1)”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Tree of Thoughts framework provides a way to translate classical insights about problem-solving into actionable methods for contemporary LMs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our Tree-of-Thought formulation is thus more versatile and handles challenging tasks on which GPT-4 only achieves very low accuracy with standard prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Another approach, LLM+P [18], goes one step further and delegates the actual planning process to a classical planner.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The authors state this as an assumption.
The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
“Such valuations do not need to be perfect, and only need to be approximately helpful for decision making.”
The paper relies on this without support.
The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
“we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
The paper relies on this without support.
The subset of authors used for human judgments is representative of human readers.
“we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, these modules maintain internal states for reasoning to ensure the coherence of thought (Yao et al., 2023; Sun et al., 2025c);”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although augmentations such as the ReAct framework [136] or Tree-of-Thoughts [164] introduce pseudo-recursive reasoning, they remain fundamentally heuristic and lack true internal models of time, causality, or state evolution.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, many early language agents simply use LLMs to propose an action (Schick et al., 2023), a sequence of actions (Huang et al., 2022b), or evaluate a fixed set of actions (Ahn et al., 2022) without intermediate reasoning or retrieval. Followup work (Yao et al., 2022b; Shinn et al., 2023; Xu et al., 2023b; Lin et al., 2023; Wang et al., 2023a; Park et al., 2023) has exploited intermediate”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tree of Thoughts (ToT) (Yao et al., 2023) can be seen as a special kind of language agent with only one external action: submitting a final solution to a reasoning problem (game of 24, creative writing, crosswords puzzle).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM reasoning methods fix a search budget by specifying a depth of reasoning (Yao et al., 2023)”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The performance of LLMs is highly sensitive to the quality of their prompts.”
From HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems (2025) · cites this paper as Yao et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The remaining difference is likely due to the fact that ToT implements a classical breadth-first search algorithm, only prompting the LLM to generate (much easier) intermediate steps and heuristic evaluations rather than full solutions and reflections.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Per problem, 100 CoT prompts costs $0.47, but running Tree of Thoughts (ToT) averages $0.74-cost-equivalent to about 150 CoT prompts.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompting is effective in mitigating hallucination by guiding them towards solutions more preferred by humans, which are possibly the ones with lower complexities, and within LLMs' capabilities.”
From Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) · cites this paper as [79]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) reasoning (Yao et al., 2023; Besta et al., 2024) extend single-agent reflection through structured deliberation”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It achieves an accuracy of 0.800(±0.0616),significantly outperforming the previous state-of-the-art method, TOT evaluated with GPT-4, with an accuracy of 0.675 (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For Game of 24, we can easily verify the answer by checking whether the generated answer is 24.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Comparing with CoT-SC, which generates all the planed steps together, ToT needs to query LLMs for each reasoning step.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts (ToT) [50] is designed to generate plans using a tree-like reasoning structure.In this approach, each node in the tree represents a”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Early approaches extend CoT into treeor graph-structured representations, such as tree-of-thoughts (ToT) [254] and graph-ofthoughts (GoT) [255], enabling LLMs to branch, evaluate, and backtrack across multiple reasoning paths.”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“LongCoT offers two principal advantages over traditional CoT: (1) it enables iterative self-correction during reasoning rather than one-shot generation, and (2) it allows for structured, multi-step exploration akin to advanced search algorithms such as ToT [254].”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [136], Chain-of-Thought (CoT) prompting [163], and Tree of Thoughts [164].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
The cited source says what the claim attributes to it.
In the citing paper
“While the earliest agents used LLMs to directly select or generate actions (Figure 1B; Ahn et al., 2022; Huang et al., 2022b), more recent agents additionally use them to reason (Yao et al., 2022b), plan (Hao et al., 2023; Yao et al., 2023), and manage long-term memory (Park et al., 2023; Wang et al., 2023a) to improve decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“During planning, ToT iteratively proposes, evaluates, and selects 'thoughts' (reasoning actions) based on LLM reasoning, and maintains them via a tree search algorithm to enable global exploration as well as local backtrack and foresight.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most language agents use language-only models for decision-making (Yao et al., 2022b; Wang et al., 2023a; Yao et al., 2023), employing a separate captioning model to convert environment observations to text when necessary (Ahn et al., 2022; Zeng et al., 2022).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Initial works have implemented classical planning and tree search (Yao et al., 2023; Hao et al., 2023; Liu et al., 2023a; Dagan et al., 2023), using toy tasks such as game of 24 or block building.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“A number of tasks that LLMs are evaluated on (e.g. writing prompts [52], constrained text generation [28], toxicity control [49, 14] , etc.) are problems without a well-defined ground truth.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“On the exact same test set, we extend our experiments to 150 (direct, non-CoT) queries with a sound verifier, and we reach 70%, comparable to ToT's reported 74%.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“Our results on the Game of 24 setting seem to contradict the results shown in [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“By reducing compositionality and offloading it to a proven classical algorithm, ToT sidesteps some of the major hurdles to LLM reasoning.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“The landmark paper by Yao et al. (2023) introducing the method has received significant attention in the community as it described a significant improvement and also highlights shortcomings of previous methods.”
From Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent studies like ToT [228], Plan-and-Solve [229] and ReAct [230] have proposed more advanced techniques for prompt design to assist in guiding LLM to engage in deeper thinking and tool planning.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [228]
The cited source says what the claim attributes to it.
In the citing paper
“It should be noted DFSDT shares a similar idea to a concurrent work: tree-of-thought (ToT) reasoning (Yao et al., 2023).”
From Counterfactually Auditable Lifecycle Certification for Autonomous Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [138], Chain-of-Thought (CoT) prompting [165], and Tree of Thoughts [166].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [166]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, Monte Carlo Tree Search (MCTS) [24] and other treebased search algorithms has been demonstrated effective for LLM to improve planning and reasoning [25-29].”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“It is now widely known and accepted that contemporary LLMs such as GPT-4 can exhibit substantial improvements in reasoning and generation capability when additional compute is devoted to inference-time strategies (e.g., chain-ofthought prompting (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022), tree-of-thought expansions (Yao et al., 2023; Long, 2023), minimum Bayes risk decoding (Suzgun”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, these expansions come at the cost of increased computational overhead per test instance (Yao et al., 2023).”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For a fair comparison, we also evaluate the TOT implementation by Yao et al. (2023) with GPT-4-Turbo.It yields an accuracy of 0.668.”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The choice of Mini Crosswords for this analysis is strategic, as it highlights how the algorithmic structure of the solvers, such as the tree search employed by TOT, significantly influences their performance (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts explores and evaluates alternative reasoning paths (Yao et al., 2023)”
From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Yao et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, this is mainly because the self-verification setting presented in the main text of this paper is not directly comparable to that of [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While ToT improved accuracy (64.55% vs. 62.52%), this was still far from GPT-4o's zero-shot performance of 94.00%”
From Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“if V (p θ , {s ′ })(s) > v thres then ▷ pruning”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“V (p θ , S)(s) ∼ p value θ (v|s) ∀s ∈ S”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- S. A. Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1): 3, 1996.
- K. E. Stanovich. Who is rational? Studies of individual differences in reasoning. Psychology Press, 1999.
Cited by
Sorted by citations
Items 101–200 of 1,294
Faithful Logical Reasoning via Symbolic Chain-of-Thought
Xu, J., Fei, H., Pan, L., Liu, Q., Lee, M.-L., & Hsu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.720 · Cited by 53
Large language model-based evolutionary optimizer: Reasoning with elitism
Brahmachary, S., Joshi, S. M., Panda, A., Koneripalli, K., Sagotra, A. K., Patel, H., Sharma, A., Jagtap, A. D., & Kalyanaraman, K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129272 · Cited by 66
Tamberg, K., & Bahşi, H. (2025) · IEEE Access · doi:10.1109/access.2025.3541146 · Cited by 77
Nie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 70
GeoLLM: A specialized large language model framework for intelligent geotechnical design
Xu, H., Zhang, N., Yin, Z., & Njock, P. G. A. (2024) · Computers and Geotechnics · doi:10.1016/j.compgeo.2024.106849 · Cited by 50
Artificial Intelligence and Entrepreneurship
Fossen, F. M., McLemore, T., & Sorgner, A. (2024) · Foundations and Trends® in Entrepreneurship · doi:10.1561/0300000130 · Cited by 62
Luo, H., Zhang, Z., Zhu, Q., Ameur, N. E. H. B., Liu, X., Fan, D., & Cai, Y. (2025) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2025.105323 · Cited by 49
Romero, O. J., Zimmerman, J., Steinfeld, A. M., & Tomasic, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27706 · Cited by 56
Industrial Foundation Models (IFMs) for intelligent manufacturing: A systematic review
Zhao, S., Liu, S., Jiang, Y., Zhao, B. M., Lv, Y., Zhang, J., Wang, L., & Zhong, R. Y. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.011 · Cited by 50
Çelik, İ., Kontkanen, S., Laru, J., & Dalyanci, A. A. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105485 · Cited by 56
Correctness Comparison of ChatGPT ‐4, Gemini, Claude‐3, and Copilot for Spatial Tasks
Hochmair, H. H., Juhász, L., & Kemp, T. (2024) · Transactions in GIS · doi:10.1111/tgis.13233 · Cited by 66
Oniani, D., Wu, X., Visweswaran, S., Kapoor, S., Kooragayalu, S., Polanska, K., & Wang, Y. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00111 · Cited by 57
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49
Benchmarking Human–AI collaboration for common evidence appraisal tools
Woelfle, T., Hirt, J., Janiaud, P., Kappos, L., Ioannidis, J. P. A., & Hemkens, L. G. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111533 · Cited by 46
Explainability in the age of large language models for healthcare
Mesinovic, M., Watkinson, P., & Zhu, T. (2025) · Communications Engineering · doi:10.1038/s44172-025-00453-y · Cited by 53
Xia, Y., Dittler, D., Jazdi, N., Chen, H., & Weyrich, M. (2024) · 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa61755.2024.10710900 · Cited by 41
Knowledge-Empowered, Collaborative, and Co-Evolving AI Models: The Post-LLM Roadmap
Wu, F., Shen, T., Bäck, T., Chen, J., Huang, G., Jin, Y., Kuang, K., Li, M., Lu, C., Miao, J., et al. (2024) · Engineering · doi:10.1016/j.eng.2024.12.008 · Cited by 47
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51
Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 56
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Yang, Y., Zhou, T., Li, K., Tao, D., Li, L., Shen, L., He, X., Jiang, J., & Shi, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02482 · Cited by 51
Deploying and Evaluating LLMs to Program Service Mobile Robots
Hu, Z., Lucchetti, F., Schlesinger, C., Saxena, Y., Freeman, A., Modak, S., Guha, A., & Biswas, J. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3360020 · Cited by 51
Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023) · arXiv · doi:10.48550/arxiv.2304.13007 · Cited by 38
(2023) · doi:10.18653/v1/2023.emnlp-main.364 · Cited by 38
Berger, A., Hillebrand, L., Leonhard, D., Deußer, T., Oliveira, T. B. F. D., Dilmaghani, T., Khaled, M. B., Kliem, B., Loitz, R., Bauckhage, C., et al. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386518 · Cited by 43
A novel large-language-model-driven framework for named entity recognition
Wang, Z., Chen, H., Xu, G., & Ren, M. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104054 · Cited by 40
Monzon, N., & Hays, F. A. (2025) · JMIR Medical Education · doi:10.2196/59210 · Cited by 61
When IoT Meet LLMs: Applications and Challenges
Kök, İ., Demirci, O., & Özdemi̇r, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825187 · Cited by 52
OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
Xing, S., Qian, C., Wang, Y., Hua, H., Tian, K., Zhou, Y., & Tu, Z. (2025) · 2025 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00113 · Cited by 37
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712
“An example of this approach is Tree of Thought (Yao et al., 2023), wherein reasoning of LLMs is enhanced through step-wise verification paired with self-consistency.”
MethodTree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models
Kim, G., Kim, S., Jeon, B., Park, J., & Kang, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.63 · Cited by 84
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities
Wei, H., Zhang, Z., He, S., Xia, T., Pan, S., & Liu, F. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.958 · Cited by 36
Understanding the planning of LLM agents: A survey
Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y., Tang, R., & Chen, E. (2024) · arXiv · doi:10.48550/arxiv.2402.02716 · Cited by 47
SayCanPay: Heuristic Planning with Large Language Models Using Learnable Domain Knowledge
Hazra, R., Martires, P. Z. D., & Raedt, L. D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.29991 · Cited by 51
Examining GPT-4's Capabilities and Enhancement with SocraSynth
Chang, E. Y. (2023) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci62032.2023.00009 · Cited by 36
MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations
Yao, H., Song, Z., Zhou, Y., Ao, T., Chen, B., & Liu, L. (2024) · ACM Transactions on Graphics · doi:10.1145/3658137 · Cited by 46
Wang, C., Kang, Y., Gong, Z., Zhao, P., Feng, Y., Zhang, W., & Li, G. (2025) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2507844 · Cited by 38
Large models in medical imaging: Advances and prospects
Fang, M., Wang, Z., Pan, S., Feng, X., Zhao, Y., Hou, D., Wu, L., Xie, X., Zhang, X.-Y., Tian, J., et al. (2025) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003699 · Cited by 43
R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization
Yang, Y., He, X., Pan, H., Jiang, X., Deng, Y., Yang, X. Y., Lu, H., Yin, D., Rao, F., Zhu, M., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00229 · Cited by 36
Large Language Models for Telecom: The Next Big Thing?Preprint
Bariah, L., Zhao, Q., Zou, H., Tian, Y. T., Bader, F., & Debbah, M. A. (2023) · doi:10.36227/techrxiv.23536440.v1 · Cited by 36
Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerationsPreprint
Okonji, O. R., Yunusov, K., & Gordon, B. (2024) · doi:10.36227/techrxiv.171527587.75649430/v1 · Cited by 51
The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey
Masterman, T., Besen, S., Sawtell, M., & Chao, A. (2024) · arXiv · doi:10.48550/arxiv.2404.11584 · Cited by 35
“For example, Plan Like a Graph (PLaG) is an approach that represents plans as directed graphs, with multiple steps being executed in parallel [15, 33].”
BackgroundA comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
An Empirical Study on Challenges for LLM Application Developers
Chen, X., Gao, C., Chen, C., Zhang, G., & Liu, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715007 · Cited by 73
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
LightVA: Lightweight Visual Analytics With LLM Agent-Based Task Planning and Execution
Zhao, Y., Wang, J., Xiang, L., Zhang, X., Guo, Z., Turkay, Ç., Zhang, Y., & Chen, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3496112 · Cited by 65
A LLM-informed multi-agent AI system for drone-based visual inspection for infrastructure
Liu, J., Li, H., Chai, C., Chen, K., & Wang, D. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103643 · Cited by 32
Shi, Y., Ren, P., Wang, J., Han, B., ValizadehAslani, T., Agbavor, F., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104533 · Cited by 35
Optimizing Large Language Models: A Deep Dive into Effective Prompt Engineering Techniques
Son, M., Won, Y.-J., & Lee, S. (2025) · Applied Sciences · doi:10.3390/app15031430 · Cited by 35
An outline of Prognostics and health management Large Model: Concepts, Paradigms, and challenges
Tao, L., Li, S., Liu, H., Huang, Q., Ma, L., Ning, G., Chen, Y., Wu, Y., Li, B., Zhang, W., et al. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112683 · Cited by 37
Recent Advances in Reinforcement Learning for Chemical Process Control
Devarakonda, V. S., Sun, W., Tang, X., & Tian, Y. (2025) · Processes · doi:10.3390/pr13061791 · Cited by 34
Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktäschel, T. (2023) · arXiv · doi:10.48550/arxiv.2309.16797 · Cited by 31
AgentLens: Visual Analysis for Agent Behaviors in LLM-Based Autonomous Systems
Lu, J., Pan, B., Chen, J., Feng, Y., Hu, J., Peng, Y., & Chen, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3394053 · Cited by 46
Large Knowledge Model: Perspectives and Challenges
Chen, H. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0001 · Cited by 56
Arafat, M. E., Sourav, S., & Tamás, O. (2024) · IEEE International Symposium on Intelligent Systems and Informatics (SISY) · doi:10.1109/sisy62279.2024.10737601 · Cited by 33
Goldstein, S., & Kirk‐Giannini, C. D. (2025) · Asian Journal of Philosophy · doi:10.1007/s44204-025-00246-2 · Cited by 44
Personalized Large Language Models
Woźniak, S., Koptyra, B., Janz, A., Kazienko, P., & Kocoń, J. (2024) · 2024 IEEE International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw65004.2024.00071 · Cited by 28
Li, F., Hogg, D., & Cohn, A. G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29811 · Cited by 30
Buster, G., Pinchuk, P., Barrons, J., McKeever, R., Levine, A., & Lopez, A. (2024) · Energy and AI · doi:10.1016/j.egyai.2024.100431 · Cited by 34
Jeon, S., & Kim, H. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110614 · Cited by 37
Domain-specific chatbots for science using embeddings
Yager, K. G. (2023) · Digital Discovery · doi:10.1039/d3dd00112a · Cited by 35
Flaharty, K. A., Hu, P., Hanchard, S. L., Ripper, M. E., Duong, D., Waikel, R. L., & Solomon, B. D. (2024) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2024.07.011 · Cited by 26
JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3511593 · Cited by 36
The argument for near-term human disempowerment through AI
Dung, L. (2024) · AI & Society · doi:10.1007/s00146-024-01930-2 · Cited by 43
Digital Life Project: Autonomous 3D Characters with Social Intelligence
Cai, Z., Jiang, J., Qing, Z., Guo, X., Zhang, M., Lin, Z., Mei, H., Chen, W., Wang, R., Yin, W., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00062 · Cited by 42
Xu, K., Zhang, K., Li, J., Huang, W. D., & Wang, Y. (2024) · Electronics · doi:10.3390/electronics14010047 · Cited by 38
Performance analysis of large language models in the domain of legal argument mining
Zubaer, A. A., Granitzer, M., & Mitrović, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1278796 · Cited by 37
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 61
Automated Control Logic Test Case Generation using Large Language Models
Koziolek, H., Ashiwal, V., Bandyopadhyay, S., & Chandrika, K. R. (2024) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa61755.2024.10711016 · Cited by 30
Wu, J., He, K., Mao, R., Shang, X., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103283 · Cited by 29
Reasoning Runtime Behavior of a Program with LLM: How Far are We?
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00012 · Cited by 87
Self-Evaluation Guided Beam Search for Reasoning
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.-Y., He, J., & Xie, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1802 · Cited by 27
Self-citationScaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Chen, Z., Zhou, K., Zhang, B., Gong, Z., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.985 · Cited by 30
Fairness-Aware Graph Neural Networks: A Survey
Chen, A., Rossi, R. A., Park, N., Trivedi, P., Wang, Y., Yu, T., Kim, S., Dernoncourt, F., & Ahmed, N. K. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649142 · Cited by 26
Mixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 386
“Tree-of-Thought (ToT) (Yao et al., 2023a) expands on the reasoning process by considering multiple paths of reasoning and self-evaluating choices.”
BackgroundLeveraging Large Language Models for Automatic Smart Contract Generation
Napoli, E. A., Barbàra, F., Gatteschi, V., & Schifanella, C. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00100 · Cited by 29
Zhong, S., Huang, Z., Gao, S., Wen, W., Lin, L., Žitnik, M., & Zhou, P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01258 · Cited by 36
Silva, D. D., Kaynak, O., El-Ayoubi, M., Mills, N., Alahakoon, D., & Manic, M. (2024) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2024.3382962 · Cited by 31
Large Language Models in Psychiatry: Current Applications, Limitations, and Future Scope
Liu, Z., Bao, Y., Zeng, S., Qian, R., Deng, M., Gu, A., Li, J., Wang, W., Cai, W., Li, W., et al. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020046 · Cited by 25
Integrating Graphs With Large Language Models: Methods and Prospects
Pan, S., Zheng, Y., & Liu, Y. (2024) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3332242 · Cited by 25
The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models
Qi, J., Xu, Z., Shen, Y., Liu, M., Jin, D., Wang, Q., & Huang, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.255 · Cited by 48
Knowledge-Infused LLM-Powered Conversational Health Agent: A Case Study for Diabetes Patients
Abbasian, M., Yang, Z., Khatibi, E., Zhang, P., Nagesh, N., Azimi, I., Jain, R., & Rahmani, A. M. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10781547 · Cited by 26
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Sarch, G., Wu, Y., Tarr, M., & Fragkiadaki, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.226 · Cited by 25
Prompt Engineering for Large Language Models in Interventional Radiology
Dietrich, N. T., Bradbury, N., & Loh, C. (2025) · American Journal of Roentgenology · doi:10.2214/ajr.25.32956 · Cited by 26
Critique of impure reason: Unveiling the reasoning behaviour of medical large language models
Sim, S. Z. Y., & Chen, T. (2025) · eLife · doi:10.7554/elife.106187 · Cited by 29
Evaluating Large Language Models for Material Selection
Grandi, D., Jain, Y. P., Groom, A., Cramer, B., & McComb, C. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4066730 · Cited by 28
Token-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1274 · Cited by 31
TRAM: Benchmarking Temporal Reasoning for Large Language Models
Wang, Y., & Zhao, Y. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.382 · Cited by 24
ChatGPT in foreign language lesson plan creation: Trends, variability, and historical biases
Dornburg, A., & Davin, K. J. (2024) · ReCALL · doi:10.1017/s0958344024000272 · Cited by 43
Wang, L., & Shen, Y. (2024) · Electronics · doi:10.3390/electronics13234584 · Cited by 23
Development and validation of the prompt engineering competence scale (PECS)
Gibreel, O., & Arpacı, İ. (2025) · Information Development · doi:10.1177/02666669251336455 · Cited by 27
Application of the Tree-of-Thoughts Framework to LLM-Enabled Domain Modeling
Mercado, J. S., Ma, Q., Cabot, J., Kelsen, P., & Proper, H. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75872-0_6 · Cited by 29
Requirements Satisfiability with In-Context Learning
Santos, S., Breaux, T. D., Norton, T. B., Haghighi, S., & Ghanavati, S. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00025 · Cited by 27
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Steenhoek, B. J., Tufano, M., Sundaresan, N., & Svyatkovskiy, A. (2025) · IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) · doi:10.1109/deeptest66595.2025.00011 · Cited by 72
Exploring the Boundaries of GPT-4 in Radiology
Liu, Q., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M. T., Tinn, R., Sharma, H., Pérez‐García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.891 · Cited by 42
Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models
Zhang, J., Wu, Q., Xu, Y., Cao, C. J., Du, Z., & Psounis, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30178 · Cited by 39
Interpretable Recommendations and Parameter-Grounded LLM Explanations with Multigraph AttentionPreprint
Leng, Y., Ruiz, R. I., & Liu, X. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3696092 · Cited by 22
Lu, Q., Zhu, L., Xu, X., Xing, Z., Harrer, S., & Whittle, J. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00028 · Cited by 38
Jin, A., Ye, Y., Lee, B., & Qiao, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3481641 · Cited by 24
Metadata
- Authors
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 11809-11822
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2305.10601
- OpenAlex
- W7133209886
- Cited by
- 1,158
- References
- 0
- Citations by year
- 2026: 476 · 2025: 320 · 2024: 284 · 2023: 77
- FWCI
- 206.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2239363)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- tree-of-thought, large language models, chain-of-thought prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
