The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Open accessAbstract
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%. Code repo with all prompts: https://github.com/princeton-nlp/tree-of-thought-llm.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Tree of Thoughts (ToT) generalizes chain-of-thought prompting by letting a language model explore, evaluate, and search over multiple intermediate “thoughts” rather than decoding left-to-right. On three tasks challenging GPT-4, ToT substantially improves success rates, including 74% on Game of 24 versus 4% for chain-of-thought.
Key findings
- On Game of 24, GPT-4 with IO prompting achieved 7.3%, CoT 4.0%, and CoT-SC (k=100) 9.0%, while ToT achieved 45% with breadth b=1 and 74% with b=5; best-of-100 CoT reached 49%.
- On Creative Writing, GPT-4 coherency scores averaged 7.56 for ToT versus 6.19 for IO and 6.93 for CoT; in a blind human comparison, ToT was judged more coherent than CoT in 41 cases, CoT more coherent in 21, and similar in 38.
- On 5×5 Mini Crosswords, ToT reached 78% letter-level, 60% word-level, and 20% game-level success, compared with IO at 38.7%, 14%, and 0%, and CoT at 40.6%, 15.6%, and 1%; ToT solved 4/20 games, and an oracle best-state variant solved 7/20.
- With GPT-3.5, ToT still beat CoT and IO on tested tasks, but on Game of 24 GPT-3.5+ToT reached only 19% versus GPT-4+ToT’s 74%; GPT-4 generation with GPT-3.5 evaluation reached 64%, while GPT-3.5 generation with GPT-4 evaluation reached 31%.
Methods
- ToT frames problem solving as search over a tree where each node is a partial solution state; instantiations specify thought decomposition, a thought generator, a state evaluator, and a search algorithm such as BFS or DFS.
- Experiments used off-the-shelf GPT-4 chat completion at temperature 0.7 with no extra training. Game of 24 used 100 hard games indexed 901–1,000 from 4nums.com with BFS keeping b=5 candidates and evaluating thoughts as sure/maybe/impossible; Creative Writing used 100 inputs from random sentences with depth 2, k=5 plans/passages, and voting; Mini Crosswords used 20 games from GooBix with DFS up to 100 steps, confidence aggregation, and pruning.
- Baselines included IO prompting, CoT prompting, CoT self-consistency, and iterative refinement; metrics included task success rates, GPT-4 coherency scores, human blind pairwise comparisons, and letter/word/game success for crosswords.
Limitations
- Stated: ToT may not be necessary for tasks GPT-4 already excels at; the paper explores only three relatively simple tasks that challenge GPT-4; ToT requires more resources and API cost than sampling methods; and the work uses an off-the-shelf LM without fine-tuning.
- Stated/evident: cost and efficiency depend heavily on prompts and search algorithms, and ToT can require 5–100 times more generated tokens than CoT; for Game of 24, ToT used 5.5k completion tokens and cost $0.74 per case versus CoT best-of-100 at 6.7k tokens and $0.47.
- Evident from text: the state evaluator is imperfect—sometimes pruning states for crosswords that are actually solvable, possibly due to rare or obsolete words—and output heuristics can be improved, since oracle best-state selection raised Mini Crosswords game success from 4/20 to 7/20; ablating pruning or backtracking generally worsened performance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
32 Own interpretation · 21 Own result · 20 Sources not found · 4 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5(a) shows average GPT-4 scores across 100 tasks, where ToT (7.56) is deemed to generate more coherent passages than IO (6.19) and CoT (6.93) on average.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
- Rests on an assumption: The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, around 60% of CoT samples already failed the task after generating the first step, or equivalently, the first three words (e.g.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we find that GPT-4 can follow the input constraints most of the time, we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“We propose three tasks that are hard even when sampling from the state-of-the-art language model, GPT-4 [23], using standard IO prompting or chain-of-thought (CoT) prompting.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4 Experiments
What the sources say
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Game of 24 (we changed 1-shot proposal prompt to 3-shot to make it work), GPT-3.5+ToT's19% is far worse than GPT-4+ToT's 74%.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Running ToT requires significantly more computations than IO or CoT prompting.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the performance of ToT is better than best of 100 independent CoT trials.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in Game of 24 (Table 7 below), solving a problem with ToT requires 5.5k completion tokens, close to 100 CoT trials (6.7k tokens).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 33
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it could actually find the correct solution for 4/20 games (though only outputting 1 via heuristic), 3 of which are games ToT+pruning cannot solve within 100 steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 2, IO, CoT, and CoT-SC prompting methods perform badly on the task, achieving only 7.3%, 4.0%, and 9.0% success rates.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, ToT with a breadth of b = 1 already achieves a success rate of 45%, while b = 5 achieves 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not surprisingly, CoT scales better than IO, and best of 100 CoT samples achieve a success rate of 49%, but still much worse than exploring more nodes in ToT (b > 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaindirectn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the former, we sample 5 scores and average them for each task output, and we find these 5 scores usually consistent, with a standard deviation of around 0.56 on average across outputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT > ToT Similar ToT > CoT 0 10 20 30 40 21 38 41 (b) Human coherency comparison”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The subset of authors used for human judgments is representative of human readers.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, search methods like ToT requires more resources (e.g.GPT-4 API cost) than sampling methods in order to improve task performances”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To further understand the importance of generation vs. evaluation, we ran GPT-4 generation + GPT-3.5 evaluation (64%) and GPT-3.5 generation + GPT-4 evaluation (31%).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“So completing Game of 24 and Creative Writing's main ToT experiments cost around 0.74 × 100 + 0.32 × 100 = 106 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Creative Writing, we find GPT-3.5+ToToutperform GPT-4+IO, and similar to GPT-4+CoT”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 4 and as expected, ToT improves over CoT on both tasks (but only slightly, given GPT-4 + CoT is already very good on such tasks, and StrategyQA's bottleneck is external knowledge, not reasoning).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, cost and efficiency of ToT highly depend on the prompts and search algorithms used, and could require 5-100 times more generated tokens than CoT.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Convenience.No extra training is needed, just a pre-trained LM is sufficient.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Originally designed to generate text, scaled-up versions of language models (LMs) such as GPT [25,26,1,23] and PaLM [5] have been shown to be increasingly capable of performing an ever wider range of tasks requiring mathematical, symbolic, commonsense, and knowledge reasoning.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [25]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT frames any problem as a search over a tree, where each node is a state s = [x, z 1•••i ] representing a partial solution with the input and the sequence of thoughts so far.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3 Tree of Thoughts: Deliberate Problem Solving with LM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generality.IO, CoT, CoT-SC, and self-refinement can be seen as special cases of ToT (i.e.trees of limited depth and breadth; Figure 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Chain-of-thought (CoT) prompting [38] was proposed to address cases where the mapping of input x to output y is non-trivial (e.g. when x is a math question and y is the final numerical answer).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“RAP [9] is a concurrent work that treats language model reasoning as planning with its internal world model, and proposes a MCTS-based method similar to ToT.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Again, the goal is not just to solve the task, as more general crosswords can be readily solved with specialized NLP pipelines [34] that leverages large-scale retrieval instead of LM.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
What the sources say
- Not matchedcites [34]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks require deductive, mathematical, commonsense, lexical reasoning abilities, and a way to incorporate systematic planning or search.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Research on”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the latter, we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Newell and colleagues characterized problem solving [21] as search through a combinatorial problem space, represented as a tree.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“CoT-SC improves upon CoT, because there are generally different thought processes for the same problem (e.g.different ways to prove the same theorem), and the output decision can be more faithful by exploring a richer set of thoughts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, within each chain there is no local exploration of different thought steps, and the”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed ToT approach extends existing planning formulations by considering multiple potentially feasible plans simultaneously at each problem-solving step, and proceeding with the most promising ones.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT actively maintains a tree of thoughts, where each thought is a coherent language sequence that serves as an intermediate step toward problem solving (Table 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The simple associative token-level choices of LMs are also reminiscent of”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, traditional decision-making procedures usually require training dedicated reward and policy models as in reinforcement learning (for example CHAI [33]), whereas we use the LM itself to provide the value estimates for decision making.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [33]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we introduce a new framework for language model inference,”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is perhaps surprising that underlying all this progress is still the original autoregressive mechanism for generating text, which makes token-level decisions one by one and in a left-to-right fashion.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modularity.The base LM, as well as the thought decomposition, generation, evaluation, and search procedures can all be varied independently.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applicable, such a deliberate heuristic can be more flexible than programmed rules, and more sample-efficient than learned models.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Adaptability.Different problem properties, LM capabilities, and resource constraints can be accommodated.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Similar to our method, self-eval decoding also follows a tree-search procedure with leaves sampled from stochastic beam search decoding, which are then evaluated by LLM itself with carefully prepared self-eval prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“This approach however differs from ours in that trees are expanded by sampling external paragraphs instead of the LM's own thoughts, and there is no reflection or voting steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but the modular flexibility of ToT allows users to customize such performance-cost tradeoffs”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“From this perspective, our method is also related to NeuroLogic A*esque decoding [19], which is inspired by A* search but introduces look-ahead heuristics that are efficient for LMs to improve the beam-search or top-k sampling decoding.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“For example, Schlag et al. [27] embeds LMs in an algorithmic search procedure to help solve problems like question answering step-by-step, in which the search trees are expanded by relevant paragraphs that might provide answers.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT is a framework that empowers LMs to more autonomously and intelligently make decisions and solve problems.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Crosswords' DFS experiments should be also within 100 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“as an initial step this work only explores three relatively simple tasks that challenges GPT-4”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This method however is constrained to sentence generation tasks, whereas our framework are designed for complex, multi-step problem solving guarded by value feedback.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“For example it can be considered as a heuristic search algorithm like A* [10], in which the heuristic at each search node is provided by the LM's selfassessment.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, LMs address a weakness of these classical methods, providing a way to solve complex problems that are not easily formalized, such as creative writing.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Their approach however, uses the PAL formulation [8] which represents thoughts as codes, which makes it difficult to tackle challenging tasks like creative writing which we consider in this paper.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, ToT also improves the interpretability of model decisions and the opportunity for human alignment, as the resulting representations are readable, high-level language reasoning instead of implicit, low-level token values.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“However, its tasks are simpler than ours, and its framework lacks the modularity to incorporate different tree search algorithms.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Deliberate search such as ToT might not be necessary for many existing tasks that GPT-4 already excels at (see Appendix B.1)”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Tree of Thoughts framework provides a way to translate classical insights about problem-solving into actionable methods for contemporary LMs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our Tree-of-Thought formulation is thus more versatile and handles challenging tasks on which GPT-4 only achieves very low accuracy with standard prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Another approach, LLM+P [18], goes one step further and delegates the actual planning process to a classical planner.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The authors state this as an assumption.
The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
“Such valuations do not need to be perfect, and only need to be approximately helpful for decision making.”
The paper relies on this without support.
The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
“we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
The paper relies on this without support.
The subset of authors used for human judgments is representative of human readers.
“we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, these modules maintain internal states for reasoning to ensure the coherence of thought (Yao et al., 2023; Sun et al., 2025c);”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although augmentations such as the ReAct framework [136] or Tree-of-Thoughts [164] introduce pseudo-recursive reasoning, they remain fundamentally heuristic and lack true internal models of time, causality, or state evolution.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, many early language agents simply use LLMs to propose an action (Schick et al., 2023), a sequence of actions (Huang et al., 2022b), or evaluate a fixed set of actions (Ahn et al., 2022) without intermediate reasoning or retrieval. Followup work (Yao et al., 2022b; Shinn et al., 2023; Xu et al., 2023b; Lin et al., 2023; Wang et al., 2023a; Park et al., 2023) has exploited intermediate”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tree of Thoughts (ToT) (Yao et al., 2023) can be seen as a special kind of language agent with only one external action: submitting a final solution to a reasoning problem (game of 24, creative writing, crosswords puzzle).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM reasoning methods fix a search budget by specifying a depth of reasoning (Yao et al., 2023)”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The performance of LLMs is highly sensitive to the quality of their prompts.”
From HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems (2025) · cites this paper as Yao et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The remaining difference is likely due to the fact that ToT implements a classical breadth-first search algorithm, only prompting the LLM to generate (much easier) intermediate steps and heuristic evaluations rather than full solutions and reflections.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Per problem, 100 CoT prompts costs $0.47, but running Tree of Thoughts (ToT) averages $0.74-cost-equivalent to about 150 CoT prompts.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompting is effective in mitigating hallucination by guiding them towards solutions more preferred by humans, which are possibly the ones with lower complexities, and within LLMs' capabilities.”
From Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) · cites this paper as [79]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) reasoning (Yao et al., 2023; Besta et al., 2024) extend single-agent reflection through structured deliberation”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It achieves an accuracy of 0.800(±0.0616),significantly outperforming the previous state-of-the-art method, TOT evaluated with GPT-4, with an accuracy of 0.675 (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For Game of 24, we can easily verify the answer by checking whether the generated answer is 24.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Comparing with CoT-SC, which generates all the planed steps together, ToT needs to query LLMs for each reasoning step.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts (ToT) [50] is designed to generate plans using a tree-like reasoning structure.In this approach, each node in the tree represents a”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Early approaches extend CoT into treeor graph-structured representations, such as tree-of-thoughts (ToT) [254] and graph-ofthoughts (GoT) [255], enabling LLMs to branch, evaluate, and backtrack across multiple reasoning paths.”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“LongCoT offers two principal advantages over traditional CoT: (1) it enables iterative self-correction during reasoning rather than one-shot generation, and (2) it allows for structured, multi-step exploration akin to advanced search algorithms such as ToT [254].”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [136], Chain-of-Thought (CoT) prompting [163], and Tree of Thoughts [164].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
The cited source says what the claim attributes to it.
In the citing paper
“While the earliest agents used LLMs to directly select or generate actions (Figure 1B; Ahn et al., 2022; Huang et al., 2022b), more recent agents additionally use them to reason (Yao et al., 2022b), plan (Hao et al., 2023; Yao et al., 2023), and manage long-term memory (Park et al., 2023; Wang et al., 2023a) to improve decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“During planning, ToT iteratively proposes, evaluates, and selects 'thoughts' (reasoning actions) based on LLM reasoning, and maintains them via a tree search algorithm to enable global exploration as well as local backtrack and foresight.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most language agents use language-only models for decision-making (Yao et al., 2022b; Wang et al., 2023a; Yao et al., 2023), employing a separate captioning model to convert environment observations to text when necessary (Ahn et al., 2022; Zeng et al., 2022).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Initial works have implemented classical planning and tree search (Yao et al., 2023; Hao et al., 2023; Liu et al., 2023a; Dagan et al., 2023), using toy tasks such as game of 24 or block building.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“A number of tasks that LLMs are evaluated on (e.g. writing prompts [52], constrained text generation [28], toxicity control [49, 14] , etc.) are problems without a well-defined ground truth.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“On the exact same test set, we extend our experiments to 150 (direct, non-CoT) queries with a sound verifier, and we reach 70%, comparable to ToT's reported 74%.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“Our results on the Game of 24 setting seem to contradict the results shown in [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“By reducing compositionality and offloading it to a proven classical algorithm, ToT sidesteps some of the major hurdles to LLM reasoning.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“The landmark paper by Yao et al. (2023) introducing the method has received significant attention in the community as it described a significant improvement and also highlights shortcomings of previous methods.”
From Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent studies like ToT [228], Plan-and-Solve [229] and ReAct [230] have proposed more advanced techniques for prompt design to assist in guiding LLM to engage in deeper thinking and tool planning.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [228]
The cited source says what the claim attributes to it.
In the citing paper
“It should be noted DFSDT shares a similar idea to a concurrent work: tree-of-thought (ToT) reasoning (Yao et al., 2023).”
From Counterfactually Auditable Lifecycle Certification for Autonomous Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [138], Chain-of-Thought (CoT) prompting [165], and Tree of Thoughts [166].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [166]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, Monte Carlo Tree Search (MCTS) [24] and other treebased search algorithms has been demonstrated effective for LLM to improve planning and reasoning [25-29].”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“It is now widely known and accepted that contemporary LLMs such as GPT-4 can exhibit substantial improvements in reasoning and generation capability when additional compute is devoted to inference-time strategies (e.g., chain-ofthought prompting (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022), tree-of-thought expansions (Yao et al., 2023; Long, 2023), minimum Bayes risk decoding (Suzgun”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, these expansions come at the cost of increased computational overhead per test instance (Yao et al., 2023).”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For a fair comparison, we also evaluate the TOT implementation by Yao et al. (2023) with GPT-4-Turbo.It yields an accuracy of 0.668.”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The choice of Mini Crosswords for this analysis is strategic, as it highlights how the algorithmic structure of the solvers, such as the tree search employed by TOT, significantly influences their performance (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts explores and evaluates alternative reasoning paths (Yao et al., 2023)”
From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Yao et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, this is mainly because the self-verification setting presented in the main text of this paper is not directly comparable to that of [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While ToT improved accuracy (64.55% vs. 62.52%), this was still far from GPT-4o's zero-shot performance of 94.00%”
From Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“if V (p θ , {s ′ })(s) > v thres then ▷ pruning”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“V (p θ , S)(s) ∼ p value θ (v|s) ∀s ∈ S”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- S. A. Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1): 3, 1996.
- K. E. Stanovich. Who is rational? Studies of individual differences in reasoning. Psychology Press, 1999.
Cited by
Sorted by citations
Items 901–1,000 of 1,294
LLM Thought Divergence and Convergence for Dialogue-Based Image Generation Control
Li, H. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01682
ICL-Agent: Improving LLM Agent Capabilities via a Novel In-Context Learning Approach
Xie, C., Sun, Y. W., & Hu, B. (2026) · International Conference on Electronics Technology and Artificial Intelligence (ETAI) · doi:10.1109/etai68332.2026.11485338
Manasa, G., Bajoria, O., Chandra, S., et al. (2026) · International Conference on Emerging Systems and Intelligent Computing (ESIC) · doi:10.1109/esic68176.2026.11496344
Enhancing TV APPs Authoring With LLM Agents
Borges, P. V., Moraes, D. D. S., Barbosa, S. D. J., & Colcher, S. (2026) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2026.3689310
Target Recommendation for Home AI Device Control Commands
Madhumitha, S. D., Jeevakumari, S. A. A., & Mandal, A. K. (2026) · International Conference on Recent Advances in Electrical, Electronics, Ubiquitous Communication, and Computational Intelligence (RAEEUCCI) · doi:10.1109/raeeucci67649.2026.11504948
CoT-F: Leveraging Chain-of-Thought Families in Large Language Models for Complex Question Answering
Zhao, F., Liu, X., Wang, L., Zhao, R., Yang, Y., & Xu, G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0369-7_35
Consensus-on-Graph: Plan-Driven Exploration and Consensus Decision-Making on Knowledge Graphs
Hu, C., Zhan, B., Yuan, L., Fan, W., & Yan, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0369-7_32
Zhao, R., Zhao, F., & Xu, G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0369-7_34
RE-GPS: Reflective Evolutionary Gradient Prompting System for Large Language Models
Kulin, N., Zhuravlev, V., Khairullin, A., & Muravyov, S. (2026) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct70069.2026.11506566
Requirement Mapping for Complex Equipment: An LLM-Enhanced GQFD Method
Feng, X., Zhao, G., Zhang, L., et al. (2026) · doi:10.1145/3806262.3806269
LEAF-SQL: Level-Wise Exploration with Adaptive Fine-Graining for Text-to-SQL Skeleton Prediction
Tan, Z., Liu, X., Shu, Q., Wan, Q., Liu, D., & Wan, C. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00181 · Cited by 1
CF-CoT: Enhancing Causal Reasoning in LLMs through Counterfactual Scenarios
Yang, J., Qi, B., Liu, L., Li, L. Y., & An, N. (2026) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2026-99-0728
Agent Evolution: Capability-Mechanism Co-evolution
Chen, Y.-F., Zhang, C., Zheng, X., & Tang, S. (2026) · 2026 International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519194
TCG-CTG: A Thinking-based Constraint Guidance Dataset for Controllable Text Generation
He, Y., Zhuang, Y., & Sun, Y. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519175
Complex Task Planning for Large Language Models via Text-Code Collaborative Reasoning
Tang, C., Long, Z., Wang, X., Wang, Z., & Li, G. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519234
STAR: A Stage-Attributed Triage and Repair Framework for RCA Agents in Microservices
Wang, J., Chen, R., Wang, Y., Liao, X., Fan, Z., Wu, X., Wu, W., Zhao, J., & Liu, X. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00057 · Cited by 2
Self-citationEfficient Table QA via TableGrid Navigation and Progressive Inference Prompting
Maurya, A., Singh, N., Javed, M., & Moured, O. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36039-7_31
Zhong, Y., Chen, J., Huang, C., Wang, X., Chen, Y., & He, T. (2026) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2026.3696968 · Cited by 1
Zheng, W., Yan, L., Chen, L., & Li, Q. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3697912
Benzarti, S., & Berrabah, C. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-16638-8_11
A Review of Large Language Models for Stock Price Forecasting from a Hedge-Fund Perspective
Zhang, O., & Zhang, Z. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536649
(15128), J. L., Zhang, Y., Ning, T., Zhao, S., & Teng, Y. (2026) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp69961.2026.11540650
Diamond of Thoughts: Synergistic Merging for Efficient Large Language Model Reasoning
Zhang, Y. (2026) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp69961.2026.11540838
Imbuing Large Language Models with Bidirectional Logic for Robust Chain Repair
Cheng, Z., (95957), W. D., Sun, J., & Lukasiewicz, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37670-1_34 · Cited by 1
Wu, H., Sun, C., Wang, C., Lu, C., Chen, Y., Wang, J., & Zhang, Z. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3700652 · Cited by 1
Li, Z., Li, G., & Shang, M. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133107
Adaptive Beam Search with Shannon Entropy for Data-Centric Reasoning in LLMs
Kim, Y., Jeong, Y., Kim, J., & Cho, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_43
Yin, J., Hines, J., Palash, M. R., Oral, S., Shankar, M. A., & Wang, F. (2026) · ISC High Performance (ISC) · doi:10.23919/isc.2026.11520499
Ballari, A., Assellaou, M., & Souissi, O. (2026) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset68627.2026.11538829
Gen-Plan: An Automated Framework for Generalized Planning in Finance Agent
Nguyen, D. K., Nguyen, M. H. N., & Tran, A. N. (2025) · Conference on Digital Economy and Fintech Innovation (DEFI) · doi:10.1109/defi67526.2025.11551640
Li, J. G., Faragó, D., Petrov, C., & Ahmed, I. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3701958
Memory Bank Compression for Continual Adaptation of Large Language Models
Katraouras, T., & Rafailidis, D. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779742
Agentic AI Frameworks: Orchestration Architectures and Autonomous LLM-Based Reasoning Systems
Yashwanth, L., Rishwanth, S., & Shiny, R. M. (2026) · International Conference on Integrated Intelligence and Communication Systems (ICIICS) · doi:10.1109/iciics67880.2026.11483602
Tang, Y., Liu, Z., Chen, Z., Yang, J., Xiang, L., Xu, Z., & He, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9875246/v1
Evaluation of AI-Generated Image Quality for Selected Artistic Styles
Kopczyński, M., & Ogar, J. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27927-9_20
Petridis, S., Liu, M. X., Fiannaca, A. J., Cai, C. J., & Terry, M. A. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3812956 · Cited by 5
[R.H.J.M.K. and S.M.H. contributed equally to this work.]
Zöller, N., Berger, J., Lin, I., Fu, N., Komarneni, J., Barabucci, G., Laskowski, K., Shia, V., Harack, B., Chu, E., et al. (2025) · MPG.PuRe (Max Planck Society)
Exploring AI-Generated Code for Health Monitoring Modeling in Complex Industrial Systems
Zhao, C., Sun, Y., Jiang, W.-M., Yang, Y., Liu, L.-L., & Gao, Z.-G. (2025) · International Conference on Mechanical Engineering and Aerospace Engineering (MEAE) · doi:10.1109/meae68077.2025.11557847
Intent-Driven Network Optimization Copilot with a Dual-Check Cognitive Agent
Chen, X., Wu, Y., Wang, N., et al. (2026) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc65185.2026.11555610
Liu, Q., Ding, Y., Liu, H., Xu, Z. Y., & Zou, Y. (2026) · Scientific Reports · doi:10.1038/s41598-026-56518-1
Rifat, F. Y., Tabashum, T., Krzak, J. J., Kruger, K. M., Graf, A., Chafetz, R. S., Davids, J. R., Bagley, A. M., Sienko, S. E., Bauer, J., et al. (2026) · doi:10.2196/preprints.103165
InScribe: Intelligent Augmentation of Natural-Language Statements with Data Facts
Chen, C., Liu, Y., Shi, D., Cao, S., & Cao, N. (2026) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis68791.2026.00021
Research Helper - Agentic AI System For Academic Writing
Jain, S., Chaudhary, R., Dhangar, S., et al. (2026) · doi:10.1109/icici68867.2026.11565032
A Large Language Model Enhanced Fault Diagnosis Framework for Chemical Processes
Liang, J., & Sin, G. (2026) · Systems and Control Transactions · doi:10.69997/sct.118894 · Cited by 1
Shehab, M. A., Omari, S., Wardat, M., & Jararweh, Y. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09766-3
A Systematic Review of Agentic AI in Healthcare: An Evidence-Informed Seven-Principle Framework
Prakash, C., Sisodia, A., & Lind, M. (2026) · Journal of Computing Theories and Applications · doi:10.62411/jcta.16171 · Cited by 2
Farea, A. A. R., Liu, Y., Alshami, F., & Ayoubi, M. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08599-y
Divya, T., Prashanthi, V., Srikanth, V., & Venu Gopalachari, M. (2026) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca69024.2026.11570324
Lee, S., & Lee, D.-G. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115378
Shi, F., & Zhang, D. (2026) · International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA) · doi:10.1109/aiita69518.2026.11567367
Jiang, Q., Zhang, L., & Gao, M. (2026) · International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA) · doi:10.1109/aiita69518.2026.11567329
CRRC: A Prompt Engineering Approach for Logical Reasoning in Large Language Models
He, B., Sun, J., Xin, H., Cui, R., & Wu, L. (2026) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc69976.2026.11560566
Tanaka, D., NOMAGUCHI, Y., & Fujita, K. (2026) · Design Science · doi:10.1017/dsj.2026.10061
MAction-SocialNav: Multi-Action Socially Compliant Navigation via Reasoning-Enhanced Prompt Tuning
Wang, Z., Zhang, X., Liu, Z., et al. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3706951
Liu, C., Yang, C., & Ma, J. (2026) · doi:10.1109/aiita69518.2026.11567142
Xu, Z., Sheng, Z., Chen, S., & Chen, T. (2026) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2026.105819
Beyond Reactive Agents: Uncertainty-Gated Meta-Reasoning for Tool-Augmented Decision-MakingPreprint
Zhang, Y., Goel, D., Ahmad, H., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6997675
E-TTS: A New Embodied Test-Time Scaling Framework for Robotic Manipulation
Ye, W., Li, P., Yuan, T., Xu, Y., Wu, X., Zhao, C., Liu, J., Liu, N., Huang, Y., & Wang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36839-3_22
Lee, T., Sim, H. S., & Goh, Y. K. (2026) · IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE) · doi:10.1109/iscaie68866.2026.11576374
Liu, Z., Zhang, Y., He, Y. Q., Paasio, H., Li, C., Semjonova, G., & Xiao, Y. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816771
A Large Language Model-Enabled Multi-Agent Collaboration Method for Complex Task Solving
Wang, S., Feng, Y., & Fang, X. (2026) · International Symposium on Computer Technology and Information Science (ISCTIS) · doi:10.1109/isctis70043.2026.11572301
Zhang, J., Tang, J., Li, S., Wang, K., & Wang, T. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8525-0_12
KORAL: Knowledge Graph Guided LLM Reasoning for SSD Operational Analysis
Akewar, M., Madireddy, S., Luo, D., & Bhimani, J. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00085 · Cited by 3
ConCise: Training-Free Conclusion-Chain State Compression for Cost-Efficient Multi-Step RAG Services
Yan, K., Tang, Z., Wang, T., & Weijia Jia, B. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00092
From surrogate modelling to AI: how machine learning is transforming building simulation
Evins, R. (2026) · Journal of Building Performance Simulation · doi:10.1080/19401493.2026.2694582
Li, C., & Yang, L. (2026) · Electronics · doi:10.3390/electronics15132863
Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2
Land, J. (2026) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd69640.2026.11637368
Li, S., Ding, Y., Li, Y., Huang, M., Wu, X., Xu, W., Wang, Y., Xu, J., & Zhou, Z. (2026) · Journal of Visualization · doi:10.1007/s12650-026-01143-3
Dong, M., Liu, X., Luo, Y., Zeng, Y., Fei, F., & Du, L. C. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00231-0
MMTR: Strategy-Guided Multimodal Table Reasoning with Reflective Self-Correction
Lixin, B., Ming, Y., & Chen, Y. (2026) · Information · doi:10.3390/info17070641
The WHWW Framework: A DIKW-Based Approach for Developing Inquiry Skills in the Age of AI
Chen, Y. L., & Chai, K. K. (2026) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet69978.2026.11585134
From Patterns to Precision: LLM-Guided Detection of Signature Verification Flaws in Smart Contracts
Wang, H., Yan, K., & Diao, W. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00035
SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
Zhou, Q., Yin, P., Zuo, P., Wang, C., & Cheng, J. S.-C. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800528 · Cited by 12
ReShift: Aha-Moment-Driven Reasoning-Level Backdoor Attacks on Vision–Language Models
Dou, Z., Zhao, Q., Gao, Z., & Biswas, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37627-5_9
Perspective Chapter: Monte Carlo for Artificial Intelligence, Foundations and Emerging Trends
Chafi, F. Z. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1016392
Large Language Model-Driven Evolutionary Optimization with a Hallucination-Resilient Mechanism
Huang, W., Wang, R., Zhang, T., Qi, S., Fan, S., & Wang, L. (2026) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125876
InferNose: A Bio-Inspired Electronic Nose System Using Chain-of-Thought for Olfactory Reasoning
Lee, C.-H., & Zhen-xin, F. (2026) · IEEE Sensors Journal · doi:10.1109/jsen.2026.3708008
Automatic Prompt Template Construction via Declarative Programming for Solving Math Word Problems
Li, H., Ma, K., & Yang, M. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581607
Stephenson, C. R., Varayil, J. E., Aakre, C. A., Croghan, I. T., & Hurt, R. T. (2026) · Journal of Primary Care & Community Health · doi:10.1177/21501319261455393
Cognitive Dual-Track Framework for Human-AI Collaborative Problem Solving
Yang, T., Zhang, D., Shi, L., Lu, Y., & Tie, Y. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582613
759Chapter 30 Reinforcement Learning in the Next AI Revolution
Balasundaram, E., Sivarethinamohan, R., Николаевна, М. С., Morozyuk, Y. V., & Kuznetsova, E. S. (2026) · De Gruyter eBooks · doi:10.1515/9783111633909-030
Wang, Y., Jia, R., Zhang, W., Lü, W., Jiang, Y., Zhu, J., Wu, S., & Chen, T. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581682
Level-Up: Iterative Optimization of Collaborative Planning and Solving Mechanism for Reasoning
Guo, M., Li, X., & Yao, J. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582627
Cao, T. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123868
Machine learning for chemotherapy decision-making in breast cancer using large language model
Nabi, M. S., Yuden, D., Choden, T. Y., et al. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1738346
A Novel LLM-based Interpretable Intrusion Detection Framework for the Internet of Things (IoT)
An, Y., & Su, H. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581648
A Decoupled Reasoning Model based on Knowledge Graph for Complex Tool Use in LLMs
Feng, J., Yuan, X., Liu, Y., Zhang, J., & Lv, Z. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582471
Gagana, K. G., Peddibhotla, C. P., Deepak, P. J., Pasha, F., & Agarwal, P. R. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-3-032-18849-6_15
Latent Echo Chambers: Token-Efficient Internal Debate via Simulated Multi-Agent Consensus
Bhowmik, R., & Nayak, A. (2026) · International Carpathian Control Conference · doi:10.1109/iccc71363.2026.11593462
Prediction-in-the-Loop: a decision-making paradigm endowing agents with foresight
Liu, Y., Yuan, M., Zhang, L., Zhou, Z., & Liu, L. (2026) · Scientia Sinica Informationis · doi:10.1360/ssi-2026-0130
Composable Trust in Agentic AI: Bridging Architectural Capability and System-Level Assurance
Al-Refai, O., Shahbaz, I., & Hammad, E. (2026) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w70714.2026.00023
Lee, Y., Choi, S., & Lee, S. W. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7079738
Peran, R., Hobor, L., Kovač, M., & Brčić, M. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591581 · Cited by 2
Approaching LLM Alignment using Agents with RAG
Fliahin, V., Turuta, O., & Turuta, O. (2026) · International Workshop of IT-professionals on Artificial Intelligence (ProfIT AI) · doi:10.55056/ceur-ws.org/vol-4164/short6.pdf
Query Mesh: Accelerating Journalistic Discovery from Data to Insights
John, A., YASHWANTH, G. R., Chinmaya, S. C., Vasudha, J., & Sadhana, V. (2026) · Algorithms for intelligent systems · doi:10.1007/978-3-032-12243-8_7
Chen, Y., Zeng, Z., & Hassan, M. S. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1784200
A survey of transformers based on their input modalities
Sharma, P., & Jain, S. K. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07378-9 · Cited by 1
Yuan, N., & Zhao, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2859-1_5
Zhu, H., Wu, J., Ling, X., Luo, T., & Zhao, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2856-0_32
Metadata
- Authors
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 11809-11822
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2305.10601
- OpenAlex
- W7133209886
- Cited by
- 1,158
- References
- 0
- Citations by year
- 2026: 476 · 2025: 320 · 2024: 284 · 2023: 77
- FWCI
- 206.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2239363)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- tree-of-thought, large language models, chain-of-thought prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
