The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Open accessAbstract
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%. Code repo with all prompts: https://github.com/princeton-nlp/tree-of-thought-llm.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Tree of Thoughts (ToT) generalizes chain-of-thought prompting by letting a language model explore, evaluate, and search over multiple intermediate “thoughts” rather than decoding left-to-right. On three tasks challenging GPT-4, ToT substantially improves success rates, including 74% on Game of 24 versus 4% for chain-of-thought.
Key findings
- On Game of 24, GPT-4 with IO prompting achieved 7.3%, CoT 4.0%, and CoT-SC (k=100) 9.0%, while ToT achieved 45% with breadth b=1 and 74% with b=5; best-of-100 CoT reached 49%.
- On Creative Writing, GPT-4 coherency scores averaged 7.56 for ToT versus 6.19 for IO and 6.93 for CoT; in a blind human comparison, ToT was judged more coherent than CoT in 41 cases, CoT more coherent in 21, and similar in 38.
- On 5×5 Mini Crosswords, ToT reached 78% letter-level, 60% word-level, and 20% game-level success, compared with IO at 38.7%, 14%, and 0%, and CoT at 40.6%, 15.6%, and 1%; ToT solved 4/20 games, and an oracle best-state variant solved 7/20.
- With GPT-3.5, ToT still beat CoT and IO on tested tasks, but on Game of 24 GPT-3.5+ToT reached only 19% versus GPT-4+ToT’s 74%; GPT-4 generation with GPT-3.5 evaluation reached 64%, while GPT-3.5 generation with GPT-4 evaluation reached 31%.
Methods
- ToT frames problem solving as search over a tree where each node is a partial solution state; instantiations specify thought decomposition, a thought generator, a state evaluator, and a search algorithm such as BFS or DFS.
- Experiments used off-the-shelf GPT-4 chat completion at temperature 0.7 with no extra training. Game of 24 used 100 hard games indexed 901–1,000 from 4nums.com with BFS keeping b=5 candidates and evaluating thoughts as sure/maybe/impossible; Creative Writing used 100 inputs from random sentences with depth 2, k=5 plans/passages, and voting; Mini Crosswords used 20 games from GooBix with DFS up to 100 steps, confidence aggregation, and pruning.
- Baselines included IO prompting, CoT prompting, CoT self-consistency, and iterative refinement; metrics included task success rates, GPT-4 coherency scores, human blind pairwise comparisons, and letter/word/game success for crosswords.
Limitations
- Stated: ToT may not be necessary for tasks GPT-4 already excels at; the paper explores only three relatively simple tasks that challenge GPT-4; ToT requires more resources and API cost than sampling methods; and the work uses an off-the-shelf LM without fine-tuning.
- Stated/evident: cost and efficiency depend heavily on prompts and search algorithms, and ToT can require 5–100 times more generated tokens than CoT; for Game of 24, ToT used 5.5k completion tokens and cost $0.74 per case versus CoT best-of-100 at 6.7k tokens and $0.47.
- Evident from text: the state evaluator is imperfect—sometimes pruning states for crosswords that are actually solvable, possibly due to rare or obsolete words—and output heuristics can be improved, since oracle best-state selection raised Mini Crosswords game success from 4/20 to 7/20; ablating pruning or backtracking generally worsened performance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
32 Own interpretation · 21 Own result · 20 Sources not found · 4 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5(a) shows average GPT-4 scores across 100 tasks, where ToT (7.56) is deemed to generate more coherent passages than IO (6.19) and CoT (6.93) on average.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
- Rests on an assumption: The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, around 60% of CoT samples already failed the task after generating the first step, or equivalently, the first three words (e.g.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we find that GPT-4 can follow the input constraints most of the time, we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“We propose three tasks that are hard even when sampling from the state-of-the-art language model, GPT-4 [23], using standard IO prompting or chain-of-thought (CoT) prompting.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4 Experiments
What the sources say
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Game of 24 (we changed 1-shot proposal prompt to 3-shot to make it work), GPT-3.5+ToT's19% is far worse than GPT-4+ToT's 74%.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Running ToT requires significantly more computations than IO or CoT prompting.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the performance of ToT is better than best of 100 independent CoT trials.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in Game of 24 (Table 7 below), solving a problem with ToT requires 5.5k completion tokens, close to 100 CoT trials (6.7k tokens).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 33
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it could actually find the correct solution for 4/20 games (though only outputting 1 via heuristic), 3 of which are games ToT+pruning cannot solve within 100 steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 2, IO, CoT, and CoT-SC prompting methods perform badly on the task, achieving only 7.3%, 4.0%, and 9.0% success rates.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, ToT with a breadth of b = 1 already achieves a success rate of 45%, while b = 5 achieves 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not surprisingly, CoT scales better than IO, and best of 100 CoT samples achieve a success rate of 49%, but still much worse than exploring more nodes in ToT (b > 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaindirectn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the former, we sample 5 scores and average them for each task output, and we find these 5 scores usually consistent, with a standard deviation of around 0.56 on average across outputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT > ToT Similar ToT > CoT 0 10 20 30 40 21 38 41 (b) Human coherency comparison”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The subset of authors used for human judgments is representative of human readers.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, search methods like ToT requires more resources (e.g.GPT-4 API cost) than sampling methods in order to improve task performances”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To further understand the importance of generation vs. evaluation, we ran GPT-4 generation + GPT-3.5 evaluation (64%) and GPT-3.5 generation + GPT-4 evaluation (31%).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“So completing Game of 24 and Creative Writing's main ToT experiments cost around 0.74 × 100 + 0.32 × 100 = 106 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Creative Writing, we find GPT-3.5+ToToutperform GPT-4+IO, and similar to GPT-4+CoT”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 4 and as expected, ToT improves over CoT on both tasks (but only slightly, given GPT-4 + CoT is already very good on such tasks, and StrategyQA's bottleneck is external knowledge, not reasoning).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, cost and efficiency of ToT highly depend on the prompts and search algorithms used, and could require 5-100 times more generated tokens than CoT.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Convenience.No extra training is needed, just a pre-trained LM is sufficient.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Originally designed to generate text, scaled-up versions of language models (LMs) such as GPT [25,26,1,23] and PaLM [5] have been shown to be increasingly capable of performing an ever wider range of tasks requiring mathematical, symbolic, commonsense, and knowledge reasoning.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [25]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT frames any problem as a search over a tree, where each node is a state s = [x, z 1•••i ] representing a partial solution with the input and the sequence of thoughts so far.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3 Tree of Thoughts: Deliberate Problem Solving with LM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generality.IO, CoT, CoT-SC, and self-refinement can be seen as special cases of ToT (i.e.trees of limited depth and breadth; Figure 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Chain-of-thought (CoT) prompting [38] was proposed to address cases where the mapping of input x to output y is non-trivial (e.g. when x is a math question and y is the final numerical answer).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“RAP [9] is a concurrent work that treats language model reasoning as planning with its internal world model, and proposes a MCTS-based method similar to ToT.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Again, the goal is not just to solve the task, as more general crosswords can be readily solved with specialized NLP pipelines [34] that leverages large-scale retrieval instead of LM.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
What the sources say
- Not matchedcites [34]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks require deductive, mathematical, commonsense, lexical reasoning abilities, and a way to incorporate systematic planning or search.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Research on”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the latter, we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Newell and colleagues characterized problem solving [21] as search through a combinatorial problem space, represented as a tree.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“CoT-SC improves upon CoT, because there are generally different thought processes for the same problem (e.g.different ways to prove the same theorem), and the output decision can be more faithful by exploring a richer set of thoughts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, within each chain there is no local exploration of different thought steps, and the”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed ToT approach extends existing planning formulations by considering multiple potentially feasible plans simultaneously at each problem-solving step, and proceeding with the most promising ones.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT actively maintains a tree of thoughts, where each thought is a coherent language sequence that serves as an intermediate step toward problem solving (Table 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The simple associative token-level choices of LMs are also reminiscent of”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, traditional decision-making procedures usually require training dedicated reward and policy models as in reinforcement learning (for example CHAI [33]), whereas we use the LM itself to provide the value estimates for decision making.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [33]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we introduce a new framework for language model inference,”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is perhaps surprising that underlying all this progress is still the original autoregressive mechanism for generating text, which makes token-level decisions one by one and in a left-to-right fashion.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modularity.The base LM, as well as the thought decomposition, generation, evaluation, and search procedures can all be varied independently.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applicable, such a deliberate heuristic can be more flexible than programmed rules, and more sample-efficient than learned models.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Adaptability.Different problem properties, LM capabilities, and resource constraints can be accommodated.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Similar to our method, self-eval decoding also follows a tree-search procedure with leaves sampled from stochastic beam search decoding, which are then evaluated by LLM itself with carefully prepared self-eval prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“This approach however differs from ours in that trees are expanded by sampling external paragraphs instead of the LM's own thoughts, and there is no reflection or voting steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but the modular flexibility of ToT allows users to customize such performance-cost tradeoffs”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“From this perspective, our method is also related to NeuroLogic A*esque decoding [19], which is inspired by A* search but introduces look-ahead heuristics that are efficient for LMs to improve the beam-search or top-k sampling decoding.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“For example, Schlag et al. [27] embeds LMs in an algorithmic search procedure to help solve problems like question answering step-by-step, in which the search trees are expanded by relevant paragraphs that might provide answers.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT is a framework that empowers LMs to more autonomously and intelligently make decisions and solve problems.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Crosswords' DFS experiments should be also within 100 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“as an initial step this work only explores three relatively simple tasks that challenges GPT-4”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This method however is constrained to sentence generation tasks, whereas our framework are designed for complex, multi-step problem solving guarded by value feedback.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“For example it can be considered as a heuristic search algorithm like A* [10], in which the heuristic at each search node is provided by the LM's selfassessment.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, LMs address a weakness of these classical methods, providing a way to solve complex problems that are not easily formalized, such as creative writing.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Their approach however, uses the PAL formulation [8] which represents thoughts as codes, which makes it difficult to tackle challenging tasks like creative writing which we consider in this paper.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, ToT also improves the interpretability of model decisions and the opportunity for human alignment, as the resulting representations are readable, high-level language reasoning instead of implicit, low-level token values.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“However, its tasks are simpler than ours, and its framework lacks the modularity to incorporate different tree search algorithms.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Deliberate search such as ToT might not be necessary for many existing tasks that GPT-4 already excels at (see Appendix B.1)”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Tree of Thoughts framework provides a way to translate classical insights about problem-solving into actionable methods for contemporary LMs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our Tree-of-Thought formulation is thus more versatile and handles challenging tasks on which GPT-4 only achieves very low accuracy with standard prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Another approach, LLM+P [18], goes one step further and delegates the actual planning process to a classical planner.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The authors state this as an assumption.
The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
“Such valuations do not need to be perfect, and only need to be approximately helpful for decision making.”
The paper relies on this without support.
The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
“we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
The paper relies on this without support.
The subset of authors used for human judgments is representative of human readers.
“we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, these modules maintain internal states for reasoning to ensure the coherence of thought (Yao et al., 2023; Sun et al., 2025c);”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although augmentations such as the ReAct framework [136] or Tree-of-Thoughts [164] introduce pseudo-recursive reasoning, they remain fundamentally heuristic and lack true internal models of time, causality, or state evolution.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, many early language agents simply use LLMs to propose an action (Schick et al., 2023), a sequence of actions (Huang et al., 2022b), or evaluate a fixed set of actions (Ahn et al., 2022) without intermediate reasoning or retrieval. Followup work (Yao et al., 2022b; Shinn et al., 2023; Xu et al., 2023b; Lin et al., 2023; Wang et al., 2023a; Park et al., 2023) has exploited intermediate”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tree of Thoughts (ToT) (Yao et al., 2023) can be seen as a special kind of language agent with only one external action: submitting a final solution to a reasoning problem (game of 24, creative writing, crosswords puzzle).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM reasoning methods fix a search budget by specifying a depth of reasoning (Yao et al., 2023)”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The performance of LLMs is highly sensitive to the quality of their prompts.”
From HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems (2025) · cites this paper as Yao et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The remaining difference is likely due to the fact that ToT implements a classical breadth-first search algorithm, only prompting the LLM to generate (much easier) intermediate steps and heuristic evaluations rather than full solutions and reflections.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Per problem, 100 CoT prompts costs $0.47, but running Tree of Thoughts (ToT) averages $0.74-cost-equivalent to about 150 CoT prompts.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompting is effective in mitigating hallucination by guiding them towards solutions more preferred by humans, which are possibly the ones with lower complexities, and within LLMs' capabilities.”
From Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) · cites this paper as [79]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) reasoning (Yao et al., 2023; Besta et al., 2024) extend single-agent reflection through structured deliberation”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It achieves an accuracy of 0.800(±0.0616),significantly outperforming the previous state-of-the-art method, TOT evaluated with GPT-4, with an accuracy of 0.675 (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For Game of 24, we can easily verify the answer by checking whether the generated answer is 24.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Comparing with CoT-SC, which generates all the planed steps together, ToT needs to query LLMs for each reasoning step.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts (ToT) [50] is designed to generate plans using a tree-like reasoning structure.In this approach, each node in the tree represents a”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Early approaches extend CoT into treeor graph-structured representations, such as tree-of-thoughts (ToT) [254] and graph-ofthoughts (GoT) [255], enabling LLMs to branch, evaluate, and backtrack across multiple reasoning paths.”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“LongCoT offers two principal advantages over traditional CoT: (1) it enables iterative self-correction during reasoning rather than one-shot generation, and (2) it allows for structured, multi-step exploration akin to advanced search algorithms such as ToT [254].”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [136], Chain-of-Thought (CoT) prompting [163], and Tree of Thoughts [164].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
The cited source says what the claim attributes to it.
In the citing paper
“While the earliest agents used LLMs to directly select or generate actions (Figure 1B; Ahn et al., 2022; Huang et al., 2022b), more recent agents additionally use them to reason (Yao et al., 2022b), plan (Hao et al., 2023; Yao et al., 2023), and manage long-term memory (Park et al., 2023; Wang et al., 2023a) to improve decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“During planning, ToT iteratively proposes, evaluates, and selects 'thoughts' (reasoning actions) based on LLM reasoning, and maintains them via a tree search algorithm to enable global exploration as well as local backtrack and foresight.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most language agents use language-only models for decision-making (Yao et al., 2022b; Wang et al., 2023a; Yao et al., 2023), employing a separate captioning model to convert environment observations to text when necessary (Ahn et al., 2022; Zeng et al., 2022).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Initial works have implemented classical planning and tree search (Yao et al., 2023; Hao et al., 2023; Liu et al., 2023a; Dagan et al., 2023), using toy tasks such as game of 24 or block building.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“A number of tasks that LLMs are evaluated on (e.g. writing prompts [52], constrained text generation [28], toxicity control [49, 14] , etc.) are problems without a well-defined ground truth.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“On the exact same test set, we extend our experiments to 150 (direct, non-CoT) queries with a sound verifier, and we reach 70%, comparable to ToT's reported 74%.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“Our results on the Game of 24 setting seem to contradict the results shown in [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“By reducing compositionality and offloading it to a proven classical algorithm, ToT sidesteps some of the major hurdles to LLM reasoning.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“The landmark paper by Yao et al. (2023) introducing the method has received significant attention in the community as it described a significant improvement and also highlights shortcomings of previous methods.”
From Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent studies like ToT [228], Plan-and-Solve [229] and ReAct [230] have proposed more advanced techniques for prompt design to assist in guiding LLM to engage in deeper thinking and tool planning.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [228]
The cited source says what the claim attributes to it.
In the citing paper
“It should be noted DFSDT shares a similar idea to a concurrent work: tree-of-thought (ToT) reasoning (Yao et al., 2023).”
From Counterfactually Auditable Lifecycle Certification for Autonomous Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [138], Chain-of-Thought (CoT) prompting [165], and Tree of Thoughts [166].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [166]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, Monte Carlo Tree Search (MCTS) [24] and other treebased search algorithms has been demonstrated effective for LLM to improve planning and reasoning [25-29].”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“It is now widely known and accepted that contemporary LLMs such as GPT-4 can exhibit substantial improvements in reasoning and generation capability when additional compute is devoted to inference-time strategies (e.g., chain-ofthought prompting (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022), tree-of-thought expansions (Yao et al., 2023; Long, 2023), minimum Bayes risk decoding (Suzgun”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, these expansions come at the cost of increased computational overhead per test instance (Yao et al., 2023).”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For a fair comparison, we also evaluate the TOT implementation by Yao et al. (2023) with GPT-4-Turbo.It yields an accuracy of 0.668.”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The choice of Mini Crosswords for this analysis is strategic, as it highlights how the algorithmic structure of the solvers, such as the tree search employed by TOT, significantly influences their performance (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts explores and evaluates alternative reasoning paths (Yao et al., 2023)”
From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Yao et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, this is mainly because the self-verification setting presented in the main text of this paper is not directly comparable to that of [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While ToT improved accuracy (64.55% vs. 62.52%), this was still far from GPT-4o's zero-shot performance of 94.00%”
From Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“if V (p θ , {s ′ })(s) > v thres then ▷ pruning”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“V (p θ , S)(s) ∼ p value θ (v|s) ∀s ∈ S”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- S. A. Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1): 3, 1996.
- K. E. Stanovich. Who is rational? Studies of individual differences in reasoning. Psychology Press, 1999.
Cited by
Sorted by citations
Items 401–500 of 1,294
Managerial decision-making and AI: A decision canvas approach
Bagchi, S. N., & Sharma, R. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.12.001 · Cited by 6
Expertise or Hallucination? A Comprehensive Evaluation of ChatGPT's Aptitude in Clinical Genetics
Zhang, Y., Ren, S., Wang, J., Zhan, C., He, M., Liu, X., Wu, R., Zhao, J., Wu, C., Fan, C., et al. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536939 · Cited by 6
Self-citationProspects and technology of embodied intelligent humanoid robots driven by AI large models
Kai, Z. E. N. G., Wang, Y., Tan, H., Fang, Q., Wang, Y., & Yuan, L. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0350 · Cited by 6
Han, Z., Wang, J., Yan, X., Jiang, Z., Zhang, Y., Liu, S., Gong, Q., & Song, C. (2025) · Applied Sciences · doi:10.3390/app15105663 · Cited by 7
Karabıyık, M. A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112556 · Cited by 7
Knowledge-Augmented Zero-Shot Method for Power Equipment Defect Grading with Chain-of-Thought LLMs
Du, J., Li, B., Chen, Z., Shen, L., Liu, P., & Ran, Z. (2025) · Electronics · doi:10.3390/electronics14153101 · Cited by 7
Chain-of-Thought ICL for GeoAI: Breaking the Hardness Barrier
Hawarey, M. (2026) · doi:10.65737/airmcs2026482 · Cited by 7
Abbineni, P., Aldowaish, S., Liechty, C., Noorzad, S., Ghalati, A. G., & Fayazi, M. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420505 · Cited by 11
Prompt Engineering Through the Lens of Optimal Control
Luo, Y., Tang, Y., Shen, C., Zhou, Z., & Dong, B. (2023) · Journal of Machine Learning · doi:10.4208/jml.231023 · Cited by 28
Jia Jie, Y. U., Luo, M., Zhou, H., & Lan, Z. (2023) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid59865.2023.00029 · Cited by 6
The Bias that Lies Beneath: Qualitative Uncovering of Stereotypes in Large Language Models
Babonnaud, W., Delouche, E., & Lahlouh, M. (2024) · Linköping electronic conference proceedings · doi:10.3384/ecp208022 · Cited by 9
Knowledge Mining using Generative AI for Causal Discovery in Electronics Production
Meier, S., Töper, F., Gebele, J., Rachinger, B., Klarmann, S., & Franke, J. (2024) · International Spring Seminar on Electronics Technology · doi:10.1109/isse61612.2024.10604102 · Cited by 6
Zhao, J., Ma, Z., Zhao, H., Zhang, X., Liu, Q., & Zhang, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5615-5_40 · Cited by 7
Self-citationTask Navigator: Decomposing Complex Tasks for Multimodal Large Language Models
Ma, F., Zhou, Y., Zhang, Y., Wu, S., Zhang, Z., He, Z., Rao, F., & Sun, X. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00230 · Cited by 6
“Tipping the Balance”: Human Intervention in Large Language Model Multi‐Agent Debate
Triem, H., & Ding, Y. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1034 · Cited by 9
Causal-driven Large Language Models with Faithful Reasoning for Knowledge Question Answering
Wang, J., Cao, D., Lu, S., Ma, Z., Xiao, J., & Chua, T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681263 · Cited by 6
Single-pass Hierarchical Text Classification with Large Language Models
Schmidt, F., Hammerfald, K., Jahren, H. H., Payberah, A. H., & Vlassov, V. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825412 · Cited by 7
A Systematic Study on the Potentials and Limitations of LLM-assisted Software Development
Michelutti, C., Eckert, J., Monecke, M., Klein, J., & Glesner, S. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852455 · Cited by 6
Collaborative garment design through group chatting with generative industrial large models
Harish, A. R., Yuan, Z., Li, M., Yang, H., & Huang, G. Q. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103366 · Cited by 6
Context and Layers in Harmony: A Unified Strategy for Mitigating LLM Hallucinations
Yu, S., Kim, G., & Kang, S. (2025) · Mathematics · doi:10.3390/math13111831 · Cited by 8
Pang, A., Wang, M., Pun, M.-O., Chen, C. S., & Xiong, X. (2026) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2026.3674284 · Cited by 7
Blanchet, J., & Han, S. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586182.3625119 · Cited by 6
Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries
Ngu, N., Lee, N., & Shakarian, P. (2024) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc59802.2024.00034 · Cited by 6
Building a literature knowledge base towards transparent biomedical AIPreprint
Huang, Y., Han, Z., Luo, X., Luo, X., Gao, Y., Zhao, M., Tang, F., Wang, Y., Chen, J., Li, C., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.22.614323 · Cited by 6
Implementing Financial Regulations Using Large Language ModelsPreprint
Fazlija, B., Ibraimi, M., Forouzandeh, A., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5010694 · Cited by 6
Symbolic Learning Enables Self-Evolving AgentsPreprint
Zhou, W., Ou, Y., Ding, S., Li, L., Wu, J., Wang, T., Chen, J., Wang, S., Xu, X., Zhang, N., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5379114 · Cited by 6
Liu, R., Geng, J., Wu, A. J., Sucholutsky, I., Lombrozo, T., & Griffiths, T. L. (2024) · arXiv · doi:10.48550/arxiv.2410.21333 · Cited by 6
“We mainly evaluated zero-shot and CoT prompts (Appendix A.2), but also experimented with tree-of-thought (Yao et al., 2023, Appendix A.4), robustness to sampling temperature (Appendix A.6), and in-context steering towards reasoning methods more suited for the statistical structure of the task…”
InfluentialSelf-citationFang, J., Peng, Y., Zhang, X., Wang, Y., Yi, X., Zhang, G., Xu, Y., Wu, B., Liu, S., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07407 · Cited by 6
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5
“These agents are equipped with reasoning (Wei et al., 2023a; Yao et al., 2023a), planning (Shen et al., 2023; Valmeekam et al., 2023), decision-making (Yang et al., 2023a; Kang et al., 2023), and long-term memory capabilities (Zhu et al., 2023a; Hu et al., 2023), and they are constructed upon the…”
BackgroundGödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1354 · Cited by 105
CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data
Zhang, Z., Li, X., Gao, Y., & Lou, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.132 · Cited by 10
Leveraging Large Language Models for Institutional Portfolio Management: Persona-Based Ensembles
Abe, Y., Matsuo, S., Kondo, R., & Hisano, R. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825362 · Cited by 8
CLogLLM: A Large Language Model Enabled Approach to Cybersecurity Log Anomaly Analysis
Ren, H., Lan, K., Sun, Z., & Liao, S. (2024) · 2024 4th International Conference on Electronic Information Engineering and Computer Communication (EIECC) · doi:10.1109/eiecc64539.2024.10929078 · Cited by 5
Integration of Generative AI with Human Expertise in HEOR: A Hybrid Intelligence Framework
Srivastava, T. E., Irfan, H., Babiy, V. V., & Swami, S. (2025) · Advances in Therapy · doi:10.1007/s12325-025-03273-w · Cited by 8
Artificial intelligent disobedience: Rethinking the agency of our artificial teammates
Mirsky, R. (2025) · AI Magazine · doi:10.1002/aaai.70011 · Cited by 12
Li, X. Y., Huang, Z., Quan, S., Peng, C., & Ma, X. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01719-x · Cited by 12
Lizarraga, A., Honig, E., & Wu, Y. N. (2025) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.70035 · Cited by 10
Zheng, J., Huang, Z., Wang, X., Qi, X., & Chen, X. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.122559 · Cited by 8
Can Artificial Intelligence help provide more sustainable feed-back?
Puertas, E., & Cano, E. (2024) · Digital Education Review · doi:10.1344/der.2024.45.50-58 · Cited by 5
McPeak, G., Sautmann, A., George, O. S., Hallal, A., Simal, E. A., Schwartz, A. L., Abaluck, J., Ravi, N., & Pless, R. B. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00094 · Cited by 6
Graph of Logic: Enhancing LLM Reasoning with Graphs and Symbolic Logic
Alotaibi, F., Kulkarni, A., & Zhou, D. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825450 · Cited by 5
Performance Characterization of Expert Router for Scalable LLM Inference
Pichlmeier, J., Ross, P., & Luckow, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826121 · Cited by 11
MAST: Multiagent Safe Transformer for Reinforcement Learning
Wei, S., Wang, X., Feng, X., & Yu, H. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3533744 · Cited by 5
Li, C., Luo, J., Jing, F., Han, Y., Ren, H., Zhang, H., & Cheng, W. (2024) · International Conference on Advanced Computer Theory and Engineering · doi:10.1109/icacte62428.2024.10871274 · Cited by 5
On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook
Fan, M., Wang, C., Chen, C., Liu, Y., & Huang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02375-w · Cited by 40
The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition
Chochlakis, G., Potamianos, A., Lerman, K., & Narayanan, S. S. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00041 · Cited by 11
Revise the NLU: A Prompting Strategy for Robust Dialogue System
Rohmatillah, M., & Chien, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446138 · Cited by 7
FI-NL2PY2SQL: Financial Industry NL2SQL Innovation Model Based on Python and Large Language Model
Du, X., Hu, S., Zhou, F., Wang, C., & Nguyen, B. M. (2025) · Future Internet · doi:10.3390/fi17010012 · Cited by 6
Language as a cognitive and social tool at the time of large language models
Borghi, A. M., Livio, C. D., Gervasi, A. M., Mannella, F., Nolfi, S., & Tummolini, L. (2024) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-024-00152-8 · Cited by 9
LIMO: Less is More for Reasoning
Ye, Y., Huang, Z., Xiao, Y., Chern, E., Xia, S., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2502.03387 · Cited by 4
“This can be achieved by augmenting LLMs with methods such as parallel sampling (Brown et al., 2024; Wang et al., 2022; Li et al., 2022) or symbolic tree search (Hao et al., 2023; Chen et al., 2024; Yao et al., 2023) to enhance reasoning ability.”
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10
Utterance Alignment of Language Models for Effective User Simulation in Task-Oriented Dialogues
Luo, X., Wang, J., & Zhang, X. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3525981 · Cited by 4
Yang, L., & Su, R. (2025) · IoT · doi:10.3390/iot6020026 · Cited by 7
Evaluating the Potential of ChatGPT to Support Climate Risk and Adaptation Assessment
Wilby, R. L. (2025) · Climate Resilience and Sustainability · doi:10.1002/cli2.70013 · Cited by 5
Accelerating primer design for amplicon sequencing using large language model-powered agents
Wang, Y., Hou, Y., Yang, L., Li, S., Tang, W., Tang, H. R., He, Q., Lin, S., Zhang, Y., Li, X., et al. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01455-z · Cited by 8
Self-citationThought graph traversal for test-time scaling in chest X-ray VLLMs
Yao, Y., Wen, Z., Tong, Y., Tian, X., Li, X., Ma, X., Xu, D., & Gedeon, T. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113639 · Cited by 8
A survey of large language models in transportation planning: modelling, design and decision-making
Jin, Y., & Ma, J. (2026) · Transportmetrica A Transport Science · doi:10.1080/23249935.2026.2631152 · Cited by 9
Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context
Ruan, K., Wang, X., Hong, J., Wang, P., Liu, Y., & Sun, H. (2026) · Nature Communications · doi:10.1038/s41467-026-70245-1 · Cited by 50
Wang, Z., Moriyama, S., Wang, W.-Y., Gangopadhyay, B., & Takamatsu, S. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3676025 · Cited by 43
Pagel, S., Jirásek, M., & Cronin, L. (2026) · Communications Chemistry · doi:10.1038/s42004-026-01993-w · Cited by 19
Large Language Models for Ontology Engineering: A Systematic Literature Review
Li, J., Garijo, D., & Poveda‐Villalón, M. (2026) · Semantic Web · doi:10.1177/22104968261465514 · Cited by 12
Wu, Q., Zhang, W., Fan, P., Wang, K., Fan, Q., Chen, W., Mao, G., & Letaief, K. B. (2026) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2026.3728865 · Cited by 6
Prompt Engineering ChatGPT for Codenames
Sidji, M., & Stephenson, M. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645591 · Cited by 4
Scaffolding learning: From specific to generic with large language models
Yin, D. S., & Yin, X. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310409 · Cited by 4
Wang, J. (2024) · IEEE International Conference on Intelligent Systems · doi:10.1109/is61756.2024.10705209 · Cited by 4
A Generative Modeling Method for Digital Twin Shop Floor
Wu, Y., Sun, Y., Wen, X., Liu, X., Bao, J., & Wang, S. (2024) · IEEE Internet Computing · doi:10.1109/mic.2024.3522301 · Cited by 5
Jailbreaking large language models: models, origins, and evolution of attacks and defenses
Li, X. H., Wu, J., Zheng, Q., Wang, H., Fan, M. Y., Hu, S., Guo, J., & Ting, L. I. U. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0196 · Cited by 4
UsmleGPT: An AI application for developing MCQs via multi-agent system
Jiang, Z., & Feng, S. (2025) · Software Impacts · doi:10.1016/j.simpa.2025.100742 · Cited by 6
Large Language Models Are Efficient Learners as Zero-Shot Speech Translators
Liu, C., Chen, L., Tang, P., Zhang, W., Li, X., Ghosh, S., Ye, Z., & Yu, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889325 · Cited by 4
Leveraging Large Language Models for Autonomous Cyber Defense: Insights from CAGE-2 Simulations
Mohammadi, H., Davis, J. J., & Kiely, M. (2025) · IEEE Intelligent Systems · doi:10.1109/mis.2025.3568209 · Cited by 11
Comprehensive survey of large models-driving intelligent decision making
Zheng, Y., Wu, T., Xiao, X., & Xu, Z. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113524 · Cited by 5
Reasoning with financial regulatory texts via Large Language Models
Fazlija, B., Ibraimi, M., Forouzandeh, A., & Fazlija, A. (2025) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2025.101067 · Cited by 5
MatchNav: LLM-based enhanced description and instruction matching in vision-and-language navigation
Wang, C., Wei, S., & Juntong, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103444 · Cited by 5
Toward standardization of GenAI-driven agentic architectures for radio access networks
Nezami, Z., Zaidi, S. A. R., Hafeez, M., Xu, J., & Djemame, K. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1621963 · Cited by 5
Sakarvadia, M., Ajith, A., Khan, A., Grzenda, D., Hudson, N., Bauer, A., Chard, K., & Foster, I. T. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.26 · Cited by 9
Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Singh, C., Morris, J. X., Rush, A. M., Gao, J., & Deng, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.384 · Cited by 18
LLM Augmentations to support Analytical Reasoning over Multiple Documents
Yousuf, R. B., Defelice, N., Sharma, M., Xu, S., & Ramakrishnan, N. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826114 · Cited by 6
Chochlakis, G., Pandiyan, N. M., Lerman, K., & Narayanan, S. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887804 · Cited by 15
Dimensionality and dynamics for next-generation artificial neural networks
Wang, G., & Fan, F. (2025) · Patterns · doi:10.1016/j.patter.2025.101231 · Cited by 5
Enhancing math reasoning ability of large language models via computation logic graphs
Zhao, D., Han, D., Wu, J. X., He, Z., Ning, B., Yuan, Y., Li, Y., Wang, C., & Song, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113905 · Cited by 4
The Path to AGI Goes through Embodiment
Tan, C., & Jaiswal, S. (2023) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v1i1.27485 · Cited by 4
ChatGPT4PCG 2 Competition: Prompt Engineering for Science Birds Level Generation
Taveekitworachai, P., Abdullah, F., Dewantoro, M. F., Xia, Y., Suntichaikul, P., Thawonmas, R., Togelius, J., & Renz, J. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645641 · Cited by 5
Flows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 34
Document-Level Event Extraction with Definition-Driven ICLPreprint
Liu, Z., & Luo, Y. (2024) · Preprints.org · doi:10.20944/preprints202408.0899.v1 · Cited by 7
Murray, M. D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4564227 · Cited by 4
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3
“While the Decomposer helps guide the Critic to the right constraints, the Critic’s quality has the most significant impact on performance.”
InfluentialOn the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“Some are scored by humans [37].”
BackgroundMethodInfluentialSWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement
Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2410.20285 · Cited by 3
“When applied to LLMs, these methods demonstrate a trade-off between diversity and quality in text generation (Yao et al., 2023).”
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3
ReTA: Recursively Thinking Ahead to Improve the Strategic Reasoning of Large Language Models
Duan, J., Wang, S., Diffenderfer, J., Sun, L., Chen, T., Kailkhura, B., & Xu, K. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.123 · Cited by 7
Meta-prompting Optimized Retrieval-Augmented Generation
Rodrigues, J., & Branco, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73503-5_17 · Cited by 3
Situated Instruction Following
Min, S. Y., Puig, X., Chaplot, D. S., Yang, T.-Y., Rai, A., Parashar, P., Salakhutdinov, R., Bisk, Y., & Mottaghi, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73030-6_12 · Cited by 8
Enhancing Robot Learning Through Cognitive Reasoning Trajectory Optimization Under Unknown Dynamics
Dong, Q., Wu, T., Zeng, P., Zang, C., Wan, G., & Cui, S. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3558648 · Cited by 3
Vaidya, N. N., Runkler, T. A., Hubauer, T., Haderlein-Hoegberg, V., & Brandt, M. M. (2025) · IEEE Symposium on Computational Intelligence in Natural Language Processing and Social Media (CI-NLPSoMe) · doi:10.1109/ci-nlpsome64976.2025.10970773 · Cited by 4
A survey on the application and research progress of large language models in financial forecasting
Wu, R., & Liu, H. (2025) · AIP Advances · doi:10.1063/5.0274031 · Cited by 5
Camera Control at the Edge with Language Models for Scene Understanding
Buynitsky, A., Ehsani, S., Pallakonda, B., & Mishra, P. (2025) · International Conference on Control, Automation and Robotics (ICCAR) · doi:10.1109/iccar64901.2025.11073044 · Cited by 3
Towards higher quality software vulnerability data using LLM-based patch filtering
Dil, C., Chen, H., & Damevski, K. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112581 · Cited by 4
Using large language models for solving textbook-style thermodynamic problems
Loubet, R., Zittlau, P., Vollmer, L., Hoffmann, M., Fellenz, S., Jirasek, F., Leitte, H., & Hasse, H. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109333 · Cited by 5
LLM-infused multi-module transformer for emotion-aware sentiment analysis in few-shot scenarios
Ahmed, K., Nadeem, M. I., Wang, G., Zuo, F., & Han, Z. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103668 · Cited by 4
Metadata
- Authors
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 11809-11822
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2305.10601
- OpenAlex
- W7133209886
- Cited by
- 1,158
- References
- 0
- Citations by year
- 2026: 476 · 2025: 320 · 2024: 284 · 2023: 77
- FWCI
- 206.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2239363)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- tree-of-thought, large language models, chain-of-thought prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
