The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Open accessAbstract
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%. Code repo with all prompts: https://github.com/princeton-nlp/tree-of-thought-llm.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Tree of Thoughts (ToT) generalizes chain-of-thought prompting by letting a language model explore, evaluate, and search over multiple intermediate “thoughts” rather than decoding left-to-right. On three tasks challenging GPT-4, ToT substantially improves success rates, including 74% on Game of 24 versus 4% for chain-of-thought.
Key findings
- On Game of 24, GPT-4 with IO prompting achieved 7.3%, CoT 4.0%, and CoT-SC (k=100) 9.0%, while ToT achieved 45% with breadth b=1 and 74% with b=5; best-of-100 CoT reached 49%.
- On Creative Writing, GPT-4 coherency scores averaged 7.56 for ToT versus 6.19 for IO and 6.93 for CoT; in a blind human comparison, ToT was judged more coherent than CoT in 41 cases, CoT more coherent in 21, and similar in 38.
- On 5×5 Mini Crosswords, ToT reached 78% letter-level, 60% word-level, and 20% game-level success, compared with IO at 38.7%, 14%, and 0%, and CoT at 40.6%, 15.6%, and 1%; ToT solved 4/20 games, and an oracle best-state variant solved 7/20.
- With GPT-3.5, ToT still beat CoT and IO on tested tasks, but on Game of 24 GPT-3.5+ToT reached only 19% versus GPT-4+ToT’s 74%; GPT-4 generation with GPT-3.5 evaluation reached 64%, while GPT-3.5 generation with GPT-4 evaluation reached 31%.
Methods
- ToT frames problem solving as search over a tree where each node is a partial solution state; instantiations specify thought decomposition, a thought generator, a state evaluator, and a search algorithm such as BFS or DFS.
- Experiments used off-the-shelf GPT-4 chat completion at temperature 0.7 with no extra training. Game of 24 used 100 hard games indexed 901–1,000 from 4nums.com with BFS keeping b=5 candidates and evaluating thoughts as sure/maybe/impossible; Creative Writing used 100 inputs from random sentences with depth 2, k=5 plans/passages, and voting; Mini Crosswords used 20 games from GooBix with DFS up to 100 steps, confidence aggregation, and pruning.
- Baselines included IO prompting, CoT prompting, CoT self-consistency, and iterative refinement; metrics included task success rates, GPT-4 coherency scores, human blind pairwise comparisons, and letter/word/game success for crosswords.
Limitations
- Stated: ToT may not be necessary for tasks GPT-4 already excels at; the paper explores only three relatively simple tasks that challenge GPT-4; ToT requires more resources and API cost than sampling methods; and the work uses an off-the-shelf LM without fine-tuning.
- Stated/evident: cost and efficiency depend heavily on prompts and search algorithms, and ToT can require 5–100 times more generated tokens than CoT; for Game of 24, ToT used 5.5k completion tokens and cost $0.74 per case versus CoT best-of-100 at 6.7k tokens and $0.47.
- Evident from text: the state evaluator is imperfect—sometimes pruning states for crosswords that are actually solvable, possibly due to rare or obsolete words—and output heuristics can be improved, since oracle best-state selection raised Mini Crosswords game success from 4/20 to 7/20; ablating pruning or backtracking generally worsened performance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
32 Own interpretation · 21 Own result · 20 Sources not found · 4 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5(a) shows average GPT-4 scores across 100 tasks, where ToT (7.56) is deemed to generate more coherent passages than IO (6.19) and CoT (6.93) on average.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
- Rests on an assumption: The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, around 60% of CoT samples already failed the task after generating the first step, or equivalently, the first three words (e.g.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we find that GPT-4 can follow the input constraints most of the time, we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“We propose three tasks that are hard even when sampling from the state-of-the-art language model, GPT-4 [23], using standard IO prompting or chain-of-thought (CoT) prompting.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4 Experiments
What the sources say
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Game of 24 (we changed 1-shot proposal prompt to 3-shot to make it work), GPT-3.5+ToT's19% is far worse than GPT-4+ToT's 74%.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Running ToT requires significantly more computations than IO or CoT prompting.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the performance of ToT is better than best of 100 independent CoT trials.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in Game of 24 (Table 7 below), solving a problem with ToT requires 5.5k completion tokens, close to 100 CoT trials (6.7k tokens).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 33
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it could actually find the correct solution for 4/20 games (though only outputting 1 via heuristic), 3 of which are games ToT+pruning cannot solve within 100 steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 2, IO, CoT, and CoT-SC prompting methods perform badly on the task, achieving only 7.3%, 4.0%, and 9.0% success rates.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, ToT with a breadth of b = 1 already achieves a success rate of 45%, while b = 5 achieves 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, IO and CoT prompting methods perform poorly with a word-level success rate less than 16%, while ToT significantly improves all metrics, achieving a word-level success rate of 60% and solving 4 out of 20 games.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not surprisingly, CoT scales better than IO, and best of 100 CoT samples achieve a success rate of 49%, but still much worse than exploring more nodes in ToT (b > 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.1 Game of 24
plaindirectn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the former, we sample 5 scores and average them for each task output, and we find these 5 scores usually consistent, with a standard deviation of around 0.56 on average across outputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT > ToT Similar ToT > CoT 0 10 20 30 40 21 38 41 (b) Human coherency comparison”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The subset of authors used for human judgments is representative of human readers.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, search methods like ToT requires more resources (e.g.GPT-4 API cost) than sampling methods in order to improve task performances”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To further understand the importance of generation vs. evaluation, we ran GPT-4 generation + GPT-3.5 evaluation (64%) and GPT-3.5 generation + GPT-4 evaluation (31%).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“So completing Game of 24 and Creative Writing's main ToT experiments cost around 0.74 × 100 + 0.32 × 100 = 106 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Creative Writing, we find GPT-3.5+ToToutperform GPT-4+IO, and similar to GPT-4+CoT”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 4 and as expected, ToT improves over CoT on both tasks (but only slightly, given GPT-4 + CoT is already very good on such tasks, and StrategyQA's bottleneck is external knowledge, not reasoning).”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, cost and efficiency of ToT highly depend on the prompts and search algorithms used, and could require 5-100 times more generated tokens than CoT.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Convenience.No extra training is needed, just a pre-trained LM is sufficient.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Originally designed to generate text, scaled-up versions of language models (LMs) such as GPT [25,26,1,23] and PaLM [5] have been shown to be increasingly capable of performing an ever wider range of tasks requiring mathematical, symbolic, commonsense, and knowledge reasoning.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [25]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT frames any problem as a search over a tree, where each node is a state s = [x, z 1•••i ] representing a partial solution with the input and the sequence of thoughts so far.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3 Tree of Thoughts: Deliberate Problem Solving with LM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generality.IO, CoT, CoT-SC, and self-refinement can be seen as special cases of ToT (i.e.trees of limited depth and breadth; Figure 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Chain-of-thought (CoT) prompting [38] was proposed to address cases where the mapping of input x to output y is non-trivial (e.g. when x is a math question and y is the final numerical answer).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“RAP [9] is a concurrent work that treats language model reasoning as planning with its internal world model, and proposes a MCTS-based method similar to ToT.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Again, the goal is not just to solve the task, as more general crosswords can be readily solved with specialized NLP pipelines [34] that leverages large-scale retrieval instead of LM.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.3 Mini crosswords
What the sources say
- Not matchedcites [34]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tasks require deductive, mathematical, commonsense, lexical reasoning abilities, and a way to incorporate systematic planning or search.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Research on”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the latter, we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 4.2 Creative writing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Newell and colleagues characterized problem solving [21] as search through a combinatorial problem space, represented as a tree.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
What the sources say
- Not matchedcites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“CoT-SC improves upon CoT, because there are generally different thought processes for the same problem (e.g.different ways to prove the same theorem), and the output decision can be more faithful by exploring a richer set of thoughts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
What the sources say
- Not matchedcites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, within each chain there is no local exploration of different thought steps, and the”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 2 Background
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed ToT approach extends existing planning formulations by considering multiple potentially feasible plans simultaneously at each problem-solving step, and proceeding with the most promising ones.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT actively maintains a tree of thoughts, where each thought is a coherent language sequence that serves as an intermediate step toward problem solving (Table 1).”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The simple associative token-level choices of LMs are also reminiscent of”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the other hand, traditional decision-making procedures usually require training dedicated reward and policy models as in reinforcement learning (for example CHAI [33]), whereas we use the LM itself to provide the value estimates for decision making.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [33]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we introduce a new framework for language model inference,”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is perhaps surprising that underlying all this progress is still the original autoregressive mechanism for generating text, which makes token-level decisions one by one and in a left-to-right fashion.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modularity.The base LM, as well as the thought decomposition, generation, evaluation, and search procedures can all be varied independently.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applicable, such a deliberate heuristic can be more flexible than programmed rules, and more sample-efficient than learned models.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- The authors state this as an assumption: The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Adaptability.Different problem properties, LM capabilities, and resource constraints can be accommodated.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Similar to our method, self-eval decoding also follows a tree-search procedure with leaves sampled from stochastic beam search decoding, which are then evaluated by LLM itself with carefully prepared self-eval prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“This approach however differs from ours in that trees are expanded by sampling external paragraphs instead of the LM's own thoughts, and there is no reflection or voting steps.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but the modular flexibility of ToT allows users to customize such performance-cost tradeoffs”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“From this perspective, our method is also related to NeuroLogic A*esque decoding [19], which is inspired by A* search but introduces look-ahead heuristics that are efficient for LMs to improve the beam-search or top-k sampling decoding.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“For example, Schlag et al. [27] embeds LMs in an algorithmic search procedure to help solve problems like question answering step-by-step, in which the search trees are expanded by relevant paragraphs that might provide answers.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ToT is a framework that empowers LMs to more autonomously and intelligently make decisions and solve problems.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Crosswords' DFS experiments should be also within 100 dollars.”
§ define the answer format of new tasks gsm8k_format = '"the answer is n" where n is a number' strategyqa_format = 'either "the answer is yes" or "the answer is no"' # define zero-shot io prompting standard_prompt = 'Answer the following question with {format}: {input}' # define thought format for zero-shot cot and zero-shot tot cot_prompt = '''Answer the following question: {input} Make a strategy then write.Your output should be of the following format: > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“as an initial step this work only explores three relatively simple tasks that challenges GPT-4”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This method however is constrained to sentence generation tasks, whereas our framework are designed for complex, multi-step problem solving guarded by value feedback.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“For example it can be considered as a heuristic search algorithm like A* [10], in which the heuristic at each search node is provided by the LM's selfassessment.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, LMs address a weakness of these classical methods, providing a way to solve complex problems that are not easily formalized, such as creative writing.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Their approach however, uses the PAL formulation [8] which represents thoughts as codes, which makes it difficult to tackle challenging tasks like creative writing which we consider in this paper.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, ToT also improves the interpretability of model decisions and the opportunity for human alignment, as the resulting representations are readable, high-level language reasoning instead of implicit, low-level token values.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Broader Impact
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“However, its tasks are simpler than ours, and its framework lacks the modularity to incorporate different tree search algorithms.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 5 Related Work
What the sources say
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Deliberate search such as ToT might not be necessary for many existing tasks that GPT-4 already excels at (see Appendix B.1)”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Tree of Thoughts framework provides a way to translate classical insights about problem-solving into actionable methods for contemporary LMs.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 6 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our Tree-of-Thought formulation is thus more versatile and handles challenging tasks on which GPT-4 only achieves very low accuracy with standard prompts.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Another approach, LLM+P [18], goes one step further and delegates the actual planning process to a classical planner.”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > Self-reflection.
What the sources say
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The authors state this as an assumption.
The LM's self-evaluations of states need only be approximately helpful for decision making, not perfect.
“Such valuations do not need to be perfect, and only need to be approximately helpful for decision making.”
The paper relies on this without support.
The GPT-4 zero-shot 1-10 scalar score is a valid measure of passage coherency.
“we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods.”
The paper relies on this without support.
The subset of authors used for human judgments is representative of human readers.
“we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, these modules maintain internal states for reasoning to ensure the coherence of thought (Yao et al., 2023; Sun et al., 2025c);”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although augmentations such as the ReAct framework [136] or Tree-of-Thoughts [164] introduce pseudo-recursive reasoning, they remain fundamentally heuristic and lack true internal models of time, causality, or state evolution.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, many early language agents simply use LLMs to propose an action (Schick et al., 2023), a sequence of actions (Huang et al., 2022b), or evaluate a fixed set of actions (Ahn et al., 2022) without intermediate reasoning or retrieval. Followup work (Yao et al., 2022b; Shinn et al., 2023; Xu et al., 2023b; Lin et al., 2023; Wang et al., 2023a; Park et al., 2023) has exploited intermediate”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tree of Thoughts (ToT) (Yao et al., 2023) can be seen as a special kind of language agent with only one external action: submitting a final solution to a reasoning problem (game of 24, creative writing, crosswords puzzle).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has no long-term memory, and only reasoning in its internal action space, but differs from all previous agents in its deliberate decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most LLM reasoning methods fix a search budget by specifying a depth of reasoning (Yao et al., 2023)”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The performance of LLMs is highly sensitive to the quality of their prompts.”
From HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems (2025) · cites this paper as Yao et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The remaining difference is likely due to the fact that ToT implements a classical breadth-first search algorithm, only prompting the LLM to generate (much easier) intermediate steps and heuristic evaluations rather than full solutions and reflections.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Per problem, 100 CoT prompts costs $0.47, but running Tree of Thoughts (ToT) averages $0.74-cost-equivalent to about 150 CoT prompts.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompting is effective in mitigating hallucination by guiding them towards solutions more preferred by humans, which are possibly the ones with lower complexities, and within LLMs' capabilities.”
From Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) · cites this paper as [79]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) reasoning (Yao et al., 2023; Besta et al., 2024) extend single-agent reflection through structured deliberation”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It achieves an accuracy of 0.800(±0.0616),significantly outperforming the previous state-of-the-art method, TOT evaluated with GPT-4, with an accuracy of 0.675 (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For Game of 24, we can easily verify the answer by checking whether the generated answer is 24.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Comparing with CoT-SC, which generates all the planed steps together, ToT needs to query LLMs for each reasoning step.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts (ToT) [50] is designed to generate plans using a tree-like reasoning structure.In this approach, each node in the tree represents a”
From A survey on large language model based autonomous agents (2024) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“Early approaches extend CoT into treeor graph-structured representations, such as tree-of-thoughts (ToT) [254] and graph-ofthoughts (GoT) [255], enabling LLMs to branch, evaluate, and backtrack across multiple reasoning paths.”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“LongCoT offers two principal advantages over traditional CoT: (1) it enables iterative self-correction during reasoning rather than one-shot generation, and (2) it allows for structured, multi-step exploration akin to advanced search algorithms such as ToT [254].”
From A Survey of Large Language Models (2026) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [136], Chain-of-Thought (CoT) prompting [163], and Tree of Thoughts [164].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [164]
The cited source says what the claim attributes to it.
In the citing paper
“While the earliest agents used LLMs to directly select or generate actions (Figure 1B; Ahn et al., 2022; Huang et al., 2022b), more recent agents additionally use them to reason (Yao et al., 2022b), plan (Hao et al., 2023; Yao et al., 2023), and manage long-term memory (Park et al., 2023; Wang et al., 2023a) to improve decision-making.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“During planning, ToT iteratively proposes, evaluates, and selects 'thoughts' (reasoning actions) based on LLM reasoning, and maintains them via a tree search algorithm to enable global exploration as well as local backtrack and foresight.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most language agents use language-only models for decision-making (Yao et al., 2022b; Wang et al., 2023a; Yao et al., 2023), employing a separate captioning model to convert environment observations to text when necessary (Ahn et al., 2022; Zeng et al., 2022).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Initial works have implemented classical planning and tree search (Yao et al., 2023; Hao et al., 2023; Liu et al., 2023a; Dagan et al., 2023), using toy tasks such as game of 24 or block building.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“A number of tasks that LLMs are evaluated on (e.g. writing prompts [52], constrained text generation [28], toxicity control [49, 14] , etc.) are problems without a well-defined ground truth.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“On the exact same test set, we extend our experiments to 150 (direct, non-CoT) queries with a sound verifier, and we reach 70%, comparable to ToT's reported 74%.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“Our results on the Game of 24 setting seem to contradict the results shown in [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“By reducing compositionality and offloading it to a proven classical algorithm, ToT sidesteps some of the major hurdles to LLM reasoning.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“The landmark paper by Yao et al. (2023) introducing the method has received significant attention in the community as it described a significant improvement and also highlights shortcomings of previous methods.”
From Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent studies like ToT [228], Plan-and-Solve [229] and ReAct [230] have proposed more advanced techniques for prompt design to assist in guiding LLM to engage in deeper thinking and tool planning.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [228]
The cited source says what the claim attributes to it.
In the citing paper
“It should be noted DFSDT shares a similar idea to a concurrent work: tree-of-thought (ToT) reasoning (Yao et al., 2023).”
From Counterfactually Auditable Lifecycle Certification for Autonomous Agents (2023) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [138], Chain-of-Thought (CoT) prompting [165], and Tree of Thoughts [166].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [166]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, Monte Carlo Tree Search (MCTS) [24] and other treebased search algorithms has been demonstrated effective for LLM to improve planning and reasoning [25-29].”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“It is now widely known and accepted that contemporary LLMs such as GPT-4 can exhibit substantial improvements in reasoning and generation capability when additional compute is devoted to inference-time strategies (e.g., chain-ofthought prompting (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022), tree-of-thought expansions (Yao et al., 2023; Long, 2023), minimum Bayes risk decoding (Suzgun”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, these expansions come at the cost of increased computational overhead per test instance (Yao et al., 2023).”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For a fair comparison, we also evaluate the TOT implementation by Yao et al. (2023) with GPT-4-Turbo.It yields an accuracy of 0.668.”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The choice of Mini Crosswords for this analysis is strategic, as it highlights how the algorithmic structure of the solvers, such as the tree search employed by TOT, significantly influences their performance (Yao et al., 2023).”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Yao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Tree of Thoughts explores and evaluates alternative reasoning paths (Yao et al., 2023)”
From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Yao et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, this is mainly because the self-verification setting presented in the main text of this paper is not directly comparable to that of [52].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While ToT improved accuracy (64.55% vs. 62.52%), this was still far from GPT-4o's zero-shot performance of 94.00%”
From Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (2024) · cites this paper as Yao et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“if V (p θ , {s ′ })(s) > v thres then ▷ pruning”
§ Tree of Thoughts: Deliberate Problem Solving with Large Language Models > 3. State evaluator V (p θ , S).
“V (p θ , S)(s) ∼ p value θ (v|s) ∀s ∈ S”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- S. A. Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1): 3, 1996.
- K. E. Stanovich. Who is rational? Studies of individual differences in reasoning. Psychology Press, 1999.
Cited by
Sorted by citations
Items 601–700 of 1,294
AI-Assisted Identification of Policy-Salient Research Priorities and Emerging IssuesPreprint
Rudd, M. A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4502450 · Cited by 2
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 3
A Large Language Model based legal assistant for governance applicationsPreprint
Mamalis, M. E., Kalampokis, E., Fitsilis, F., Theodorakopoulos, G., & Tarabanis, K. A. (2024) · doi:10.31219/osf.io/94fmq · Cited by 2
Knapp, R., Johnson, B., & Busta, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.05.622126 · Cited by 2
Large Language Models in Finance: ReasoningPreprint
Alonso, M. N. I. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5048316 · Cited by 2
LLM for Large-Scale Optimization Model Auto-Formulation: A Lightweight Few-Shot Learning ApproachPreprint
Liang, K., Lu, Y., Mao, J., Sun, S., yang, C., Zeng, C., Jin, X., Qin, H., Zhu, R., & Teo, C. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5329027 · Cited by 4
Simulating Human-like Daily Activities with Desire-driven Autonomy
Wang, Y., Chen, Y., Zhong, F., Ma, L., & Wang, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.06435 · Cited by 2
“Inspired by the work of Tree of Thoughts (Yao et al., 2024), we argue that the paradigm of presenting multiple candidate activities and evaluating them enables the agent to select the best action.”
A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis
Spieser, J., Balapour, A., Meller, J., Patra, K. C., & Shamsaei, B. (2026) · Methods and Protocols · doi:10.3390/mps9020033 · Cited by 5
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
Preble: Efficient Distributed Prompt Scheduling for LLM Serving
Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1
“Such example demonstration and parallel generation technique is common in recent prompt engineering, for example, with ReAct [58], Tree-of-Thoughts [57], and Self Consistency [51].”
BackgroundMethodSkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Li, X., Liu, Y., Chen, W., You, B., Di, Z., He, Y., Zheng, S., Choe, K. W., Sun, J., Wang, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12670 · Cited by 1
“Tests must include informative error messages and use parametrization rather than duplication.”
Robeyns, M., Szummer, M., & Aitchison, L. (2025) · arXiv · doi:10.48550/arxiv.2504.15228 · Cited by 117
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1
“LLMs have made impressive advances in language understanding, dialogue generation, and reasoning, enabled by techniques such as large-scale pretraining [2, 3, 6, 18, 36, 56], chain-of-thought prompting [31, 59, 51, 58, 66], and reinforcement learning with human feedback (RLHF) [4, 43].”
AI and Prompt Architecture – A Literature Review
Ansara, C. (2023) · International Journal of Computer Applications · doi:10.5120/ijca2023923133 · Cited by 1
From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models
Yan, J., Wang, C., Zhang, T., He, X., Huang, J., & Zhang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.828 · Cited by 5
Mejía-Trejo, J. (2024) · doi:10.55965/abib.2024.9786076984505 · Cited by 3
RTaC: A Generalized Framework for Tooling
Bhavsar, N., Thakur, A., Singh, A. L., & Patwa, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_5 · Cited by 1
Are We Keeping up With the Innovation in Generative AI?
Eldh, S. (2024) · IEEE Software · doi:10.1109/ms.2024.3441908 · Cited by 1
İSTEM MÜHENDİSLİĞİNİN BÜYÜK DİL MODELLERİNİN PERFORMANSINDAKİ ROLÜ: ANALİZ VE UYGULAMA ÖRNEKLERİ
Tan, F. G., Yüksel, A. S., & Karabıyık, M. A. (2024) · Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi · doi:10.17780/ksujes.1480838 · Cited by 3
A Systematic Literature Review on Automated Scoring of Mathematical Constructed Responses
Kim, S., & Ha, M. (2024) · Journal of Curriculum and Evaluation · doi:10.29221/jce.2024.27.4.293 · Cited by 2
Automating Resume Analysis: Knowledge Graphs via Prompt Engineering
Lazzarinetti, G., Manzoni, S., & Zoppis, I. F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-80607-0_17 · Cited by 1
The Art of Replication: Lifelike Avatars with Personalized Conversational Style
Nasser, M. M. A. A., Gaglio, G. F., Seidita, V., & Chella, A. (2025) · Robotics · doi:10.3390/robotics14030033 · Cited by 1
Lisboa, M. O., Costa, H., Coura, P., Freitas, I., Villela, M. L. B., & Ferreira, R. (2025) · doi:10.5753/educomp.2025.4927 · Cited by 1
Agent Trajectory Explorer: Visualizing and Providing Feedback on Agent Trajectories
Desmond, M. J., Lee, J. Y., Ibrahim, I., Johnson, J. M., Sil, A., MacNair, J., & Puri, R. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i28.35350 · Cited by 5
Redefining Annotation Practices: Leveraging Large Language Models for Discourse Annotation
Ostyakova, L., Mikhailova, A., & Konovalov, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88036-0_7 · Cited by 1
Viswanathan, G., Samdani, G., & Dixit, Y. (2025) · International Journal of Advanced Information Technology · doi:10.5121/ijait.2025.15202 · Cited by 3
Cost, Complexity, and Efficacy of Prompt Engineering Techniques for Large Language Models
Cherukuri, M. (2025) · International Journal on Science and Technology · doi:10.71097/ijsat.v16.i2.2584 · Cited by 1
Self-Regeneration: Step-Level Regeneration Improves the LLM's Causal Reasoning Ability
Jiang, W., Zhao, S., & Dai, H.-W. (2025) · International Conference on Computer Supported Cooperative Work in Design · doi:10.1109/cscwd64889.2025.11033548
Advancing Table Understanding of Large Language Models via Feature Re-ordering
Wang, G., Chen, Y., Chen, H., Fan, X., Wang, J., Li, X., Hu, M., Chang, C.-Y., & Hu, X. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3748239.3748248 · Cited by 1
Multiagent Systems for Educational Transformation: A Swarm of Intelligent Personal Assistants
Arjona-Giner, S., & Largo, F. L. (2025) · Lecture notes in educational technology · doi:10.1007/978-981-96-5658-5_132 · Cited by 1
Distilling Closed-Source LLM’s Knowledge for Locally Stable and Economic Biomedical Entity Linking
Ai, Y., Ning, Z., Dai, W., Wang, P., Du, Y., Cui, W., Liu, K., & Zhou, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0027-7_9 · Cited by 1
He, G., Zhang, M., Su, T., Ma, L., & Zhu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129273 · Cited by 3
Adaptive-TOD: An LLM-driven and adaptive agent for diverse interaction modes
Li, P., Yang, Q., Xu, S., Li, X., Li, Z., Chen, W., Liu, Y., Guo, T., Tang, J., & Wen, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130991 · Cited by 2
The Hitchhiker’s Guide to Autonomous Research: A Survey of Scientific Agents
Wang, X., Xu, J., Lian, S., Chen, Y., Guo, H., Zhu, F., Shao, Y., Ren, M., Yi, H., Feng, A. H., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3721500 · Cited by 4
Dynamic Asymmetric Attention for Enhanced Reasoning and Interpretability in LLMs
Wen, F., Lu, X., Yu, H., Lu, C., Li, H., & Shi, X. (2025) · Symmetry · doi:10.3390/sym17081303 · Cited by 1
Ensemble ToT and Its Application to Automatic Grading
Ito, Y., & Ma, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02049-9_18 · Cited by 1
LLMAGENTNET: A COLLABORATIVE NETWORK OF AUTONOMOUS AI AGENTS FOR COMPLEX TASK EXECUTION
Бідочко, А., & Виклюк, Я. І. (2025) · Науковий вісник НЛТУ України · doi:10.36930/40350412 · Cited by 3
López, M. Á., & Petrelli, M. (2025) · Artificial Intelligence in Geosciences · doi:10.1016/j.aiig.2025.100153 · Cited by 1
Language as Data: A Survey of Natural Language Processing for Economics and Finance
Lucchetti, A. H., & Cajueiro, D. O. (2025) · Journal of Economic Surveys · doi:10.1111/joes.70014 · Cited by 1
Eliciting problem specifications for LLM-Modulo cognitive systems
Wray, R. E., Kirk, J. R., & Laird, J. E. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101409 · Cited by 5
Hwang, W., Nguyen, Q.-V., & Sun, M.-T. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129939 · Cited by 1
SEER: Self-Enhancing Chain-of-Thought Compression for Reasoning Models
Huang, K., Liu, S., Hu, X., Xu, T., Bao, L., & Xia, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832120 · Cited by 14
User-Oriented Fuzzy Requirement Clarification for Agent Services Delivery Based on LLMs
Zhang, B., Chu, D., Xiong, S., Gong, Q., & Tu, Z. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3624915 · Cited by 1
Opportunities of Large Language Models in the Development of Organic Electrosynthesis
Yu, Z., Yang, Q., & Mo, Y. (2025) · ACS electrochemistry. · doi:10.1021/acselectrochem.5c00377 · Cited by 2
Personalized Reward Modeling for Text-to-Image Generation
Lee, J., Heo, R., & Lee, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37356-4_25 · Cited by 6
Assessing data extraction in randomized clinical trials with large language models
Yisha, Z., Zou, P., Li, S., Zhang, L., Guo, L., Gu, A., Liu, G., Liu, T., & Wang, X. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02729-5 · Cited by 5
EMRDCM : An experience-enhanced multi-role debate method based on compromise mechanisms
Zhang, Y., Feng, J., Yiming, X., Wang, Z., Yang, T., & Wang, X. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01022-1 · Cited by 1
Zeng, Z., Cheng, Q., Hu, X., Li, W., Ding, W., & Liu, Z. (2026) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2025.3647956 · Cited by 4
Two Shots are Enough: Reliable Constrained Generation with LLMs
Mondal, M., Dolamic, L., Cudré-Mauroux, P., & Audiffren, J. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402199 · Cited by 1
From Verification Burden to Trusted Collaboration: Design Goals for LLM-Assisted Literature Reviews
Nogueira, B., Geyer, W., Anderson, A. W., Li, T. J.-J., Kim, D., Moniz, N., & Chawla, N. V. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742414.3794723 · Cited by 1
Agentic Meta-Orchestrator for Multi-Task Copilots
Zhu, X., & Zhou, Y. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00206 · Cited by 1
Are They All Good? Evaluating the Quality of CoTs in LLM-Based Code Generation
Zhang, B., Zhang, L., Luo, Z., Du, Y., Liu, F., Wang, S., & Shi, L. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3676295 · Cited by 10
DARE: A Dialectical Framework for Adversarial and Evidence-Aware RAG
Sadhu, S., Datta, S., & Basu, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_27 · Cited by 1
Adversarial Disillusion Against PGD Attacks via Multimodal Generative Chain-of-Thought Reasoning
Chu, H., Ou, W., Li, X., & Ma, W. (2026) · International Conference on Artificial Intelligence and Social Network Systems (AISNS) · doi:10.1109/aisns67921.2026.11440364 · Cited by 1
Toxicbias-reasoning: a multicultural dataset for social bias detection with human-aligned reasoning
Kumar, A., Gurve, M. K., Ahlawat, S., Prasad, Y., & Singh, V. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09916-w · Cited by 1
VSP: Diagnosing the Dual Challenges of Perception and Reasoning in Spatial Planning Tasks for MLLMS
Wu, Q., Zhao, H., Saxon, M., Bui, T., Wang, W. Y., Zhang, Y., & Chang, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00219 · Cited by 1
TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
Bruches, E. P., Alperovich, V., Baturova, D., Derunets, R., Grebenkin, D., Mkrtchyan, G., Sedukhin, O., Klementev, M., Bondarenko, I., Bushkov, N., et al. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c67878.2026.00057 · Cited by 1
CodeMind: Evaluating Large Language Models for Implicit and Explicit Code Execution Reasoning
Liu, C., Chen, Y., & Jabbarvand, R. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3691250 · Cited by 82
The Silent Scholar Problem: A Probabilistic Framework for Breaking Epistemic Asymmetry in LLM Agents
Chong, Z., Ohsaki, H., & Ng, B. (2026) · doi:10.1109/robothia68364.2026.11506698 · Cited by 1
Xu, Z., & Mao, D. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519251 · Cited by 1
Shen, M.-T., & Joung, Y.-J. (2026) · International Workshop on Agentic Engineering · doi:10.1145/3786167.3788424 · Cited by 1
Huang, Y., Liu, D., & Chao, C.-Y. (2026) · Applied Intelligence · doi:10.1007/s10489-025-07080-2 · Cited by 1
Focus Session: Large Language Models in Physical Design: From Data Generation to Intelligent Agents
Wu, B.-Y., Dey, A., Rovinski, A., & Chhabria, V. A. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539185 · Cited by 1
Kim, Y., Kim, J., Cho, Y., & Choi, M. (2026) · BMC Nursing · doi:10.1186/s12912-026-04949-6 · Cited by 1
Shen, J., Peng, Z., & Owen, L. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00085 · Cited by 2
When the Manual Lies: A Realistic Benchmark to Evaluate MCP Poisoning Attacks for LLM Agents
Liu, S., Tang, X., Yang, X., Liang, L. Y., Zhou, B., Xiao, W., & Liu, W. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582512 · Cited by 3
Bridging Causal Reasoning and Fuzzy Calibration for Trustworthy AI Risk Assessment
McConomy, G., Liu, J., Glass, D., Bond, R., & Toner, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2497-5_18 · Cited by 1
Taricco, G. (2026) · Entropy · doi:10.3390/e28070811 · Cited by 1
Destroy-repair of reasoning chains via adaptive large neighborhood search in small language models
Qasim, K. U., Zhang, J., Aslam, M. W., et al. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123959 · Cited by 1
Agentic AI Safety: A Structured Review of Open Problems and Their Regulatory Anchoring
Valenta, T., Rozinek, O., & Horálek, J. (2026) · AI · doi:10.3390/ai7080298 · Cited by 2
KC-Agent: A Dual-Process Cognitive Architecture for Efficient ML Model Improvement
Bravo-Rocca, G., Guitart, J., Dholakia, A., Ellison, D. D., & Jain, P. K. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00186 · Cited by 1
RS-Claw: Progressive Active Tool Exploration via Hierarchical Skill Trees for Remote Sensing Agents
Liu, L., Wang, Z., Li, Z., Ouyang, K., Tang, Z., Liu, C., Li, H., Yu, H., Yang, W., Yang, C., et al. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3720894 · Cited by 2
Wu, Z., Xiao, F., Li, S., Peng, J., Li, H., & Li, H. (2026) · Building Simulation · doi:10.1007/s12273-026-1467-3 · Cited by 1
A lifecycle-oriented survey of LLM-based agents: Workflow, graph augmentation, and future directions
Kang, C., & Li, K. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134838 · Cited by 3
-, S. B., Ranjan, S., Adhitya, M., Suri, V., Hassija, V., Chamola, V., & Hussain, A. (2026) · Cognitive Computation · doi:10.1007/s12559-026-10619-1 · Cited by 5
Intelligent Agents for Smart Agriculture: Architectures, Applications, and Future Challenges
Tao, W., Sang, Q., Chen, C., & Mao, Q. (2026) · Agriculture · doi:10.3390/agriculture16171808 · Cited by 2
Xiang, J., Ye, T., Liu, P., Zhang, Y. Z., & Wang, W. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830313 · Cited by 1
Thought flow nets: From single predictions to trains of model thought
Schuff, H., Adel, H., & Vu, N. T. (2024) · Natural language processing. · doi:10.1017/nlp.2024.41 · Cited by 1
Prompting Steganography: A New Paradigm
Wu, H. (2024) · Electronic Imaging · doi:10.2352/ei.2024.36.4.mwsf-338 · Cited by 2
SwarMind: Harnessing Large Language Models for Flock Dynamics
Mounsif, M., Zehnder, K., Motie, Y., & Adam-Gaxotte, Z. (2023) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi59957.2023.10458573 · Cited by 1
TraitsPrompt: Does Personality Traits Influence the Performance of a Large Language Model?
Zhu, Q., Lan, T., Xue, X., Yu, Z., & He, H. (2024) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd61410.2024.10580436 · Cited by 1
Quantifying Bias in Agentic Large Language Models: A Benchmarking Approach
Fernando, R., Norton, I., Dogra, P., et al. (2024) · Information Communication Technologies Conference (ICTC) · doi:10.1109/ictc61510.2024.10601938 · Cited by 1
AraT5GQA: Arabic Question Answering model using automatic generated dataset
Nakhleh, S., Mustafa, A., & Najadat, H. (2024) · International Conference on Information and Communication Systems (ICICS) · doi:10.1109/icics63486.2024.10638274 · Cited by 1
ToFC: Tree-of-Fact with Continued Best-First Search for Commonsense Reasoning
Dai, L., & Qin, K. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649921 · Cited by 1
Embodied Reasoning with Self-Feedback
Kak, P., & Jain, S. (2024) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct62155.2024.10677317 · Cited by 1
Cho, B. C., & Bang, J. (2024) · Journal of Broadcast Engineering · doi:10.5909/jbe.2024.29.5.596 · Cited by 1
Retrieval Augmented Tree of Thoughts
Liu, X., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_36 · Cited by 1
The Use of Generative Artificial Intelligence for Business Decision-Making
Mummar, A., Pacheco, A., Saunders, S., Ratliff-Villarreal, L., & Schmitt, A. (2024) · The Pinnacle A Journal by Scholar-Practitioners · doi:10.61643/c16259 · Cited by 1
Reseach on Prompmt Engineering for Generative AI to Create Business Data Analysis Reports
Park, Y., Kim, D., Yoo, S., Bae, Y., & Kim, K. (2024) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2024.22.11.167
Dragoni, D., & Margottini, M. (2025) · Journal of Educational Cultural and Psychological Studies (ECPS Journal) · doi:10.7358/ecps-2024-030-drma · Cited by 3
Problem Solving-Oriented Programming Knowledge Tracing from Behavior to Thought
Huang, T., Tang, L., Yang, H., Ou, X., Hu, S., Geng, J., & Hu, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889146 · Cited by 1
Revisiting the Non-Determinism of Code Generation by the GPT-3.5 Large Language Model
Sawadogo, S., Sabané, A., Kafando, R., Kaboré, A. K., & Bissyandé, T. F. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00012 · Cited by 1
A Second-Generation Agentic Framework for Generative Ai-Driven Augmented Reality Educational Games
Ordoumpozanis, K., & Apostolidis, H. (2025) · IEEE Global Engineering Education Conference · doi:10.1109/educon62633.2025.11016327 · Cited by 3
Güttinger, J., Fenger, J.-N., Kirchmeier, T., & Legat, C. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-85316-6_2 · Cited by 1
Baby’s CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
Zhang, Z., Yang, H., Ma, B., Rügamer, D., & Nie, E. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.13 · Cited by 5
Building generalizable deep learning solutions for mobile sensing
Xu, H. (2024) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/174809 · Cited by 1
Liu, Y., Shen, R., Zhou, L., Xiao, Q., Yuan, J., & Li, Y. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.19.594895 · Cited by 1
Exploring the Boundaries of Financial Statement Fraud Detection with Large Language ModelsPreprint
Kirkos, E., Boskou, G., Chatzipetrou, E., Tiakas, E., & Σπάθης, Χ. Τ. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4842962 · Cited by 5
Exploring the Boundaries of Financial Statement Fraud Detection with Large Language ModelsPreprint
Boskou, G., Chatzipetrou, E., Tiakas, E., Kirkos, E., & Spathis, C. T. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4897041 · Cited by 1
LLMs as Method Actors: A Model for Prompt Engineering and ArchitecturePreprint
Doyle, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5030804 · Cited by 5
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 4
Metadata
- Authors
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 11809-11822
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2305.10601
- OpenAlex
- W7133209886
- Cited by
- 1,158
- References
- 0
- Citations by year
- 2026: 476 · 2025: 320 · 2024: 284 · 2023: 77
- FWCI
- 206.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2239363)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- tree-of-thought, large language models, chain-of-thought prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
