The DOI matches its DataCite record; 6 of the 42 references it cites were found in a registry or an index; 12 of the 18 sources its claims cite were checked for support.
Toolformer: Language Models Can Teach Themselves to Use Tools
Open accessAbstract
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q\&A system, two different search engines, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Toolformer is a 6.7B GPT-J model trained self-supervised to insert API calls for question answering, Wikipedia search, a calculator, machine translation, and a calendar, deciding when and how to call them based on whether calls reduce future-token loss. It improves zero-shot downstream performance, often beating much larger GPT-3/OPT models, without increasing perplexity when API calls are disabled.
Key findings
- On LAMA subsets, Toolformer improves over the best baseline by 11.7, 5.2, and 18.6 points on SQuAD, Google-RE, and T-REx, respectively, and uses the question answering tool in 98.1% of cases; it outperforms OPT (66B) and GPT-3 (175B).
- On math benchmarks, Toolformer reaches ASDiv 40.4, SVAMP 29.4, and MAWPS 44.0, versus GPT-J’s 7.5, 5.2, and 9.9, using the calculator for 97.9% of examples.
- On question answering, Toolformer scores WebQS 26.3, NQ 17.7, and TriviaQA 48.8, mostly using Wikipedia search (99.3%), but still lags GPT-3 (175B) at 29.0, 22.6, and 65.9.
- On temporal datasets, Toolformer reaches DATESET 27.3 versus GPT-J 3.9, with the calendar used for 54.8% of examples; on TEMPLAMA it scores 16.3, but the calendar is used for only 0.2%.
- Language modeling perplexity is not degraded: Toolformer (disabled) has WikiText 10.3 and CCNet 10.5, matching GPT-J + CC. Tool use emerges around 775M parameters.
Methods
- Self-supervised pipeline: sample candidate API calls with an LM, execute them, filter calls by whether they reduce the weighted loss over future tokens by at least τ_f, merge useful calls into a CCNet subset, and finetune GPT-J on the augmented dataset.
- Tools: Atlas for QA, BM25 retriever over Wikipedia for search, a Python calculator for basic arithmetic, 600M NLLB for machine translation, and a calendar API; up to 25k examples per API, batch size 128, learning rate 1e-4.
- Evaluation: zero-shot downstream tasks with greedy decoding modified to allow `<API>` if among top k=10 tokens, at most one API call per input; baselines include GPT-J, GPT-J + CC, Toolformer (disabled), OPT (66B), and GPT-3 (175B).
Limitations
- Stated: Toolformer cannot chain tools, cannot use tools interactively, is sensitive to input wording, is sample-inefficient for some tools (e.g., over a million documents yield only a few thousand useful calculator calls), and does not account for tool-dependent computational cost.
- Evident from the text: the one-API-call-per-example restriction and independent sampling prevent chained calls; TEMPLAMA gains are not attributable to the calendar tool; MLQA does not consistently outperform GPT-J because CCNet finetuning can cause distribution shift; QA lags GPT-3 partly because the search engine is simple and Toolformer cannot reformulate queries or browse multiple results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
18 Own interpretation · 18 Own result · 17 No basis given · 6 Own method · 5 Partly supported · 3 Supported · 3 Sources not found · 1 Mixed support · 1 Not checked yet · accuracy not yet measured
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“or limit tool use to task-specific settings only (e.g., Gao et al., 2022;Parisi et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Disputedcites Gao et al., 2022 sourceExperimental — accuracy not yet measured
Section 7 (Related Work; page unavailable) explicitly contrasts PAL with task-specific tools, saying its Python interpreter handles both arithmetic and dates without specialized modules. Section 4 (Experimental Setup; page unavailable) also describes evaluations across mathematical, symbolic, and algorithmic tasks. The paper therefore does not limit PAL’s tool use to task-specific settings only. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Partly supportedcites Parisi et al., 2022 sourceExperimental — accuracy not yet measured
The Results (p. 3) evaluate TALM on two specific tasks, Natural Questions and MathQA, with tools suited to each. But the Conclusion (p. 4) describes TALM as a framework for arbitrary tools that makes exploring additional tools and tasks possible. The paper’s evaluations are task-specific; its approach is not limited to task-specific settings only. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“These limitations include an inability to access up-to-date information on recent events (Komeili et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Komeili et al., 2022 source
The Introduction says static language models cannot account for new information because their knowledge is frozen at training, and contrasts them with an internet-augmented model that can use the latest information. The Conclusions likewise describe models limited to knowledge in their fixed training set. This supports the claim as a limitation of LMs without dynamic access to outside information. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“existing approaches either rely on large amounts of human annotations (Komeili et al., 2022;Thoppilan et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Komeili et al., 2022 source
Section 3.2 says the internet-search model's modules require supervised context–search-query and context–response pairs, supplied by the data collected for this work. Section 4.1 reports a substantial crowdsourced dataset: 9,633 dialogues and 82,952 training utterances. This supports the claim for Komeili et al.'s approach, without establishing that every external-tool approach has the same requirement. (LLM, conf=0.91)
- Supportedcites Thoppilan et al., 2022 source
Section 6.2 (p. 8) describes LaMDA learning to consult external tools using 40,000 annotated dialog turns and 9,000 turns with human-assigned labels. This supports the claim that this tool-using approach relies on substantial human annotation, although the Discussion (p. 15) calls the amount modest relative to pre-training data. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“a lack of mathematical skills to perform precise calculations (Patel et al., 2021)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Partly supportedcites Patel et al., 2021 sourceExperimental — accuracy not yet measured
The Results on SVAMP report that the best-performing model achieved 43.8% accuracy, supporting a limitation in solving elementary arithmetic word problems. But the Problem Formulation section says the models generate expressions that are then evaluated to calculate answers; the paper does not establish a general inability of language models to perform precise calculations. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This is due to the fact that API calls for each tool are generated independently; as a consequence, there are no examples of chained tool use in the finetuning dataset.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“increasing k leads to the model doing API calls for more examples -from 40.3% and 8.5% with k = 1 (i.e., regular greedy decoding) to 98.1% and 100% for k = 10.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 5 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Toolformer achieves stronger results even when API calls are disabled.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Depending on the tool, our method is also very sample-inefficient; for example, processing more than a million documents results in only a few thousand examples of useful calls to the calculator API.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the considerable improvement of Toolformer compared to other models can be fully accredited to the calendar tool, which it makes use of for 54.8% of all examples.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.5 Temporal Datasets
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 16
- It may state two separate claims as one. 22
- The quote only partly states this claim. 3
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“and the related tendency to hallucinate facts (Maynez et al., 2020;Ji et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Maynez et al., 2020 source
Section 2.1 says a decoder functioning like an open-ended language model is prone to extrinsic hallucinations. Section 5.3 reports that over 90% of the hallucinations produced by one summarization model were erroneous. The paper thus supports the broad claim, though its empirical results concern abstractive summarization models. (LLM, conf=0.88)
- Not matchedcites Ji et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models achieve impressive zeroand few-shot results on a variety of natural language processing tasks (Brown et al., 2020;Chowdhery et al., 2022, i.a.)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Not matchedcites Brown et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Chowdhery et al., 2022 sourceExperimental — accuracy not yet measured
The Conclusion (p. 50) reports outstanding few-shot performance across English NLP tasks, and Section 6.1 (p. 11) describes results on 29 benchmarks. Section 6.8 (p. 33) evaluates zero-shot performance on five tasks, but the passages do not establish the claimed impressive zero-shot results across a variety of tasks. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
What the sources say
- Partly supportedcites Jiang et al., 2020 sourceExperimental — accuracy not yet measured
The Evaluation Results (p. 5) show that changing prompt wording substantially changed BERT’s accuracy, supporting prompt sensitivity in the paper’s knowledge-retrieval task. The paper does not establish the claim specifically for both zero-shot and few-shot settings; its prompt-selection method uses training examples (Top-1 Prompt Selection, p. 4). (LLM, conf=0.86)
- Supportedcites Schick and Schütze, 2021a source
Section 5 (Analysis; page unavailable) reports a large performance gap between the best and worst prompt patterns even after few-shot finetuning. This supports the prompt-sensitivity point for which Schick and Schütze are cited; the sentence also cites Jiang et al. for the broader zero-shot context. (LLM, conf=0.90)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“and an unawareness of the progression of time (Dhingra et al., 2022).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Dhingra et al., 2022 source
The Introduction says standard LM training generally ignores temporal metadata and that standard LMs have no knowledge of what is “new” or “old.” This supports the claim as a broad description of standard LMs, while the paper distinguishes them from models trained with temporal context. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“TALM (Parisi et al., 2022), an approach that uses a similar self-supervised objective for teaching a model to use a calculator and a search engine, but explores this only in settings where a model is finetuned for downstream tasks.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 6 Related Work
What the sources say
- Partly supportedcites Parisi et al., 2022 sourceExperimental — accuracy not yet measured
Sections 3.1 and 3.2 (p. 3) describe retrieval and calculation tools for downstream question-answering and math tasks; §2.2 (p. 2) says TALM is fine-tuned during iterative self-play. But §2.2 describes supervised input–target pairs and a small set of tool-use demonstrations, not a self-supervised objective. The main retrieval experiment uses a BM25 index; a public search engine is used for an out-of-distribution test (§3.4, p. 4). (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ability to leverage the provided tools only emerges at around 775M parameters: smaller models achieve similar performance both with and without tools.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.4 Scaling Laws
plaindirectn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“improvements on TEMPLAMA can not be attributed to the calendar tool, which is only used for 0.2% of all examples, but mostly to the Wikipedia search and question answering tools”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.5 Temporal Datasets
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“high values of L - i -L + i typically correspond to useful API calls, whereas low values correspond to API calls that do not provide any information that is useful for predicting future tokens.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > Data Quality
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“OPT and GPT-3 perform surprisingly weak across all languages, mostly because they fail to provide an answer in English despite being instructed to do so.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, when deciding whether or not to make an API call, Toolformer currently does not take into account the tool-dependent, computational cost incurred from making an API call.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Depending on the language, this tool is used for 63.8% to 94.9% of all examples; the only exception to this is Hindi, for which the machine translation tool is used in only 7.3% of cases.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As one would expect, finetuning on CCNet leads to slightly improved performance on a different CC-Net subset, but it slightly deteriorates performance on WikiText, presumably because the original pre-”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Toolformer, which is based on a pretrained GPT-J model (Wang and Komatsuzaki, 2021) with 6.7B parameters, achieves much stronger zero-shot results, clearly outperforming a much larger GPT-3 model (Brown et al., 2020) and several other baselines on various tasks.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Not matchedcites Brown et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for 97.9% of all examples the model decides to ask the calculator tool for help.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Toolformer clearly outperforms all other models based on GPT-J, this time mostly relying on the Wikipedia search API (99.3%) to find relevant information.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.3 Question Answering
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using the Wikipedia search tool for most examples, Toolformer clearly outperforms baselines of the same size, but falls short of GPT-3 (175B).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Toolformer outperforms all baselines for both TEM-PLAMA and DATESET.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.5 Temporal Datasets
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nonetheless, allowing the model to make API calls more than doubles performance for all tasks, and also clearly outperforms the much larger OPT and GPT-3 models.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Toolformer outperforms all baselines, but does not make use of the calendar tool for TEMPLAMA.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the model independently decides to ask the question answering tool for the required information in almost all cases (98.1%); for only very few examples, it uses a different tool (0.7%) or no tool at all (1.2%).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.1 LAMA
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“improving upon the best baseline by 11.7, 5.2 and 18.6 points, respectively.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.1 LAMA
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“A potential reason for GPT-J not suffering from this problem is that it was trained on more multilingual data than both OPT and GPT-3, including the EuroParl corpus (Koehn, 2005;Gao et al., 2020).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
What the sources say
- Not matchedcites Koehn, 2005
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Gao et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we prevent Toolformer from using the Wikipedia Search API to avoid giving it an unfair advantage.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.1 LAMA
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In a similar spirit to these approaches, Toolformer is trained on its own predictions after applying a perplexity-based filtering step.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 6 Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The remaining examples after filtering are close enough to the original distribution so that the model's language modeling abilities remain unaffected.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“some amount of noise in the API calls that are not filtered can actually be useful as it forces the model finetuned on C * to not always blindly follow the results of each call it makes.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > Data Quality
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This ensures that the model does not lose any of its generality and language modeling abilities.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We surmise that this is because the model is finetuned on many examples of API calls and their results, improving its own mathematical capabilities.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, Toolformer learns for itself to explicitly asks for the right information.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 6 Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For Toolformer, we disable the question answering tool as this would make solving the tasks trivial”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.3 Question Answering
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we only at most one API call per input to make sure the model does not get stuck in a loop where it constantly calls APIs without producing any actual output.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2 Downstream Tasks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While during data generation the model can look ahead to generate API calls, this is not possible at inference time, so we want to dissuade the model from calling the API in such cases.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > A.1 Implementation Question Answering
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“difficulties in understanding low-resource languages (Lin et al., 2021)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Not matchedcites Lin et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This filtering mechanism allows us to generate data more efficiently by focusing our API call generations in places where the MT tool is likely to be helpful.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > A.1 Implementation Question Answering
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is mainly because for some languages, finetuning on CCNet deteriorates performance; this might be due to a distribution shift compared to GPT-J's original pretraining data.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We do not evaluate the perplexity of Toolformer with API calls enabled as computing the probability pM (xt | x1, . . ., xt-1) of token xt given x1, . . ., xt-1 would require marginalizing over all potential API calls that the model could make at position t, which is intractable.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is likely due to both the simplicity of our search engine (in many cases, it returns results that are clearly not a good match for a given query) and the inability of Toolformer to interact with it, e.g., by reformulating its query if results are not helpful or by browsing through multiple of the top results.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.3 Question Answering
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
What humans find useful may be different from what a model finds useful.
“what humans find useful may be different from what a model finds useful.”
The authors state this as an assumption.
An API call is helpful to the model if providing both its input and output makes it easier for the model to predict future tokens, compared to not receiving the API call or receiving only its input.
“Intuitively, an API call is helpful to M if providing it with both the input and the output of this call makes it easier for the model to predict future tokens, compared to not receiving the API call at all, or receiving only its input.”
The authors state this as an assumption.
For creating the calendar dataset, the calendar date in such cases should be the date that the document was created.
“we operate under the assumption that the calendar date in such cases should be the date that the document was created.”
The authors state this as an assumption.
The remaining examples after filtering are close enough to the original distribution so that the model's language modeling abilities remain unaffected.
“While this filtering alters the distribution of training examples, we assume that the remaining examples are close enough to the original distribution so that M 's language modeling abilities remain unaffected.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“These approaches demonstrate that RL can improve complex behavior sequences in LLMs.”
From Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning (2025) · cites this paper as Schick et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Earlier efforts in this line of work rely on supervised fine-tuning (SFT), which trains the LLM on high-quality tool-use trajectories to explicitly demonstrate how tools should be invoked and integrated into task execution (Schick et al., 2023; Du et al., 2024; Liu et al., 2025g; Wang et al., 2025e).”
From A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (2025) · cites this paper as Schick et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While AI Agents have garnered considerable attention for their ability to automate structured tasks using LLMs and interfaces to specific tools, the literature highlights significant theoretical and practical limitations that inhibit their reliability, generalization, and long-term autonomy [136,162].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [162]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite this, LLMs have inherent limitations in their enclosed reasoning, particularly when addressing complex situations that extend beyond textual boundaries (Schick et al., 2023).”
From Scaling Large Language Model-based Multi-Agent Collaboration (2024) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, many early language agents simply use LLMs to propose an action (Schick et al., 2023), a sequence of actions (Huang et al., 2022b), or evaluate a fixed set of actions (Ahn et al., 2022) without intermediate reasoning or retrieval. Followup work (Yao et al., 2022b; Shinn et al., 2023; Xu et al., 2023b; Lin et al., 2023; Wang et al., 2023a; Park et al., 2023) has exploited intermediate”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent work has equipped dialogue agents with the ability to use tools such as calculators and calendars, and to consult external websites 24,25 .”
From Role play with large language models (2023) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Modern LLM agents can select tools, maintain state across steps, and modify the plan as new evidence emerges.[13], [14].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reliability comes mainly from typed tool interfaces, provenance-aware retrieval, explicit budgets, independent gates, and rollback-aware execution, not from model fluency alone [12], [37], [13], [75], [76].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“General agent frameworks motivate the separation between reasoning and acting, but operations demands that the separation is enforceable and auditable rather than merely prompt-stated [14], [13], [159], [161].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Context engineering primarily addresses the structural organization of the interaction interface between LLMs and their operational environment. This includes optimizing tool-integrated reasoning and selection pipelines (Qin et al., 2024a; Schick et al., 2023; Jia and Li, 2025) and standardizing communication protocols, such as MCP (Qiu et al., 2025c).”
From Memory in the Age of AI Agents (2025) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“language models have further acquired capabilities such as retrieval, calculation, and program invocation, extending multi-step task solving from pure text generation to the joint process of reasoning and execution [3]”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond conversational history, tool-based agents must retain tool inputs, execution outputs, and intermediate states, many of which are repeatedly referenced in subsequent reasoning steps (Schick et al., 2023).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, another critical feature of LLMs is their capability to use tools for automated online search, task design, and experiment execution [203–205], which exhibits their further capabilities in automatically collecting and preprocessing biomedical datasets, designing bioinformatic analyses to identify disease-related molecular regulations, conducting literature reviews for result inte”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [203]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-based approaches may assist by summarizing large-scale interaction logs, extracting structured event representations, or generating hypotheses about plausible attack paths; recent work on tool-augmented language models and automated log analysis suggests their usefulness in triage and forensic workflows (Schick et al., 2023;Solomon et al., 2026;Yao et al., 2022).”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the literature highlights significant theoretical and practical limitations that inhibit their reliability, generalization, and long-term autonomy [138], [164].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These agents not only communicate and coordinate with one another but also interface with external resources such as tools, APIs, and structured data sources [24, 16].”
From Collaborative Memory: Multi-User Memory Sharing in LLM Agents with Dynamic Access Control (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This separation has been shown to improve model performance on a range of logical and mathematical reasoning tasks (Chen et al., 2022;Gao et al., 2023;Schick et al., 2023;Wang et al., 2022;Zhou et al., 2022;Cobbe et al., 2021).”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notable examples of such systems include the likes of LaMDA (Thoppilan et al., 2022), Toolformer (Schick et al., 2023), and Mind's Eye (Liu et al., 2022).”
From ChatGPT, extended: large language models and the extended mind (2025) · cites this paper as Schick et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“A promising design paradigm for AI agents [65] is to combine an LLM with tools that interact with a broader environment [14, 35, 40, 47, 51, 53, 55, 69].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“ReAct interleaves language reasoning with environment actions [16], while Toolformer demonstrates self-supervised learning of when and how to call external APIs [17].”
From HELIX: Model-Harness Co-evolution for Recursive Self-Improvement (2026) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“methods such as Toolformer showed that LLMs can learn to call external APIs to aid in problem-solving [18].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“Toolformer employs a training approach to determine the optimal usage of specific APIs and integrates the obtained results into subsequent token predictions.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [172]
The cited source says what the claim attributes to it.
In the citing paper
“Skill-related work has expanded from narrow function calling and tool learning toward reusable capability packages, registries, and progressive disclosure mechanisms [OpenAI, 2023a, Schick et al., 2023, Wang et al., 2023a, Anthropic, 2025, 2026, Jiang et al., 2026b].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Schick et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The motivation behind Toolformer [246] aligns closely with ReAct; however, it goes a step further by combining diverse tools within a single model. This integration provides the model with flexible decision-making abilities and improved generalization capabilities, achieved through a simple yet effective self-supervised method.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [246]
The cited source says what the claim attributes to it.
In the citing paper
“This motivates an established alternative-the tool-use paradigm [9, 10]-in which, rather than perform every reasoning step internally, a model delegates well-specified tasks to external tools.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Various studies (Liu et al., 2023;Parisi et al., 2022;Schick et al., 2023;Shen et al., 2023;Lu et al., 2023;Paranjape et al., 2023;Liang et al., 2023) have demonstrated that supplementing LLMs with tools, such as calculators, search engines, translation systems, calendars, or even API calls on other models, can help solve tasks that are not easily addressed by LLMs alone.”
From Large language models as tool makers (2023) · cites this paper as Schick et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“As discussed before, Agentic AI systems inherit the modularity of AI Agents but extend their architecture to support distributed intelligence, inter-agent communication, and iterative planning.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [162]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Indirect prompting could be leveraged to leak users' chat sessions [52] or exfiltrate users' data (e.g., credentials, personal information).”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Agentic AI systems consist of multiple agents, each assigned a specialized function or task (e.g., a summarizer, a retriever, or a planner).”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [164]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Agentic AI systems inherit the modularity of AI Agents but extend their architecture to support distributed intelligence, inter-agent communication, and iterative planning.”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [164]
Experimental — accuracy not yet measuredIn the citing paper
“Equipping an LLM with a suite of tools, such as calculators or web search, has proven superiority in tasks that require multistep reasoning and planning17,18.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Toolformer: Language Models Can Teach Themselves to Use Tools > A.2 Prompts
“This is three times less than the 2169 goals last year.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > A.2 Prompts
“This is twenty goals more than the [Calculator(723 -20)] 703 goals last year.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Ricardo Baeza-Yates, Berthier Ribeiro-Neto, et al. 1999. Modern information retrieval, volume 463. ACM press New York.
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Lin- guistics.
- Sergey Brin. 1999. Extracting patterns and relations from the world wide web. In The World Wide Web and Databases, pages 172-183, Berlin, Heidelberg. Springer Berlin Heidelberg.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In International Conference on Learning Representations.
- Gautier Izacard and Edouard Grave. 2021. Distilling knowledge from reader to retriever for question an- swering. In International Conference on Learning Representations.
- Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79-86.
- David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Pro- ceedings of the Human Language Technology Con- ference of the NAACL, Main Conference, pages 152- 159, New York City, USA. Association for Compu- tational Linguistics.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adaptation of statisti- cal parsers trained on small datasets. In Proceed- ings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 616-623, Prague, Czech Republic. Association for Computational Lin- guistics.
- Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Cited by
Sorted by citations
Items 3,201–3,207 of 3,207
TRACER: A Multimodal Multi-agent Workflow for Transparent Chest X-ray Assistance
Wang, Z., Wang, H., Martinez, R., & Saddik, A. E. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02469-y
Intent-Preserving Neuro-Symbolic Repair for Safe LLM-Driven Financial Transactions
Sophie, H., Ross, C., & Diaz, N. (2026) · Journal of Computing and Electronic Information Management · doi:10.54097/aeq3sg57
Dillibatcha, S. C. (2026) · World Journal of Advanced Engineering Technology and Sciences · doi:10.30574/wjaets.2026.21.2.0492
EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms
Yuan, S., Song, K., Chen, J., Tan, X., Li, D., & Yang, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14228
InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
Abhyankar, R., He, Z., Srivatsa, V., Zhang, H., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.01869
Self-Challenging Language Model Agents
Zhou, Y., Levine, S., Weston, J., Li, X., & Sukhbaatar, S. (2025) · arXiv · doi:10.48550/arxiv.2506.01716 · Cited by 1
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citation
Metadata
- Authors
- Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T.
- ORCID
- Zettlemoyer, Luke
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 68539-68551
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.04761
- OpenAlex
- W7133214447
- Cited by
- 1,150
- References
- 0
- Citations by year
- 2026: 730 · 2025: 136 · 2024: 165 · 2023: 119
- FWCI
- 492.67
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- tool use, language models, self-supervised learning, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
