The DOI matches its DataCite record; 6 of the 42 references it cites were found in a registry or an index; 12 of the 18 sources its claims cite were checked for support.
Toolformer: Language Models Can Teach Themselves to Use Tools
Open accessAbstract
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q\&A system, two different search engines, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Toolformer is a 6.7B GPT-J model trained self-supervised to insert API calls for question answering, Wikipedia search, a calculator, machine translation, and a calendar, deciding when and how to call them based on whether calls reduce future-token loss. It improves zero-shot downstream performance, often beating much larger GPT-3/OPT models, without increasing perplexity when API calls are disabled.
Key findings
- On LAMA subsets, Toolformer improves over the best baseline by 11.7, 5.2, and 18.6 points on SQuAD, Google-RE, and T-REx, respectively, and uses the question answering tool in 98.1% of cases; it outperforms OPT (66B) and GPT-3 (175B).
- On math benchmarks, Toolformer reaches ASDiv 40.4, SVAMP 29.4, and MAWPS 44.0, versus GPT-J’s 7.5, 5.2, and 9.9, using the calculator for 97.9% of examples.
- On question answering, Toolformer scores WebQS 26.3, NQ 17.7, and TriviaQA 48.8, mostly using Wikipedia search (99.3%), but still lags GPT-3 (175B) at 29.0, 22.6, and 65.9.
- On temporal datasets, Toolformer reaches DATESET 27.3 versus GPT-J 3.9, with the calendar used for 54.8% of examples; on TEMPLAMA it scores 16.3, but the calendar is used for only 0.2%.
- Language modeling perplexity is not degraded: Toolformer (disabled) has WikiText 10.3 and CCNet 10.5, matching GPT-J + CC. Tool use emerges around 775M parameters.
Methods
- Self-supervised pipeline: sample candidate API calls with an LM, execute them, filter calls by whether they reduce the weighted loss over future tokens by at least τ_f, merge useful calls into a CCNet subset, and finetune GPT-J on the augmented dataset.
- Tools: Atlas for QA, BM25 retriever over Wikipedia for search, a Python calculator for basic arithmetic, 600M NLLB for machine translation, and a calendar API; up to 25k examples per API, batch size 128, learning rate 1e-4.
- Evaluation: zero-shot downstream tasks with greedy decoding modified to allow `<API>` if among top k=10 tokens, at most one API call per input; baselines include GPT-J, GPT-J + CC, Toolformer (disabled), OPT (66B), and GPT-3 (175B).
Limitations
- Stated: Toolformer cannot chain tools, cannot use tools interactively, is sensitive to input wording, is sample-inefficient for some tools (e.g., over a million documents yield only a few thousand useful calculator calls), and does not account for tool-dependent computational cost.
- Evident from the text: the one-API-call-per-example restriction and independent sampling prevent chained calls; TEMPLAMA gains are not attributable to the calendar tool; MLQA does not consistently outperform GPT-J because CCNet finetuning can cause distribution shift; QA lags GPT-3 partly because the search engine is simple and Toolformer cannot reformulate queries or browse multiple results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
12 of 18 cited sources checked · 6 supported · 1 flagged · accuracy not yet measured
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“or limit tool use to task-specific settings only (e.g., Gao et al., 2022;Parisi et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Disputedcites Gao et al., 2022 sourceExperimental — accuracy not yet measured
Section 7 (Related Work; page unavailable) explicitly contrasts PAL with task-specific tools, saying its Python interpreter handles both arithmetic and dates without specialized modules. Section 4 (Experimental Setup; page unavailable) also describes evaluations across mathematical, symbolic, and algorithmic tasks. The paper therefore does not limit PAL’s tool use to task-specific settings only. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Partly supportedcites Parisi et al., 2022 sourceExperimental — accuracy not yet measured
The Results (p. 3) evaluate TALM on two specific tasks, Natural Questions and MathQA, with tools suited to each. But the Conclusion (p. 4) describes TALM as a framework for arbitrary tools that makes exploring additional tools and tasks possible. The paper’s evaluations are task-specific; its approach is not limited to task-specific settings only. (LLM, conf=0.84)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“These limitations include an inability to access up-to-date information on recent events (Komeili et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Komeili et al., 2022 source
The Introduction says static language models cannot account for new information because their knowledge is frozen at training, and contrasts them with an internet-augmented model that can use the latest information. The Conclusions likewise describe models limited to knowledge in their fixed training set. This supports the claim as a limitation of LMs without dynamic access to outside information. (LLM, conf=0.93)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“existing approaches either rely on large amounts of human annotations (Komeili et al., 2022;Thoppilan et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Komeili et al., 2022 source
Section 3.2 says the internet-search model's modules require supervised context–search-query and context–response pairs, supplied by the data collected for this work. Section 4.1 reports a substantial crowdsourced dataset: 9,633 dialogues and 82,952 training utterances. This supports the claim for Komeili et al.'s approach, without establishing that every external-tool approach has the same requirement. (LLM, conf=0.91)
- Supportedcites Thoppilan et al., 2022 source
Section 6.2 (p. 8) describes LaMDA learning to consult external tools using 40,000 annotated dialog turns and 9,000 turns with human-assigned labels. This supports the claim that this tool-using approach relies on substantial human annotation, although the Discussion (p. 15) calls the amount modest relative to pre-training data. (LLM, conf=0.86)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“a lack of mathematical skills to perform precise calculations (Patel et al., 2021)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Partly supportedcites Patel et al., 2021 sourceExperimental — accuracy not yet measured
The Results on SVAMP report that the best-performing model achieved 43.8% accuracy, supporting a limitation in solving elementary arithmetic word problems. But the Problem Formulation section says the models generate expressions that are then evaluated to calculate answers; the paper does not establish a general inability of language models to perform precise calculations. (LLM, conf=0.86)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This is due to the fact that API calls for each tool are generated independently; as a consequence, there are no examples of chained tool use in the finetuning dataset.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“increasing k leads to the model doing API calls for more examples -from 40.3% and 8.5% with k = 1 (i.e., regular greedy decoding) to 98.1% and 100% for k = 10.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 5 Analysis
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Toolformer achieves stronger results even when API calls are disabled.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Depending on the tool, our method is also very sample-inefficient; for example, processing more than a million documents results in only a few thousand examples of useful calls to the calculator API.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the considerable improvement of Toolformer compared to other models can be fully accredited to the calendar tool, which it makes use of for 54.8% of all examples.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.5 Temporal Datasets
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 16
- It may state two separate claims as one. 22
- The quote only partly states this claim. 3
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“and the related tendency to hallucinate facts (Maynez et al., 2020;Ji et al., 2022)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Maynez et al., 2020 source
Section 2.1 says a decoder functioning like an open-ended language model is prone to extrinsic hallucinations. Section 5.3 reports that over 90% of the hallucinations produced by one summarization model were erroneous. The paper thus supports the broad claim, though its empirical results concern abstractive summarization models. (LLM, conf=0.88)
- Not matchedcites Ji et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models achieve impressive zeroand few-shot results on a variety of natural language processing tasks (Brown et al., 2020;Chowdhery et al., 2022, i.a.)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Not matchedcites Brown et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Chowdhery et al., 2022 sourceExperimental — accuracy not yet measured
The Conclusion (p. 50) reports outstanding few-shot performance across English NLP tasks, and Section 6.1 (p. 11) describes results on 29 benchmarks. Section 6.8 (p. 33) evaluates zero-shot performance on five tasks, but the passages do not establish the claimed impressive zero-shot results across a variety of tasks. (LLM, conf=0.82)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
What the sources say
- Partly supportedcites Jiang et al., 2020 sourceExperimental — accuracy not yet measured
The Evaluation Results (p. 5) show that changing prompt wording substantially changed BERT’s accuracy, supporting prompt sensitivity in the paper’s knowledge-retrieval task. The paper does not establish the claim specifically for both zero-shot and few-shot settings; its prompt-selection method uses training examples (Top-1 Prompt Selection, p. 4). (LLM, conf=0.86)
- Supportedcites Schick and Schütze, 2021a source
Section 5 (Analysis; page unavailable) reports a large performance gap between the best and worst prompt patterns even after few-shot finetuning. This supports the prompt-sensitivity point for which Schick and Schütze are cited; the sentence also cites Jiang et al. for the broader zero-shot context. (LLM, conf=0.90)
Strength:boosted· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“and an unawareness of the progression of time (Dhingra et al., 2022).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Supportedcites Dhingra et al., 2022 source
The Introduction says standard LM training generally ignores temporal metadata and that standard LMs have no knowledge of what is “new” or “old.” This supports the claim as a broad description of standard LMs, while the paper distinguishes them from models trained with temporal context. (LLM, conf=0.91)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“TALM (Parisi et al., 2022), an approach that uses a similar self-supervised objective for teaching a model to use a calculator and a search engine, but explores this only in settings where a model is finetuned for downstream tasks.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 6 Related Work
What the sources say
- Partly supportedcites Parisi et al., 2022 sourceExperimental — accuracy not yet measured
Sections 3.1 and 3.2 (p. 3) describe retrieval and calculation tools for downstream question-answering and math tasks; §2.2 (p. 2) says TALM is fine-tuned during iterative self-play. But §2.2 describes supervised input–target pairs and a small set of tool-use demonstrations, not a self-supervised objective. The main retrieval experiment uses a BM25 index; a public search engine is used for an out-of-distribution test (§3.4, p. 4). (LLM, conf=0.86)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ability to leverage the provided tools only emerges at around 775M parameters: smaller models achieve similar performance both with and without tools.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.4 Scaling Laws
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“improvements on TEMPLAMA can not be attributed to the calendar tool, which is only used for 0.2% of all examples, but mostly to the Wikipedia search and question answering tools”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.5 Temporal Datasets
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“high values of L - i -L + i typically correspond to useful API calls, whereas low values correspond to API calls that do not provide any information that is useful for predicting future tokens.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > Data Quality
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“OPT and GPT-3 perform surprisingly weak across all languages, mostly because they fail to provide an answer in English despite being instructed to do so.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, when deciding whether or not to make an API call, Toolformer currently does not take into account the tool-dependent, computational cost incurred from making an API call.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 7 Limitations
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Depending on the language, this tool is used for 63.8% to 94.9% of all examples; the only exception to this is Hindi, for which the machine translation tool is used in only 7.3% of cases.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As one would expect, finetuning on CCNet leads to slightly improved performance on a different CC-Net subset, but it slightly deteriorates performance on WikiText, presumably because the original pre-”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“Toolformer, which is based on a pretrained GPT-J model (Wang and Komatsuzaki, 2021) with 6.7B parameters, achieves much stronger zero-shot results, clearly outperforming a much larger GPT-3 model (Brown et al., 2020) and several other baselines on various tasks.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Not matchedcites Brown et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for 97.9% of all examples the model decides to ask the calculator tool for help.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Toolformer clearly outperforms all other models based on GPT-J, this time mostly relying on the Wikipedia search API (99.3%) to find relevant information.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.3 Question Answering
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using the Wikipedia search tool for most examples, Toolformer clearly outperforms baselines of the same size, but falls short of GPT-3 (175B).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Toolformer outperforms all baselines for both TEM-PLAMA and DATESET.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.5 Temporal Datasets
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nonetheless, allowing the model to make API calls more than doubles performance for all tasks, and also clearly outperforms the much larger OPT and GPT-3 models.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Toolformer outperforms all baselines, but does not make use of the calendar tool for TEMPLAMA.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the model independently decides to ask the question answering tool for the required information in almost all cases (98.1%); for only very few examples, it uses a different tool (0.7%) or no tool at all (1.2%).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.1 LAMA
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“improving upon the best baseline by 11.7, 5.2 and 18.6 points, respectively.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.1 LAMA
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one. The quote only partly states this claim.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“A potential reason for GPT-J not suffering from this problem is that it was trained on more multilingual data than both OPT and GPT-3, including the EuroParl corpus (Koehn, 2005;Gao et al., 2020).”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
What the sources say
- Not matchedcites Koehn, 2005
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Gao et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we prevent Toolformer from using the Wikipedia Search API to avoid giving it an unfair advantage.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.1 LAMA
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In a similar spirit to these approaches, Toolformer is trained on its own predictions after applying a perplexity-based filtering step.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 6 Related Work
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The remaining examples after filtering are close enough to the original distribution so that the model's language modeling abilities remain unaffected.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“some amount of noise in the API calls that are not filtered can actually be useful as it forces the model finetuned on C * to not always blindly follow the results of each call it makes.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > Data Quality
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This ensures that the model does not lose any of its generality and language modeling abilities.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We surmise that this is because the model is finetuned on many examples of API calls and their results, improving its own mathematical capabilities.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.2 Math Datasets
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, Toolformer learns for itself to explicitly asks for the right information.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 6 Related Work
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For Toolformer, we disable the question answering tool as this would make solving the tasks trivial”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.3 Question Answering
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we only at most one API call per input to make sure the model does not get stuck in a loop where it constantly calls APIs without producing any actual output.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2 Downstream Tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While during data generation the model can look ahead to generate API calls, this is not possible at inference time, so we want to dissuade the model from calling the API in such cases.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > A.1 Implementation Question Answering
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“difficulties in understanding low-resource languages (Lin et al., 2021)”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 1 Introduction
What the sources say
- Not matchedcites Lin et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This filtering mechanism allows us to generate data more efficiently by focusing our API call generations in places where the MT tool is likely to be helpful.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > A.1 Implementation Question Answering
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is mainly because for some languages, finetuning on CCNet deteriorates performance; this might be due to a distribution shift compared to GPT-J's original pretraining data.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.4 Multilingual Question Answering
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We do not evaluate the perplexity of Toolformer with API calls enabled as computing the probability pM (xt | x1, . . ., xt-1) of token xt given x1, . . ., xt-1 would require marginalizing over all potential API calls that the model could make at position t, which is intractable.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > D DATESET
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is likely due to both the simplicity of our search engine (in many cases, it returns results that are clearly not a good match for a given query) and the inability of Toolformer to interact with it, e.g., by reformulating its query if results are not helpful or by browsing through multiple of the top results.”
§ Toolformer: Language Models Can Teach Themselves to Use Tools > 4.2.3 Question Answering
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper relies on this without support.
What humans find useful may be different from what a model finds useful.
“what humans find useful may be different from what a model finds useful.”
The authors state this as an assumption.
An API call is helpful to the model if providing both its input and output makes it easier for the model to predict future tokens, compared to not receiving the API call or receiving only its input.
“Intuitively, an API call is helpful to M if providing it with both the input and the output of this call makes it easier for the model to predict future tokens, compared to not receiving the API call at all, or receiving only its input.”
The authors state this as an assumption.
For creating the calendar dataset, the calendar date in such cases should be the date that the document was created.
“we operate under the assumption that the calendar date in such cases should be the date that the document was created.”
The authors state this as an assumption.
The remaining examples after filtering are close enough to the original distribution so that the model's language modeling abilities remain unaffected.
“While this filtering alters the distribution of training examples, we assume that the remaining examples are close enough to the original distribution so that M 's language modeling abilities remain unaffected.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“These approaches demonstrate that RL can improve complex behavior sequences in LLMs.”
From Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning (2025) · cites this paper as Schick et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Earlier efforts in this line of work rely on supervised fine-tuning (SFT), which trains the LLM on high-quality tool-use trajectories to explicitly demonstrate how tools should be invoked and integrated into task execution (Schick et al., 2023; Du et al., 2024; Liu et al., 2025g; Wang et al., 2025e).”
From A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (2025) · cites this paper as Schick et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While AI Agents have garnered considerable attention for their ability to automate structured tasks using LLMs and interfaces to specific tools, the literature highlights significant theoretical and practical limitations that inhibit their reliability, generalization, and long-term autonomy [136,162].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [162]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite this, LLMs have inherent limitations in their enclosed reasoning, particularly when addressing complex situations that extend beyond textual boundaries (Schick et al., 2023).”
From Scaling Large Language Model-based Multi-Agent Collaboration (2024) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, many early language agents simply use LLMs to propose an action (Schick et al., 2023), a sequence of actions (Huang et al., 2022b), or evaluate a fixed set of actions (Ahn et al., 2022) without intermediate reasoning or retrieval. Followup work (Yao et al., 2022b; Shinn et al., 2023; Xu et al., 2023b; Lin et al., 2023; Wang et al., 2023a; Park et al., 2023) has exploited intermediate”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent work has equipped dialogue agents with the ability to use tools such as calculators and calendars, and to consult external websites 24,25 .”
From Role play with large language models (2023) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Modern LLM agents can select tools, maintain state across steps, and modify the plan as new evidence emerges.[13], [14].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reliability comes mainly from typed tool interfaces, provenance-aware retrieval, explicit budgets, independent gates, and rollback-aware execution, not from model fluency alone [12], [37], [13], [75], [76].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“General agent frameworks motivate the separation between reasoning and acting, but operations demands that the separation is enforceable and auditable rather than merely prompt-stated [14], [13], [159], [161].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Context engineering primarily addresses the structural organization of the interaction interface between LLMs and their operational environment. This includes optimizing tool-integrated reasoning and selection pipelines (Qin et al., 2024a; Schick et al., 2023; Jia and Li, 2025) and standardizing communication protocols, such as MCP (Qiu et al., 2025c).”
From Memory in the Age of AI Agents (2025) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“language models have further acquired capabilities such as retrieval, calculation, and program invocation, extending multi-step task solving from pure text generation to the joint process of reasoning and execution [3]”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond conversational history, tool-based agents must retain tool inputs, execution outputs, and intermediate states, many of which are repeatedly referenced in subsequent reasoning steps (Schick et al., 2023).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, another critical feature of LLMs is their capability to use tools for automated online search, task design, and experiment execution [203–205], which exhibits their further capabilities in automatically collecting and preprocessing biomedical datasets, designing bioinformatic analyses to identify disease-related molecular regulations, conducting literature reviews for result inte”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [203]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-based approaches may assist by summarizing large-scale interaction logs, extracting structured event representations, or generating hypotheses about plausible attack paths; recent work on tool-augmented language models and automated log analysis suggests their usefulness in triage and forensic workflows (Schick et al., 2023;Solomon et al., 2026;Yao et al., 2022).”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the literature highlights significant theoretical and practical limitations that inhibit their reliability, generalization, and long-term autonomy [138], [164].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These agents not only communicate and coordinate with one another but also interface with external resources such as tools, APIs, and structured data sources [24, 16].”
From Collaborative Memory: Multi-User Memory Sharing in LLM Agents with Dynamic Access Control (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This separation has been shown to improve model performance on a range of logical and mathematical reasoning tasks (Chen et al., 2022;Gao et al., 2023;Schick et al., 2023;Wang et al., 2022;Zhou et al., 2022;Cobbe et al., 2021).”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Schick et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notable examples of such systems include the likes of LaMDA (Thoppilan et al., 2022), Toolformer (Schick et al., 2023), and Mind's Eye (Liu et al., 2022).”
From ChatGPT, extended: large language models and the extended mind (2025) · cites this paper as Schick et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“A promising design paradigm for AI agents [65] is to combine an LLM with tools that interact with a broader environment [14, 35, 40, 47, 51, 53, 55, 69].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“ReAct interleaves language reasoning with environment actions [16], while Toolformer demonstrates self-supervised learning of when and how to call external APIs [17].”
From HELIX: Model-Harness Co-evolution for Recursive Self-Improvement (2026) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“methods such as Toolformer showed that LLMs can learn to call external APIs to aid in problem-solving [18].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“Toolformer employs a training approach to determine the optimal usage of specific APIs and integrates the obtained results into subsequent token predictions.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [172]
The cited source says what the claim attributes to it.
In the citing paper
“Skill-related work has expanded from narrow function calling and tool learning toward reusable capability packages, registries, and progressive disclosure mechanisms [OpenAI, 2023a, Schick et al., 2023, Wang et al., 2023a, Anthropic, 2025, 2026, Jiang et al., 2026b].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Schick et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The motivation behind Toolformer [246] aligns closely with ReAct; however, it goes a step further by combining diverse tools within a single model. This integration provides the model with flexible decision-making abilities and improved generalization capabilities, achieved through a simple yet effective self-supervised method.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [246]
The cited source says what the claim attributes to it.
In the citing paper
“This motivates an established alternative-the tool-use paradigm [9, 10]-in which, rather than perform every reasoning step internally, a model delegates well-specified tasks to external tools.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Various studies (Liu et al., 2023;Parisi et al., 2022;Schick et al., 2023;Shen et al., 2023;Lu et al., 2023;Paranjape et al., 2023;Liang et al., 2023) have demonstrated that supplementing LLMs with tools, such as calculators, search engines, translation systems, calendars, or even API calls on other models, can help solve tasks that are not easily addressed by LLMs alone.”
From Large language models as tool makers (2023) · cites this paper as Schick et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“As discussed before, Agentic AI systems inherit the modularity of AI Agents but extend their architecture to support distributed intelligence, inter-agent communication, and iterative planning.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [162]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Agents can also implement a Tree-of-Thoughts strategy, collectively exploring different reasoning branches in parallel and exchanging information to decide on the best path forward [18,20].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Indirect prompting could be leveraged to leak users' chat sessions [52] or exfiltrate users' data (e.g., credentials, personal information).”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Agentic AI systems consist of multiple agents, each assigned a specialized function or task (e.g., a summarizer, a retriever, or a planner).”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [164]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Agentic AI systems inherit the modularity of AI Agents but extend their architecture to support distributed intelligence, inter-agent communication, and iterative planning.”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [164]
Experimental — accuracy not yet measuredIn the citing paper
“Equipping an LLM with a suite of tools, such as calculators or web search, has proven superiority in tasks that require multistep reasoning and planning17,18.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Prompt example changes last year's goal total
MinorThe calculator prompt is supposed to add an API call to the input text, but its output changes last year's total from 2169 goals to 703 goals.
“This is three times less than the 2169 goals last year.”
“This is twenty goals more than the [Calculator(723 -20)] 703 goals last year.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Ricardo Baeza-Yates, Berthier Ribeiro-Neto, et al. 1999. Modern information retrieval, volume 463. ACM press New York.
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Lin- guistics.
- Sergey Brin. 1999. Extracting patterns and relations from the world wide web. In The World Wide Web and Databases, pages 172-183, Berlin, Heidelberg. Springer Berlin Heidelberg.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In International Conference on Learning Representations.
- Gautier Izacard and Edouard Grave. 2021. Distilling knowledge from reader to retriever for question an- swering. In International Conference on Learning Representations.
- Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79-86.
- David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Pro- ceedings of the Human Language Technology Con- ference of the NAACL, Main Conference, pages 152- 159, New York City, USA. Association for Compu- tational Linguistics.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adaptation of statisti- cal parsers trained on small datasets. In Proceed- ings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 616-623, Prague, Czech Republic. Association for Computational Lin- guistics.
- Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Cited by
Sorted by citations
Items 101–200 of 3,205
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137
“…al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a), integration with image processing systems, (Alayrac et al., 2022; OpenAI, 2023b), or integration with other software tools (Nakano et al., 2021; Menick et al., 2022; Collins et al., 2022; Schick et al., 2023; OpenAI, 2023a).”
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Language Models can Solve Computer Tasks
Kim, G., Baldi, P., & McAleer, S. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1723 · Cited by 137
Large Language Models for Education: A SurveyPreprint
Xu, H., Gan, W., Qi, Z., Wu, J., & Yu, P. S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.13001 · Cited by 137
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Yang, R., Song, L., Li, Y., Zhao, S., Ge, Y., Li, X., & Shan, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3149 · Cited by 25
Mind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134
Large Language Model Capabilities in Perioperative Risk Prediction and Prognostication
Chung, P., Fong, C. T., Walters, A. M., Aghaeepour, N., Yetişgen, M., & O’Reilly-Shah, V. N. (2024) · JAMA Surgery · doi:10.1001/jamasurg.2024.1621 · Cited by 140
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.691 · Cited by 141
(2023) · doi:10.52202/075280-1220 · Cited by 134
Verify-and-edit: A knowledge-enhanced chain-of-thought framework
Zhao, R., Li, X., Joty, S., Qin, C., & Bing, L. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.320 · Cited by 55
LLMLight: Large Language Models as Traffic Signal Control Agents
Lai, S., Zhao, X., Zhang, W., Liu, H., & Xiong, H. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709379 · Cited by 131
Characterizing Manipulation from AI Systems
Carroll, M., Chan, A., Ashton, H., & Krueger, D. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623226 · Cited by 131
Materials science in the era of large language models: a perspective
Lei, G., Docherty, R., & Cooper, S. J. (2024) · Digital Discovery · doi:10.1039/d4dd00074a · Cited by 130
Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives
Hu, S., Huang, T., İlhan, F., Tekin, S. F., & Liu, L. (2023) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa58951.2023.00044 · Cited by 134
Tool Documentation Enables Zero-Shot Tool-Usage with Large Language ModelsPreprint
Hsieh, C.-Y., Chen, S., Li, C., Fujii, Y., Ratner, A., Lee, C.-Y., Krishna, R., & Pfister, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.00675 · Cited by 126
Barman, D., Guo, Z., & Conlan, O. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100545 · Cited by 127
Chen, W., Ma, X., Wang, X., & Cohen, W. W. (2022) · arXiv · doi:10.48550/arxiv.2211.12588 · Cited by 125
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.187 · Cited by 124
Fact-Checking Complex Claims with Program-Guided Reasoning
Pan, L., Wu, X., Lu, X., Luu, A. T., Wang, W. Y., Kan, M., & Nakov, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.386 · Cited by 130
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent
Yang, J., Chen, X., Qian, S., Madaan, N., Iyengar, M., Fouhey, D. F., & Chai, J. Y. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610443 · Cited by 128
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 592
DriveLLM: Charting the Path Toward Full Autonomous Driving With Large Language Models
Cui, Y., Huang, S., Zhong, J., Liu, Z., Wang, Y., Sun, C., Li, B., Wang, X., & Khajepour, A. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3327715 · Cited by 122
Prompting Is Programming: A Query Language for Large Language Models
Beurer-Kellner, L., Fischer, M., & Vechev, M. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591300 · Cited by 117
Algorithm Evolution Using Large Language ModelPreprint
Liu, F., Tong, X., Yuan, M., & Zhang, Q. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.18894.43847 · Cited by 117
Towards end-to-end automation of AI research
Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., & Clune, J. (2026) · Nature · doi:10.1038/s41586-026-10265-5 · Cited by 116
Small Models are Valuable Plug-ins for Large Language Models
Xu, C., Yi‐chong, X., Wang, S., Liu, Y., Zhu, C., & McAuley, J. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.18 · Cited by 116
Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
Long, Y., Li, X., Cai, W., & Dong, H. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611565 · Cited by 115
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and LearnPreprint
Gao, D., Ji, L., Zhou, L., Lin, K. Q., Chen, J., Fan, Z., & Shou, M. Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.08640 · Cited by 115
VideoAgent: A Memory-Augmented Multimodal Agent for Video Understanding
Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z. F., & Li, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72670-5_5 · Cited by 114
Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially FastPreprint
Gu, X., Zheng, X., Pang, T., Du, C., Liu, Q., Wang, Y., Jiang, J., & Lin, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.08567 · Cited by 121
Large language models for chemistry robotics
Yoshikawa, N., Skreta, M., Darvish, K., Arellano-Rubach, S., Ji, Z., Bjørn Kristensen, L., Li, A. Z., Zhao, Y., Xu, H., Kuramshin, A., et al. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10136-2 · Cited by 112
Information Retrieval meets Large Language Models: A strategic report from Chinese IR community
Ai, Q., Bai, T., Cao, Z., Chang, Y., Chen, J., Chen, Z., Cheng, Z., Dong, S., Dou, Z., Feng, F., et al. (2023) · AI Open · doi:10.1016/j.aiopen.2023.08.001 · Cited by 115
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 112
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Hao, S., Liu, T., Wang, Z., & Hu, Z. (2023) · neural information processing systems · doi:10.52202/075280-1988 · Cited by 109
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2024) · Intelligent Computing · doi:10.34133/icomputing.0063 · Cited by 108
D-Bot: Database Diagnosis System using Large Language Models
Zhou, X., Li, G., Sun, Z., Liu, Z., Chen, W., Wu, J., Liu, J., Feng, R., & Zeng, G. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3675034.3675043 · Cited by 108
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic TasksPreprint
Liu, T., & Low, B. K. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14201 · Cited by 105
Evaluation of large language models for discovery of gene set function
Hu, M., Alkhairy, S., Lee, I., Pillich, R. T., Fong, D., Smith, K., Bachelder, R., Ideker, T., & Pratt, D. (2025) · Nature Methods · doi:10.1038/s41592-024-02525-x · Cited by 104
LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., & Stone, P. (2023) · arXiv · doi:10.48550/arxiv.2304.11477 · Cited by 104
AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API CallsPreprint
Du, Y., Wei, F., & Zhang, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.04253 · Cited by 103
Tian, Y., Li, X., Zhang, H., Zhao, C., Li, B., Wang, X., Wang, X., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3307012 · Cited by 102
ScreenAgent: A Vision Language Model-driven Computer Control Agent
Niu, R., Li, J., Wang, S., Fu, Y., Hu, X., Leng, X., Kong, H., Chang, Y., & Wang, Q. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/711 · Cited by 102
Guan, L., Valmeekam, K., Sreedharan, S., & Subbarao Kambhampati, B. (2023) · neural information processing systems · doi:10.52202/075280-3459 · Cited by 111
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101
“While prior work often retrieves only once at the beginning, Jiang et al. (2023) propose to adaptively retrieve passages for generation on top of a proprietary LLM or Schick et al. (2023) train an LM to generate API calls for named entities.”
BackgroundInfluentialHow to Bridge the Gap Between Modalities: Survey on Multimodal Large Language Model
Song, S., Li, X., Li, S., Zhao, S., Yu, J., Ma, J., Mao, X., Zhang, W., & Wang, M. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3527978 · Cited by 105
Prompt Injection attack against LLM-integrated Applications
Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., Zhang, T., Liu, Y., Wang, H., Zheng, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05499 · Cited by 100
Large Language Models as Analogical ReasonersPreprint
Yasunaga, M., Chen, X., Li, Y., Pasupat, P., Leskovec, J., Liang, P., Ed, H., & Zhou, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01714 · Cited by 108
RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
Wang, Z., Liu, Z., Zhang, Y., Zhong, A., Wang, J., Yin, F., Fan, L., Wu, L., & Wen, Q. (2024) · Proceedings of the 33rd ACM International Conference on Information and Knowledge Management · doi:10.1145/3627673.3680016 · Cited by 99
Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024) · Review of Accounting Studies · doi:10.1007/s11142-024-09833-9 · Cited by 195
Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks
Gao, Y., Xiong, Y., Wang, M., & Wang, H. (2026) · ACM Transactions on Information Systems · doi:10.1145/3837856 · Cited by 104
AI Reasoning in Deep Learning Era: From Symbolic AI to Neural–Symbolic AI
Liang, B., Wang, Y., & Tong, C. (2025) · Mathematics · doi:10.3390/math13111707 · Cited by 100
MathChat: Converse to Tackle Challenging Math Problems with LLM Agents
Wu, Y., Jia, F., Zhang, S., Li, H., Zhu, E., Wang, Y., Lee, Y. T., Peng, R., Wu, Q., & Wang, C. (2023) · arXiv · doi:10.48550/arxiv.2306.01337 · Cited by 98
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
Yang, K., Swope, A., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R. J., & Anandkumar, A. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0944 · Cited by 97
Process Reward Models for LLM Agents: Practical Framework and DirectionsPreprint
Choudhury, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.10325 · Cited by 97
Change-Agent: Toward Interactive Comprehensive Remote Sensing Change Interpretation and Analysis
Liu, C., Chen, K., Zhang, H., Qi, Z., Zou, Z., & Shi, Z. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3425815 · Cited by 96
Proactive Conversational Agents in the Post-ChatGPT World
Liao, L., Yang, G. H., & Shah, C. S. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3594250 · Cited by 95
Augmenting Black-box LLMs with Medical Textbooks for Biomedical Question Answering
Wang, Y., Ma, X., & Chen, W. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.95 · Cited by 93
Large Language Models can Learn RulesPreprint
Zhu, Z., Xue, Y., Chen, X., Zhou, D., Tang, J., Schuurmans, D., & Dai, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07064 · Cited by 95
Assessing LLMs in malicious code deobfuscation of real-world malware campaigns
Patsakis, C., Casino, F., & Lykousas, N. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124912 · Cited by 92
Xu, S., Pang, L., Shen, H., Cheng, X., & Chua, T. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645363 · Cited by 91
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Yang, W., Bi, X., Lin, Y., Chen, S., Zhou, J., & Sun, X. (2024) · neural information processing systems · doi:10.52202/079017-3201 · Cited by 91
Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World DataPreprint
Marchal, N., Xu, R., Elasmar, R., Gabriel, I., Goldberg, B., & Isaac, W. S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.13843 · Cited by 91
Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations
Huang, X., Lian, J., Lei, Y., Yao, J., Lian, D., & Xie, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3731446 · Cited by 89
Self-Reflection in LLM Agents: Effects on Problem-Solving Performance
Renze, M., & Guven, E. (2024) · arXiv · doi:10.48550/arxiv.2405.06682 · Cited by 89
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 87
A Large Language Model Enhanced Conversational Recommender SystemPreprint
Feng, Y., Liu, S., Xue, Z., Cai, Q., Hu, L., Jiang, P., Gai, K., & Sun, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.06212 · Cited by 87
NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
Shao, M., Jancheska, S., Udeshi, M., Dolan-Gavitt, B., Xi, H., Milner, K., Chen, B., Yin, M., Garg, S., Krishnamurthy, P., et al. (2024) · neural information processing systems · doi:10.52202/079017-1832 · Cited by 86
Planning In Natural Language Improves LLM Search For Code GenerationPreprint
Wang, E., Cassano, F., Wu, C., Bai, Y., Song, W., Nath, V., Han, Z., Hendryx, S., Yue, S., & Zhang, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.03733 · Cited by 85
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84
Chen, J., Yuan, S., Ye, R., Majumder, B. P., & Richardson, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05746 · Cited by 84
DrugAgent: Automating AI-aided Drug Discovery Programming through LLM Multi-Agent CollaborationPreprint
Liu, S., Lu, Y., Chen, S., Hu, X., Zhao, J., Fu, T., & Zhao, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.15692 · Cited by 84
MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP ServersPreprint
Wang, Z., Gao, Y., Wang, Y., Liu, S., Sun, H., Cheng, H., Shi, G., Du, H., & Li, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.14925 · Cited by 88
Alpha-GPT: Human-AI Interactive Alpha Mining for Quantitative Investment
Wang, S., Yuan, H., Zhou, L. L., Ni, L. M.-S., Shum, H., & Guo, J. (2025) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2025.emnlp-demos.14 · Cited by 83
Buehler, M. J. (2024) · ACS Engineering Au · doi:10.1021/acsengineeringau.3c00058 · Cited by 82
Strategic Reasoning with Language ModelsPreprint
Gandhi, K., Sadigh, D., & Goodman, N. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.19165 · Cited by 85
ChatDev: Communicative Agents for Software Development
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.07924 · Cited by 81
Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning
Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-024-00345-9 · Cited by 80
Generative artificial intelligence, patient safety and healthcare quality: a review
Howell, M. D. (2024) · BMJ Quality & Safety · doi:10.1136/bmjqs-2023-016690 · Cited by 83
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
Kim, J., Lee, J. H., Kim, S., Park, J., Yoo, K. M., Kwon, S. J., & Lee, D. (2023) · neural information processing systems · doi:10.52202/075280-1569 · Cited by 80
Interactive Natural Language ProcessingPreprint
Wang, Z., Zhang, G., Yang, K., Shi, N., Zhou, W., Hao, S., Xiong, G., Li, Y., Sim, M. Y., Chen, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13246 · Cited by 90
Agent-based learning of materials datasets from the scientific literature
Ansari, M., & Moosavi, S. M. (2024) · Digital Discovery · doi:10.1039/d4dd00252k · Cited by 82
Retrieving Multimodal Information for Augmented Generation: A SurveyPreprint
Zhao, R. C., Chen, H., Wang, W., Jiao, F., Do, X. L., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.10868 · Cited by 87
Foundation Models for Music: A SurveyPreprint
Ma, Y., Øland, A., Ragni, A., Sette, B. M. D., Saitis, C., Donahue, C., Lin, C., Plachouras, C., Benetos, E., Shatri, E., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.14340 · Cited by 83
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Efficient Prompting for LLM-Based Generative Internet of Things
Xiao, B., Kantarcı, B., Kang, J., Niyato, D. T., & Guizani, M. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3470210 · Cited by 77
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O'Keefe, C., Whittlestone, J., Avin, S., Brundage, M., Bullock, J., Cass-Beggs, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03718 · Cited by 77
GPT Can Solve Mathematical Problems Without a CalculatorPreprint
Yang, Z., Ding, M., Lv, Q., Jiang, Z., He, Z., Guo, Y., Bai, J., & Tang, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.03241 · Cited by 77
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 75
Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem SolvingPreprint
Mai, X., Xu, H., Li, Z., W, X., Wang, W., Hu, J., Zhang, Y., & Zhang, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.07773 · Cited by 75
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75
ChatGPT Participates in a Computer Science ExamPreprint
Bordt, S., & Ulrike Von Luxburg, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09461 · Cited by 77
Towards Evaluating AI Systems for Moral Status Using Self-ReportsPreprint
Perez, E., & Long, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.08576 · Cited by 78
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
Zhang, W., Zhao, L., Xia, H., Sun, S., Sun, J., Qin, M., Li, X., Zhao, Y., Zhao, Y., Cai, X., et al. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671801 · Cited by 74
Grammar Prompting for Domain-Specific Language Generation with Large Language Models
Wang, B., Wang, Z., Wang, X., Cao, Y., Saurous, R. A., & Kim, Y. (2023) · neural information processing systems · doi:10.52202/075280-2837 · Cited by 74
An Empirical Evaluation of LLMs for Solving Offensive Security ChallengesPreprint
Shao, M., Chen, B., Jancheska, S., Dolan-Gavitt, B., Garg, S., Karri, R., & Shafique, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.11814 · Cited by 74
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.813 · Cited by 73
Metadata
- Authors
- Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T.
- ORCID
- Zettlemoyer, Luke
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 68539-68551
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.04761
- OpenAlex
- W7133214447
- Cited by
- 1,150
- References
- 0
- Citations by year
- 2026: 730 · 2025: 136 · 2024: 165 · 2023: 119
- FWCI
- 492.67
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- tool use, language models, self-supervised learning, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
