Cited by 200AI-based Problem Solving and PlanningDomain Adaptation and Few-Shot LearningTopic Modeling

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 11 of the 20 sources its claims cite were checked for support.

Expel: LLM agents are experiential learners

Open access
Zhao, Andrew, Huang, Daniel, Xu, Quentin, Lin, Matthieu, Liu, Yong-Jin, Huang, Gao·2024·Proceedings of the AAAI Conference on Artificial Intelligence

Abstract

The recent surge in research interest in applying large language models (LLMs) to decision-making tasks has flourished by leveraging the extensive world knowledge embedded in LLMs. While there is a growing demand to tailor LLMs for custom decision-making tasks, finetuning them for specific tasks is resource-intensive and may diminish the model's generalization capabilities. Moreover, state-of-the-art language models like GPT-4 and Claude are primarily accessible through API calls, with their parametric weights remaining proprietary and unavailable to the public. This scenario emphasizes the growing need for new methodologies that allow learning from agent experiences without requiring parametric updates. To address these problems, we introduce the Experiential Learning (ExpeL) agent. Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks. At inference, the agent recalls its extracted insights and past experiences to make informed decisions. Our empirical results highlight the robust learning efficacy of the ExpeL agent, indicating a consistent enhancement in its performance as it accumulates experiences. We further explore the emerging capabilities and transfer learning potential of the ExpeL agent through qualitative observations and additional experiments.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

ExpeL is an LLM agent that learns from its own task experiences without parameter updates by storing trajectories, extracting natural-language insights, and retrieving similar successful trajectories for inference. It improves over ReAct/Act baselines across HotpotQA, ALFWorld, and WebShop and shows forward transfer from HotpotQA to FEVER.

Key findings

  • ExpeL consistently outperformed ReAct and Act across HotpotQA, ALFWorld, and WebShop; in ablations, ExpeL reached 39.0 ± 1.7 success on HotpotQA vs. ReAct 28.0 ± 1.4, and 59.0 ± 0.3 on ALFWorld vs. ReAct 40.0 ± 0.3.
  • Both insight extraction and retrieval mattered: insights-only/retrieval-only variants scored 36%/31% on HotpotQA, 50%/55% on ALFWorld, and 37%/38% on WebShop, with WebShop scores of 0.675/0.67, respectively.
  • Cross-task learning matched Reflexion on HotpotQA (40% at R3 vs. 39%) and outperformed it on ALFWorld (54% at R3 vs. 59%) without repeated attempts.
  • Transfer from HotpotQA to FEVER improved performance for both transfer agents, with the agent using few in-context examples improving more; ablations showed learned insights beat hand-crafted ones (32.0 ± 1.1), reflections hurt (29.0 ± 0.4), gpt-3.5 insights were lower (32.0 ± 0.4), and task-similarity retrieval was best (59.0 ± 0.3 vs. reasoning similarity 48.5 ± 2.1 and random 42.5 ± 0.8).

Methods

  • ExpeL uses ReAct as its base planner; during training it gathers experiences by retrying tasks up to Z times with Reflexion, stores trajectories in an experience pool, and extracts natural-language insights via LLM operators ADD/EDIT/UPVOTE/DOWNVOTE from fail/success pairs and lists of successes. At evaluation, it augments task specifications with insights and retrieves top-k successful trajectories by task similarity using Faiss/kNN/all-mpnet-base-v2.
  • Evaluated on HotpotQA, ALFWorld, WebShop, and FEVER transfer; four-fold validation, success rate (exact match for HotpotQA/FEVER, task completion for ALFWorld, matching item for WebShop), plus WebShop reward and ALFWorld task-type breakdown. Baselines were ReAct, Act, and imitation learning; actions used gpt-3.5-turbo-0613, insights used gpt-4-0613, with temperature 0 and greedy decoding.
  • Transfer used HotpotQA as source and FEVER as target, adapting source insights with target fewshot examples and comparing against ReAct, Act, and transfer without task demonstrations.

Limitations

  • Stated: the work investigates textual observations only, limiting real-world applicability; closed-source API LLMs may be off-limits in some applications; insights currently fit the context window but lifelong learning may need extra retrieval; prompting techniques lack the theoretical underpinnings of reinforcement learning.
  • Evident: only deterministic environments are considered; evaluation is limited to four text-based benchmarks with four-fold validation; transfer assumes source and target tasks share common knowledge.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1609/aaai.v38i17.29936

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 of 20 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured

Show all 42 claims

  • Probably not a substantial claim. 18
  • It may state two separate claims as one. 24
  • The quote only partly states this claim. 1
36 of 36
  • The authors state this as an assumption.

    The source and target tasks exhibit common knowledge.

    “Like most transfer learning settings, we assume that the source and target tasks exhibit common knowledge.”
  • The paper relies on this without support.

    Some of the most potent language models are proprietary.

    “Furthermore, some of the most potent language models are proprietary.”
  • The paper relies on this without support.

    WebShop presents a unique challenge, requiring both website-based reasoning and precise execution of actions.

    “Furthermore, WebShop presents a unique challenge, requiring both website-based reasoning (price comparisons, query reformulation, etc.) and precise execution of actions (searching, clicking, option selection, etc.).”
  • The paper relies on this without support.

    Human learning occurs mainly either by storing successful trajectories in memory or by extracting high-level insights from experiences.

    “Human learning occurs mainly either by storing successful trajectories in memory, which can be later recalled as specific examples, or by extracting high-level insights from experiences, enabling generalization to novel situations.”
  • The paper relies on this without support.

    The source and target tasks are inherently different, so no experience pool can be retrieved from for transfer.

    “Notice that since source and target tasks are inherently different, we do not have an experience pool to retrieve from; thus, the ExpeL Transfer agents use the existing fixed fewshot examples as in-context examples.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Show all 31 references

Research works printed but not linked: 6

  1. NeurIPS.
  2. Du, M.; He, F.; Zou, N.; Tao, D.; and Hu, X. 2022. Shortcut Learning of Large Language Models in Natural Language Understanding: A Survey. arXiv preprint.
  3. NeurIPS.
  4. Maas; Carey; Wheeler; Saatchi; Billington; and Shamash. 2023. To Infinity and Beyond: SHOW-1 and Showrunner Agents in Multi-Agent Simulations. arXiv preprint.
  5. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023a. LLaMA: Open and Efficient Foun- dation Language Models. arXiv preprint.
  6. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint.

Cited by

Sorted by citations

Items 101–200 of 200

Metadata

Authors
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G.
Affiliations
Tsinghua University (CN)
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Published
2024-03-24
Volume
38
Issue
17
Pages
19632-19642
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
OpenAlex
W4393160747
Cited by
153
References
31
Citations by year
2026: 61 · 2025: 63 · 2024: 28
FWCI
44.76
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Natural Science Foundation of China (U2336214, 62332019, 62022048)
  • Tsinghua University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, LLM agents, experiential learning, decision-making, transfer learning, nonparametric learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review