PreprintCited by 159Speech and dialogue systemsTopic ModelingAI in Service Interactions

Beyond Goldfish Memory: Long-Term Open-Domain Conversation

Open access
Xu, Jing, Szlam, Arthur, Weston, Jason·2021·arXiv

Abstract

Despite recent improvements in open-domain dialogue models, state of the art models are trained and evaluated on short conversations with little context. In contrast, the long-term conversation setting has hardly been studied. In this work we collect and release a human-human dataset consisting of multiple chat sessions whereby the speaking partners learn about each other's interests and discuss the things they have learnt from past sessions. We show how existing models trained on existing datasets perform poorly in this long-term conversation setting in both automatic and human evaluations, and we study long-context models that can perform much better. In particular, we find retrieval-augmented methods and methods with an ability to summarize and recall previous conversations outperform the standard encoder-decoder architectures currently considered state of the art.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.07567

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MSC is a human-human multi-session open-domain dialogue dataset in which speakers learn about each other across repeated conversations.

    “MSC [27] is a human-human multi-session open-domain dialogue dataset in which speakers learn about each other across repeated conversations.”

    From MRMMIA: Membership Inference Attacks on Memory in Chat Agents (2026) · cites this paper as [27]

  • OpenAI released a dataset, PRM800K, which resulted in a state-of-the-art reward model.

    “Thus, OpenAI released a dataset, namely, PRM800K, which resulted in a SOTA reward model [153].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [153]

    The Introduction describes the authors’ Multi-Session Chat dataset and models for long-term conversation, not PRM800K. Searches for “PRM800K” and “process reward model” found no passage in our copy addressing OpenAI’s release or the claimed reward-model result. (LLM, conf=0.76)

    Experimental — accuracy not yet measured
  • The effectiveness of the system is constrained by the performance of the reward model itself.

    “Since the effectiveness of the system is constrained by the performance of the reward model itself, emphasis has been placed on determining the most effective training methods [153].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [153]

    The Predicted summary models and 5.1 Human Evaluation sections discuss dialogue summaries and conversational performance, not a reward model. Searches for “reward model” and the claimed performance constraint found no passage addressing it; page numbers were not provided. (LLM, conf=0.73)

    Experimental — accuracy not yet measured
  • Memory-augmented methods improve the memory capabilities of LLMs without requiring fine-tuning.

    “Numerous approaches have demonstrated that memory-augmented methods improve the memory capabilities of LLMs without requiring fine-tuning, which is often impossible for API-based models [25, 42, 46].”
  • Memory-augmented methods retrieve user-specific information from past interactions, addressing the limited context window and enabling integration of past information into the input prompt.

    “Memory-augmented methods extend this concept by retrieving user-specific information from past interactions. This addresses the challenge of a limited context window and enables the integration of past information into the input prompt [25, 29, 45].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 41 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Xu, J., Szlam, A., & Weston, J.
Affiliations
Meta (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
5180-5197
ISSN
0736-587X (ISSN-L)
OpenAlex
W3186804217
Cited by
148
References
41
Citations by year
2026: 21 · 2025: 25 · 2024: 22 · 2023: 45 · 2022: 30 · 2021: 5
FWCI
16.87
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
Keywords
open-domain dialogue, long-context models, retrieval-augmented generation, dialogue summarization, chatbot evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review