CoChat
Sign Up
PreprintCited by 1,345Topic ModelingNatural Language Processing TechniquesNeural Networks and Applications

End-to-end memory networks

Open access
Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, Fergus, Rob·2015·arXiv
DOI 10.48550/arxiv.1503.08895arXiv 1503.08895
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (Weston et al., 2015) but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings. It can also be seen as an extension of RNNsearch to the case where multiple computational steps (hops) are performed per output symbol. The flexibility of the model allows us to apply it to tasks as diverse as (synthetic) question answering and to language modeling. For the former our approach is competitive with Memory Networks, but with less supervision. For the latter, on the Penn TreeBank and Text8 datasets our approach demonstrates comparable performance to RNNs and LSTMs. In both cases we show that the key concept of multiple computational hops yields improved results.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1503.08895

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Memory-augmented neural networks (MemN2N, NTM, DNC) focus on algorithmic tasks rather than long-term personalization.

    “MemN2N [20], NTM [21], and DNC [22] introduced differentiable external memory for neural networks. However, these focus on algorithmic tasks rather than long-term personalization.”

    From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [20]

    The Abstract (p. 1) and Conclusions (p. 8) describe MemN2N's applications as synthetic question answering and language modeling. Related Work (p. 3) explicitly distinguishes its textual reasoning tasks from the abstract sorting and recall tasks tackled by the NTM. The paper supports MemN2N's use of external memory, but contradicts the characterization of MemN2N as focused on algorithmic tasks; it does not assess long-term personalization. (LLM, conf=0.86)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • S-FFN is viewed as a form of neural memory, and like a sparse memory, it only activates a subset of memory cells upon retrieval.

    “S FFN is viewed as a form of neural memory (Sukhbaatar et al. 2015), and like a sparse memory, it only activates a subset of memory cells upon retrieval (Lample et al. 2019).”

    From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Sukhbaatar et al. 2015

    The Introduction (p. 1) describes a neural network that reads from external memory, supporting the cited paper as an example of neural memory. It does not discuss S-FFN or establish that S-FFN is a form of neural memory. The subset-activation clause is attributed to Lample et al., not to this source. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 21

  1. [2][2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR) , 2015. -
  2. [4][4] J. Chung, C ¸ . G¨ ulc ¸ehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint: 1412.3555 , 2014. - · arXiv
  3. [5][5] S. Das, C. L. Giles, and G.-Z. Sun. Learning context-free grammars: Capabilities and limitations of a recurrent neural network with an external stack memory. In In Proceedings of The Fourteenth Annual Conference of Cognitive Science Society , 1992. -
  4. [6][6] J. Goodman. A bit of progress in language modeling. CoRR , cs.CL/0108005, 2001. - · arXiv
  5. [7][7] A. Graves. Generating sequences with recurrent neural networks. arXiv preprint: 1308.0850 , 2013. - · arXiv
  6. [8][8] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint: 1410.5401 , 2014. - · arXiv
  7. [9][9] K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: A recurrent neural network for image generation. CoRR , abs/1502.04623, 2015. - · arXiv
  8. [11][11] A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. NIPS , 2015. -
  9. [12][12] J. Koutn´ ık, K. Greff, F. J. Gomez, and J. Schmidhuber. A clockwork RNN. In ICML , 2014. -
  10. [13][13] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english: The Penn Treebank. Comput. Linguist. , 19(2):313-330, June 1993. -
  11. [14][14] T. Mikolov. Statistical language models based on neural networks. Ph. D. thesis, Brno University of Technology , 2012. -
  12. [15][15] T. Mikolov, A. Joulin, S. Chopra, M. Mathieu, and M. Ranzato. Learning longer memory in recurrent neural networks. arXiv preprint: 1412.7753 , 2014. - · arXiv
  13. [16][16] M. C. Mozer and S. Das. A connectionist symbol manipulator that discovers the structure of context-free languages. NIPS , pages 863-863, 1993. -
  14. [17][17] B. Peng, Z. Lu, H. Li, and K. Wong. Towards Neural Network-based Reasoning. ArXiv preprint: 1508.05508 , 2015. - · arXiv
  15. [18][18] J. Pollack. The induction of dynamical recognizers. Machine Learning , 7(2-3):227-252, 1991. -
  16. [19][19] K. Steinbuch and U. Piske. Learning matrices and their applications. IEEE Transactions on Electronic Computers , 12:846-862, 1963. -
  17. [20][20] M. Sundermeyer, R. Schl¨ uter, and H. Ney. LSTM neural networks for language modeling. In Interspeech , pages 194-197, 2012. -
  18. [21][21] W. K. Taylor. Pattern recognition by means of automatic analogue apparatus. Proceedings of The Institution of Electrical Engineers , 106:198-209, 1959. -
  19. [22][22] J. Weston, A. Bordes, S. Chopra, and T. Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint: 1502.05698 , 2015. - · arXiv
  20. [23][23] J. Weston, S. Chopra, and A. Bordes. Memory networks. In International Conference on Learning Representations (ICLR) , 2015. -
  21. [25][25] W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014. · arXiv

Cited by

Sorted by citations

Authors
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R.
Affiliations
  • Courant Institute of Mathematical Sciences (US)
  • Meta (United States) (US)
ORCID
Fergus, Rob
Venue
arXiv
Publisher
arXiv
Published
2015
Volume
28
Pages
2440-2448
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1503.08895
arXiv
1503.08895
OpenAlex
W1793121960
Cited by
1,345
References
2
Citations by year
2026: 2 · 2025: 52 · 2024: 78 · 2023: 116 · 2022: 128 · 2021: 196 · 2020: 229 · 2019: 251 · 2018: 156 · 2017: 90 · 2016: 44 · 2015: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
memory networks, recurrent attention, external memory, end-to-end learning, question answering, language modeling, supervised learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review