CoChat
Sign Up
PreprintCited by 302Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

Open access
Khandelwal, Urvashi, He, He, Qi, Peng, Jurafsky, Dan·2018·arXiv
DOI 10.48550/arxiv.1805.04623arXiv 1805.04623
Open this paper in CoChat — free accountOpen PDF

Abstract

We know very little about how neural language models (LM) use prior linguistic context. In this paper, we investigate the role of context in an LSTM LM, through ablation studies. Specifically, we analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped. On two standard datasets, Penn Treebank and WikiText-2, we find that the model is capable of using about 200 tokens of context on average, but sharply distinguishes nearby context (recent 50 tokens) from the distant history. The model is highly sensitive to the order of words within the most recent sentence, but ignores word order in the long-range context (beyond 50 tokens), suggesting the distant past is modeled only as a rough semantic field or topic. We further find that the neural caching model (Grave et al., 2017b) especially helps the LSTM to copy words from within this distant context. Overall, our analysis not only provides a better understanding of how neural LMs use their context, but also sheds light on recent success from cache-based models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1805.04623

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs lack native mechanisms for stable and structured memory, resulting in inconsistent recall, degraded long-horizon reasoning, and limited support for tasks requiring persistent and organized memory.

    “Moreover, LLMs lack native mechanisms for stable and structured memory, resulting in inconsistent recall, degraded long-horizon reasoning, and limited support for tasks requiring persistent and organized memory (Khandelwal et al., 2018; Maharana et al., 2024).”

    From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Khandelwal et al., 2018

    The Abstract and §6.1 find that the LSTM language model uses distant context mainly as a rough semantic representation rather than reliably copying specific distant words. Section 6.2 reports that an external cache improves long-range word copying. These findings support a narrower concern about long-range recall, but the paper does not establish that LLMs generally lack stable, structured memory or that this causes degraded long-horizon reasoning and poor performance on tasks requiring persistent, organized memory. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 27

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine- grained analysis of sentence embeddings using auxiliary prediction tasks. International Con- ference on Learning Representations (ICLR) https://openreview.net/pdf?id=BJh6Ztuxl.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations (ICLR) https://arxiv.org/pdf/1409.0473.pdf. · arXiv
  3. Jordan Boyd-Graber and David Blei. 2009. Syn- tactic topic models. In Advances in neu- ral information processing systems. pages 185- 192. https://papers.nips.cc/paper/3398-syntactic- topic-models.pdf.
  4. Ciprian Chelba, Mohammad Norouzi, and Samy Bengio. 2017. N-gram language model- ing using recurrent neural network esti- mation. arXiv preprint arXiv:1703.10724 https://arxiv.org/pdf/1703.10724.pdf. · arXiv
  5. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. Interna- tional Conference on Machine Learning (ICML) https://arxiv.org/pdf/1612.08083.pdf. · arXiv
  6. Yarin Gal and Zoubin Ghahramani. 2016. A theoret- ically grounded application of dropout in recurrent neural networks. In Advances in neural informa- tion processing systems (NIPS). pages 1019-1027. https://arxiv.org/pdf/1512.05287.pdf. · arXiv
  7. Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry Heck. 2016. Contextual lstm (clstm) models for large scale nlp tasks. Work- shop on Large-scale Deep Learning for Data Min- ing, KDD https://arxiv.org/pdf/1602.06291.pdf. · arXiv
  8. Edouard Grave, Moustapha M Cisse, and Ar- mand Joulin. 2017a. Unbounded cache model for online language modeling with open vo- cabulary. In Advances in Neural Information Processing Systems (NIPS). pages 6044-6054. https://papers.nips.cc/paper/7185-unbounded- cache-model-for-online-language-modeling-with- open-vocabulary.pdf.
  9. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017b. Improving Neural Language Mod- els with a Continuous Cache. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=B184E5qee.
  10. Alex Graves. 2013. Generating se- quences with recurrent neural net- works. arXiv preprint arXiv:1308.0850 https://arxiv.org/pdf/1308.0850.pdf. · arXiv
  11. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks princi- ple: Reading children's books with explicit memory representations. International Con- ference on Learning Representations (ICLR) https://arxiv.org/pdf/1511.02301.pdf. · arXiv
  12. Sepp Hochreiter and Jürgen Schmidhu- ber. 1997. Long short-term memory. Neural computation 9(8):1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735. · DOI · Paper page
  13. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. Inter- national Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=r1aPbsFle.
  14. Rafal Jozefowicz, Oriol Vinyals, Mike Schus- ter, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language mod- eling. arXiv preprint arXiv:1602.02410 https://arxiv.org/pdf/1602.02410.pdf. · arXiv
  15. Jey Han Lau, Timothy Baldwin, and Trevor Cohn. 2017. Topically Driven Neural Language Model. Association for Computational Linguistics (ACL) https://doi.org/10.18653/v1/P17-1033. · DOI
  16. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. North American As- sociation of Computational Linguistics (NAACL) http://www.aclweb.org/anthology/N16-1082.
  17. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics (TACL) http://aclweb.org/anthology/Q16-1037.
  18. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David Mc- Closky. 2014. The stanford corenlp natural lan- guage processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations. pages 55-60. https://doi.org/10.3115/v1/P14-5010. · DOI
  19. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn tree- bank. Computational linguistics 19(2):313-330. http://aclweb.org/anthology/J93-2004.
  20. Gabor Melis, Chris Dyer, and Phil Blunsom. 2018. On the State of the Art of Evalua- tion in Neural Language Models. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=ByJHuTgA-.
  21. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and Optimizing LSTM Language Models. International Con- ference on Learning Representations (ICLR) https://openreview.net/pdf?id=SyyGPP0TZ.
  22. Stephen Merity, Caiming Xiong, James Brad- bury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. International Con- ference on Learning Representations (ICLR) https://openreview.net/pdf?id=Byj72udxe.
  23. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recur- rent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association.
  24. Ofir Press and Lior Wolf. 2017. Using the output em- bedding to improve language models. European Chapter of the Association for Computational Lin- guistics http://aclweb.org/anthology/E17-2025.
  25. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In International Con- ference on Machine Learning (ICML). pages 1058- 1066.
  26. Tian Wang and Kyunghyun Cho. 2016. Larger-Context Language Modelling with Recurrent Neural Net- work. Association for Computational Linguistics (ACL) https://doi.org/10.18653/v1/P16-1125. · DOI
  27. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2018. Breaking the softmax bottleneck: a high-rank rnn language model. Inter- national Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=HkwZSG- CZ.

Cited by

Sorted by citations

Authors
Khandelwal, U., He, H., Qi, P., & Jurafsky, D.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
284-294
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.1805.04623
arXiv
1805.04623
OpenAlex
W2963951265
Cited by
302
References
0
Citations by year
2026: 9 · 2025: 9 · 2024: 24 · 2023: 42 · 2022: 43 · 2021: 55 · 2020: 48 · 2019: 57 · 2018: 15
FWCI
17.87
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1514268)
  • Defense Advanced Research Projects Agency
  • Army Research Office
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
neural language models, long-range context, LSTM, context length, perplexity, semantic field, topic modeling, ablation study, Penn Treebank, WikiText-2

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review