CoChat
Sign Up
PreprintCited by 0Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory

Open access
He, Zexue, Karlinsky, Leonid, Kim, Donghyun, McAuley, Julian, Krotov, Dmitry, Feris, Rogerio·2024·arXiv
DOI 10.48550/arxiv.2402.13449arXiv 2402.13449
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) struggle to handle long input sequences due to high memory and runtime costs. Memory-augmented models have emerged as a promising solution to this problem, but current methods are hindered by limited memory capacity and require costly re-training to integrate with a new LLM. In this work, we introduce an associative memory module which can be coupled to any pre-trained (frozen) attention-based LLM without re-training, enabling it to handle arbitrarily long input sequences. Unlike previous methods, our associative memory module consolidates representations of individual tokens into a non-parametric distribution model, dynamically managed by properly balancing the novelty and recency of the incoming data. By retrieving information from this consolidated associative memory, the base LLM can achieve significant (up to 29.7% on Arxiv) perplexity reduction in long-context modeling compared to other baselines evaluated on standard benchmarks. This architecture, which we call CAMELoT (Consolidated Associative Memory Enhanced Long Transformer), demonstrates superior performance even with a tiny context window of 128 tokens, and also enables improved in-context learning with a much larger set of demonstrations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.13449

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing memory-augmented approaches often rely on LRU eviction for memory management, which may not be optimal for retaining highly relevant information.

    “However, these approaches often rely on LRU eviction for memory management, which may not be optimal for retaining highly relevant information.”

    From Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models (2026) · cites this paper as [4]

    Section 4.1.1 (page not available) says the Memorizing Transformer replaces older caches with newer ones as its memory fills. Section 4.1.2 says this can overwrite older memories and limit long-term utility. The paper does not establish that memory-augmented approaches often use LRU eviction specifically; the example it describes uses a circular cache. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • CAMELoT introduces an associative memory module to improve context handling within LLMs.

    “CAMELoT [4] introduces an associative memory module to improve context handling within LLMs.”

    From Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models (2026) · cites this paper as [4]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Research works printed but not linked: 38

  1. Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. · arXiv
  2. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. · arXiv
  3. Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. · arXiv
  4. Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 35:16344-16359, 2022.
  5. Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023. · arXiv
  6. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
  7. Ge, T., Hu, J., Wang, X., Chen, S.-Q., and Wei, F. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945, 2023. · arXiv
  8. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. · arXiv
  9. Graves, A., Wayne, G., Reynolds, M., Harley, T., Dani- helka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471-476, 2016.
  10. Hao, Y., Sun, Y., Dong, L., Han, Z., Gu, Y., and Wei, F. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713, 2022. · arXiv
  11. He, Z., Wang, Y., Yan, A., Liu, Y., Chang, E., Gentili, A., McAuley, J., and Hsu, C.-n. Medeval: A multi-level, multi-task, and multi-domain medical benchmark for lan- guage model evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8725-8744, 2023.
  12. Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2558, 1982.
  13. Kanerva, P. Sparse distributed memory. MIT press, 1988.
  14. Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020. · arXiv
  15. Kohonen, T. Associative memory: A system-theoretical ap- proach, volume 17. Springer Science & Business Media, 2012.
  16. Krotov, D. and Hopfield, J. J. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
  17. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
  18. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilac- qua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023. · arXiv
  19. Mu, J., Li, X. L., and Goodman, N. Learning to compress prompts with gist tokens. arXiv preprint arXiv:2304.08467, 2023. · arXiv
  20. Packer, C., Fang, V., Patil, S. G., Lin, K., Wooders, S., and Gonzalez, J. E. Memgpt: Towards llms as operating systems. arXiv preprint arXiv:2310.08560, 2023. · arXiv
  21. Park, S. and Bak, J. Memoria: Hebbian memory architecture for human-like sequential processing. arXiv preprint arXiv:2310.03052, 2023. · arXiv
  22. Polu, S. and Sutskever, I. Generative language model- ing for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020. · arXiv
  23. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
  24. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  25. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long- range sequence modelling. arXiv preprint, 2019a. URL https://arxiv.org/abs/1911.05507. · arXiv
  26. Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019b. · arXiv
  27. Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Lev- skaya, A., and Shlens, J. Stand-alone self-attention in vision models. Advances in neural information process- ing systems, 32, 2019.
  28. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D., Kopp, M. K., et al. Hopfield networks is all you need. In International Conference on Learning Representations, 2021.
  29. Sara, S. J. Retrieval and reconsolidation: toward a neurobi- ology of remembering. Learning & memory, 7(2):73-84, 2000.
  30. Tworkowski, S., Staniszewski, K., Pacek, M., Wu, Y., Michalewski, H., and Miłoś, P. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170, 2023. · arXiv
  31. Tyulmankov, D., Fang, C., Vadaparty, A., and Yang, G. R. Biological learning in key-value memory networks. Ad- vances in Neural Information Processing Systems, 34: 22247-22258, 2021.
  32. Vig, J., Learning, M., and Belinkov, Y. Analyzing the structure of attention in a transformer language model. ACL 2019, pp. 63, 2019.
  33. Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. · arXiv
  34. Wang, W., Dong, L., Cheng, H., Liu, X., Yan, X., Gao, J., and Wei, F. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174, 2023. · arXiv
  35. Weston, J., Chopra, S., and Bordes, A. Memory networks. arXiv preprint arXiv:1410.3916, 2014. · arXiv
  36. Willshaw, D. J., Buneman, O. P., and Longuet-Higgins, H. C. Non-holographic associative memory. Nature, 222(5197): 960-962, 1969.
  37. Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C. Mem- orizing transformers. arXiv preprint arXiv:2203.08913, 2022. · arXiv
  38. Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Al- berti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer se- quences. Advances in neural information processing systems, 33:17283-17297, 2020.

Authors
He, Z., Karlinsky, L., Kim, D., McAuley, J., Krotov, D., & Feris, R.
Affiliations
  • IBM (United States) (US)
  • Korea University (KR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.13449
arXiv
2402.13449
OpenAlex
W4392085961
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, associative memory, long-context modeling, memory-augmented transformer, in-context learning, perplexity reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review