CoChat
Sign Up
PreprintCited by 20Neural Networks and ApplicationsMemory and Neural MechanismsMemory Processes and Influences

Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

Open access
Millidge, Beren, Salvatori, Tommaso, Song, Yuhang, Lukasiewicz, Thomas, Bogacz, Rafal·2022·arXiv
DOI 10.48550/arxiv.2202.04557arXiv 2202.04557PMC PMC7614148
Open this paper in CoChat — free account

Abstract

A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possesses close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov et al (2020) to express general associative memory models using neural network dynamics with only second-order interactions between neurons, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2202.04557

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The three components of the generalized Hopfield network not only generalize existing models but also extend their functional capabilities.

    “This paper demonstrates that these three components not only generalize existing models but also extend their functional capabilities.”

    From The AI Hippocampus: How Far are We From Human Memory? (2026) · cites this paper as Millidge et al. (2022)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. Abu-Mostafa, Y., & Jacques, J. S. (1985). Information capacity of the Hopfield model. IEEE Transactions on Information Theory , 31 (4), 461-464.
  2. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . · arXiv
  3. Bricken, T., & Pehlevan, C. (2021). Attention approximates sparse distributed memory. arXiv preprint arXiv:2111.05498 . · arXiv
  4. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., . . . others (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . · arXiv
  5. Caputo, B., & Niemann, H. (2002). Storage capacity of kernel associative memories. In International conference on artificial neural networks (pp. 51-56).
  6. Chen, H., Lee, Y., Sun, G., Lee, H., Maxwell, T., & Giles, C. L. (1986). High order correlation model for associative memory. In Aip conference proceedings (Vol. 151, pp. 86-99).
  7. Demircigil, M., Heusel, J., L¨ owe, M., Upgang, S., & Vermet, F. (2017). On a model of associative memory with huge storage capacity. Journal of Statistical Physics , 168 (2), 288-299.
  8. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . · arXiv
  9. Geva, M., Schuster, R., Berant, J., & Levy, O. (2020). Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913 . · arXiv
  10. Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021). Memory-efficient transformers via topk attention. arXiv preprint arXiv:2106.06899 . · arXiv
  11. Hinton, G. E., & Anderson, J. A. (2014). Parallel models of associative memory: Updated edition . Psychology Press.
  12. Horn, D., & Usher, M. (1988). Capacities of multiconnected memory models. Journal de Physique , 49 (3), 389-395.
  13. Jiang, Y., & Pehlevan, C. (2020). Associative memory in iterated overparameterized sigmoid autoencoders. In International conference on machine learning (pp. 4828-4838).
  14. Kanerva, P. (1988). Sparse distributed memory . MIT Press.
  15. Kanerva, P. (1992). Sparse distributed memory and related models (Vol. 92). NASA Ames Research Center, Research Institute for Advanced Computer Science.
  16. Kaya, M., & Bilge, H. S ¸ . (2019). Deep metric learning: A survey. Symmetry , 11 (9), 1066.
  17. Keeler, J. D. (1988). Comparison between Kanerva's SDM and Hopfield-type neural networks. Cognitive Science , 12 (3), 299-329.
  18. Kirkpatrick, S., & Sherrington, D. (1978). Infinite-ranged models of spin-glasses. Physical Review B , 17 (11), 4384.
  19. Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020). Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 . · arXiv
  20. Krotov, D. (2021). Hierarchical associative memory. arXiv preprint arXiv:2107.06446 . · arXiv
  21. Krotov, D., & Hopfield, J. (2020). Large associative memory problem in neurobiology and machine learning. arXiv preprint arXiv:2008.06996 . · arXiv
  22. Krotov, D., & Hopfield, J. J. (2016). Dense associative memory for pattern recognition. Advances in Neural Information Processing Systems , 29 , 1172-1180.
  23. Kulis, B., et al. (2013). Metric learning: A survey. Foundations and Trends® in Machine Learning , 5 (4), 287-364.
  24. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog , 1 (8), 9.
  25. Radhakrishnan, A., Yang, K., Belkin, M., & Uhler, C. (2018). Memorization in overparameterized autoencoders. arXiv preprint arXiv:1810.10333 . · arXiv
  26. Ramsauer, H., Sch¨ afl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., . . . others (2020). Hopfield networks is all you need. arXiv preprint arXiv:2008.02217 . · arXiv
  27. Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A. (2019). Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470 . · arXiv
  28. Tang, F., & Kopp, M. (2021). A remark on a paper of Krotov and Hopfield [arxiv: 2008.06996]. arXiv preprint arXiv:2105.15034 . · arXiv
  29. Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2020). Efficient transformers: A survey. arXiv preprint arXiv:2009.06732 . · arXiv
  30. Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., . . . others (2021). MLP-mixer: An all-MLP architecture for vision. In Thirty-fifth conference on neural information processing systems.
  31. Tsodyks, M., & Sejnowski, T. (1995). Associative memory and hippocampal place cells. International Journal of Neural Systems , 6 , 81-86.
  32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
  33. Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 . · arXiv
  34. Wu, Y., Hu, J., Wu, W., Zhou, Y., & Du, K. (2012). Storage capacity of the Hopfield network associative memory. In 2012 5th international conference on intelligent computation technology and automation (pp. 330-336).
  35. Yang, L. (2007). An overview of distance metric learning. In Proceedings of the computer vision and pattern recognition conference.

Cited by

Sorted by citations

Related

Authors
Millidge, B., Salvatori, T., Song, Y., Lukasiewicz, T., & Bogacz, R.
Affiliations
  • University of Oxford (GB)
  • TU Wien (AT)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Volume
162
Pages
15561-15583
DOI
10.48550/arxiv.2202.04557
arXiv
2202.04557
PMC
PMC7614148
OpenAlex
W4319460387
Cited by
19
References
10
Citations by year
2026: 1 · 2025: 5 · 2024: 7 · 2023: 6
FWCI
1.48
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • AXA Research Fund
  • China Scholarship Council (EP/N510129/1)
  • Medical Research Council (EP/N510129/1)
  • Engineering and Physical Sciences Research Council (N510129, EP/N510129/, EP/N510129/1, EP/R013667/1)
  • Biotechnology and Biological Sciences Research Council (BB/S006338/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
Keywords
Hopfield neural network, associative memory, similarity measures, Lyapunov function, energy function, memory capacity, Euclidean distance similarity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review