CoChat
Sign Up
Cited by 71Advanced Memory and Neural ComputingNeural Networks and ApplicationsNeural Networks and Reservoir Computing

Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search

Open access
Mao, Ruibin, Wen, Bo, Kazemi, Arman, Zhao, Yahui, Laguna, Ann Franchesca, Lin, Rui, Wong, Ngai, Niemier, Michael +5 more·2022·Nature Communications
DOI 10.1038/s41467-022-33629-7PMC PMC9587027

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Lifelong on-device learning is a key challenge for machine intelligence, and this requires learning from few, often single, samples. Memory-augmented neural networks have been proposed to achieve the goal, but the memory module must be stored in off-chip memory, heavily limiting the practical use. In this work, we experimentally validated that all different structures in the memory-augmented neural network can be implemented in a fully integrated memristive crossbar platform with an accuracy that closely matches digital hardware. The successful demonstration is supported by implementing new functions in crossbars, including the crossbar-based content-addressable memory and locality sensitive hashing exploiting the intrinsic stochasticity of memristor devices. Simulations show that such an implementation can be efficiently scaled up for one-shot learning on more complex tasks. The successful demonstration paves the way for practical on-device lifelong learning and opens possibilities for novel attention-based algorithms that were not possible in conventional hardware.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-022-33629-7

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In-memory computing is widely studied as an energy-efficient approach for hyperdimensional computing.

    “Therefore, in-memory computing is widely studied as an energy-efficient approach for hyperdimensional computing [135, 140,141,142].”

    From Brain-inspired computing systems: a systematic literature review (2024) · cites this paper as [140]

    The Introduction (page unavailable) describes in-memory hyperdimensional computing as a promising approach and cites several prototypes, supporting the claim that it is being studied. The paper’s energy-efficiency comparisons concern its own memristive memory-augmented neural network, not hyperdimensional computing specifically, so the claimed energy-efficiency rationale is not established for that approach. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. [1]Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D. & Lillicrap, T. Meta-learning with memory-augmented neural networks. In Balcan, M. F. & Weinberger, K. Q. (eds.) Proceedings of The 33rd International Conference on Machine Learning, vol. 48 of Proceedings of Machine Learning Research, 1842-1850 (PMLR, New York, New York, USA, 2016). http://proceedings.mlr.press/v48/santoro16.html.
  2. [2]Stevens, J. R., Ranjan, A., Das, D., Kaul, B. & Raghunathan, A. Manna: An accelerator for memory-augmented neural networks. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, 794-806 (2019).
  3. [4]Von Neumann J First draft of a report on the edvac IEEE Annals Hist. Comput. 1993 15 27 75 10.1109/85.238389 · DOI · Paper page
  4. [5]Strubell, E., Ganesh, A. & McCallum, A. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645-3650 (Association for Computational Linguistics, Florence, Italy, 2019). https://www.aclweb.org/anthology/P19-1355.
  5. [6]Li, D., Chen, X., Becchi, M. & Zong, Z. Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus. In 2016 IEEE international conferences on big data and cloud computing (BDCloud), social computing and networking (SocialCom), sustainable computing and communications (SustainCom)(BDCloud-SocialCom-SustainCom), 477-484 (IEEE, 2016).
  6. [9]Xia Q Yang JJ Memristive crossbar arrays for brain-inspired computing Nat. Mater. 2019 18 309 323 10.1038/s41563-019-0291-x 30894760 · DOI · Paper page
  7. [10]Li C Long short-term memory networks in memristor crossbar arrays Nat. Mach. Intell. 2019 1 49 57 10.1038/s42256-018-0001-4 · DOI · Paper page
  8. [15]Li C Analogue signal and image processing with large memristor crossbars Nat. Electr. 2018 1 52 59 10.1038/s41928-017-0002-z · DOI · Paper page
  9. [16]Sheridan PM Sparse coding with memristor networks Nat. Nanotechnol. 2017 12 784 789 10.1038/nnano.2017.83 28530717 · DOI · Paper page
  10. [18]Zidan MA A general memristor-based partial differential equation solver Nat. Electr. 2018 1 411 420 10.1038/s41928-018-0100-6 · DOI
  11. [19]Yang, K. et al. Transiently chaotic simulated annealing based on intrinsic nonlinearity of memristors for efficient solution of optimization problems. Sci. Adv.6, (2020). PMC7428342 32851168
  12. [20]Cai F Power-efficient combinatorial optimization using intrinsic noise in memristor Hopfield neural networks Nat. Electr. 2020 3 409 418 10.1038/s41928-020-0436-6 · DOI
  13. [21]Hu, M. et al. Dot-product engine for neuromorphic computing: Programming 1t1m crossbar to accelerate matrix-vector multiplication. In 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC), 1-6 (IEEE, 2016).
  14. [22]Karunaratne, G. et al. In-memory hyperdimensional computing. Nat. Electr. 3 (2020). 10.1038/s41928-020-0410-3. 1906.01548. · DOI · Paper page
  15. [23]Karunaratne G Robust high-dimensional memory-augmented neural networks Nat. Commun. 2021 12 1 12 10.1038/s41467-021-22364-0 33927202 PMC8084980 · DOI
  16. [24]Karunaratne, G. et al. In-memory realization of in-situ few-shot continual learning with a dynamically evolving explicit memory. arXiv preprint arXiv:2207.06810 (2022). · arXiv
  17. [25]Kaiser, Ł., Nachum, O., Roy, A. & Bengio, S. Learning to remember rare events. arXiv preprint arXiv:1703.03129 (2017). · arXiv
  18. [26]Laguna, A. F., Kazemi, A., Niemier, M. & Hu, X. S. In-memory computing based accelerator for transformer networks for long sequences. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), 1839-1844 (IEEE, 2021).
  19. [27]Kitaev, N., Kaiser, L. & Levskaya, A. Reformer: The efficient transformer. In International Conference on Learning Representations (2020). https://openreview.net/forum?id=rkgNKkHtvB.
  20. [28]Ni K Ferroelectric ternary content-addressable memory for one-shot learning Nat. Electr. 2019 2 521 529 10.1038/s41928-019-0321-3 · DOI
  21. [29]Laguna, A. F., Yin, X., Reis, D., Niemier, M. & Hu, X. S. Ferroelectric FET based in-memory computing for few-shot learning. Proceedings of the ACM Great Lakes Symposium on VLSI, GLSVLSI373-378 (2019).
  22. [30]Li, H. et al. Sapiens: A 64-kb rram-based non-volatile associative memory for one-shot learning and inference at the edge. IEEE Transactions on Electron Devices (2021).
  23. [31]Lake, B., Salakhutdinov, R., Gross, J. & Tenenbaum, J. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, vol. 33 (2011).
  24. [32]Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K. & Wierstra, D. Matching networks for one shot learning. arXiv preprint arXiv:1606.04080 (2016). · arXiv
  25. [34]Shafiee A Isaac: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars ACM SIGARCH Comput. Archit. News 2016 44 14 26 10.1145/3007787.3001139 · DOI
  26. [35]Datar, M., Immorlica, N., Indyk, P. & Mirrokni, V. S. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, 253-262 (2004).
  27. [36]Sheng X Low-conductance and multilevel cmos-integrated nanoscale oxide memristors Adv. Electr. Mater. 2019 5 1800876 10.1002/aelm.201800876 · DOI
  28. [38]Gionis, A. et al. Similarity search in high dimensions via hashing. In Vldb, vol. 99, 518-529 (1999).
  29. [39]Shinde, R., Goel, A., Gupta, P. & Dutta, D. Similarity search and locality sensitive hashing using ternary content addressable memories. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, 375-386 (2010).
  30. [40]Huang Q Feng J Zhang Y Fang Q Ng W Query-aware locality-sensitive hashing for approximate nearest neighbor search Proceedings of the VLDB Endowment 2015 9 1 12 10.14778/2850469.2850470 · DOI
  31. [41]Kazemi, A. et al. A device non-ideality resilient approach for mapping neural networks to crossbar arrays. In 2020 57th ACM/IEEE Design Automation Conference (DAC), 1-6 (IEEE, 2020).
  32. [42]Niu G Geometric conductive filament confinement by nanotips for resistive switching of hfo 2-rram devices with high performance Sci. Rep. 2016 6 1 9 27181525 PMC4867633
  33. [43]Li J Montoye RK Ishii M Chang L 1 mb 0.41 μm2 2t-2r cell nonvolatile tcam with two-bit encoding and clocked self-referenced sensing IEEE J. Solid-State Circuits 2013 49 896 907 10.1109/JSSC.2013.2292055 · DOI
  34. [44]Lin, C.-C. et al. 7.4 a 256b-wordlength reram-based tcam with 1ns search-time and 14 × improvement in wordlength-energyefficiency-density product using 2.5 t1r cell. In 2016 IEEE International Solid-State Circuits Conference (ISSCC), 136-137 (IEEE, 2016).
  35. [45]Fedorov, V. V., Abusultan, M. & Khatri, S. P. An area-efficient ternary cam design using floating gate transistors. In 2014 IEEE 32nd International Conference on Computer Design (ICCD), 55-60 (IEEE, 2014).
  36. [46]Apalkov D Dieny B Slaughter J Magnetoresistive random access memory Proc. IEEE 2016 104 1796 1830 10.1109/JPROC.2016.2590142 · DOI
  37. [47]Ambrogio S Statistical fluctuations in hfo x resistive-switching memory: part i-set/reset variability IEEE Trans Electron Dev. 2014 61 2912 2919 10.1109/TED.2014.2330200 · DOI
  38. [49]Wang, Y., Chao, W.-L., Weinberger, K. Q. & van der Maaten, L. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.04623 (2019). · arXiv
  39. [50]Mao, R. et al. Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search. TLSH_MANN: v1.0.0 (2022). 10.5281/zenodo.7093390. PMC9587027 36271072 · DOI

Cited by

Sorted by citations

Authors
Mao, R., Wen, B., Kazemi, A., Zhao, Y., Laguna, A. F., Lin, R., Wong, N., Niemier, M., Hu, X. S., Sheng, X., Graves, C. E., Strachan, J. P., & Li, C.
Affiliations
  • University of Hong Kong (HK)
  • University of Notre Dame (US)
  • Hewlett-Packard (United States) (US)
  • Forschungszentrum Jülich (DE)
Corresponding author
  • Graves, Catherine E.
  • Strachan, John Paul
  • Li, Can
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2022-10-21
Volume
13
Issue
1
Pages
6284-6284
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-022-33629-7
PMC
PMC9587027
OpenAlex
W4307044055
Cited by
71
References
9
Citations by year
2026: 8 · 2025: 33 · 2024: 17 · 2023: 12 · 2022: 1
FWCI
4.31
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Semiconductor Research Corporation
  • National Natural Science Foundation of China (62122005)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Reservoir Computing
Keywords
memristor crossbar, memory-augmented neural networks, on-device learning, one-shot learning, content-addressable memory, locality-sensitive hashing, hardware implementation of neural networks, similarity search, attention mechanism
MeSH terms
Algorithms, Artificial Intelligence, Computers, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review