The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training
Abstract
Learning in multilayer neural networks (MNNs) relies on continuous updating of large matrices of synaptic weights by local rules. Such locality can be exploited for massive parallelism when implementing MNNs in hardware. However, these update rules require a multiply and accumulate operation for each synaptic weight, which is challenging to implement compactly using CMOS. In this paper, a method for performing these update operations simultaneously (incremental outer products) using memristor-based arrays is proposed. The method is based on the fact that, approximately, given a voltage pulse, the conductivity of a memristor will increment proportionally to the pulse duration multiplied by the pulse magnitude if the increment is sufficiently small. The proposed method uses a synaptic circuit composed of a small number of components per synapse: one memristor and two CMOS transistors. This circuit is expected to consume between 2% and 8% of the area and static power of previous CMOS-only hardware alternatives. Such a circuit can compactly implement hardware MNNs trainable by scalable algorithms based on online gradient descent (e.g., backpropagation). The utility and robustness of the proposed memristor-based circuit are demonstrated on standard supervised learning tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,410
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
Strukov, D. B., Snider, G. S., Stewart, D. R., & Williams, R. S. (2008) · Nature · doi:10.1038/nature06932 · Cited by 11,828
Memristor—The Missing Circuit Element
Chua, L. (1971) · IEEE Transactions on Circuit Theory · doi:10.1109/tct.1971.1083337 · Cited by 10,268 · cited 3×
“The memristor was originally proposed [11], [12] as the missing fourth fundamental passive circuit element.”
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
(1991) · Springer eBooks · doi:10.1007/978-3-7091-2840-4 · Cited by 5,095
WIDROW, B., & HOFF, M. E. (1960) · doi:10.21236/ad0241531 · Cited by 4,009
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199
Best practices for convolutional neural networks applied to visual document analysis
Simard, P. Y., Steinkraus, D., & Platt, J. C. (2005) · Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings. · doi:10.1109/icdar.2003.1227801 · Cited by 2,741
Memristive devices and systems
Chua, L. O., & Kang, S. M. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10092 · Cited by 2,604 · cited 2×
“In [Appendix A, 39], it is explained how to modify the circuit to work with more realistic memristive devices [40, 14], instead of the classical memristor model [11], given a few conditions.”
Simplified neuron model as a principal component analyzer
Oja, E. (1982) · Journal of Mathematical Biology · doi:10.1007/bf00275687 · Cited by 2,309 · cited 1×
“, backpropagation and the perceptron learning rule [32]) and other ML algorithms [45]–[47], enables a massively parallel hardware design, as explained in Section III.”
Neural networks for pattern recognition
Marshall, J. A. (1995) · Neural Networks · doi:10.1016/0893-6080(95)90002-0 · Cited by 1,889
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 1×
“2383395 massive computational power is available [1]–[3] (see [4] for related press).”
BackgroundMorgan, D. (1986) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1986.1164869 · Cited by 1,681
Pegasos: primal estimated sub-gradient solver for SVM
Shalev-Shwartz, S., Singer, Y., Srebro, N., & Cotter, A. (2011) · Mathematical Programming · doi:10.1007/s10107-010-0420-4 · Cited by 1,477
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
LeCun, Y. A., Bottou, L., Orr, G. B., & Müller, K.-R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_3 · Cited by 1,264
The Tradeoffs of Large-Scale Learning
Bottou, L., & Bousquet, O. (2011) · Optimization for Machine Learning · doi:10.7551/mitpress/8996.003.0015 · Cited by 1,128
Deep, Big, Simple Neural Nets for Handwritten Digit Recognition
Cireşan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2010) · Neural Computation · doi:10.1162/neco_a_00052 · Cited by 1,078
TEAM: ThrEshold Adaptive Memristor Model
Kvatinsky, S., Friedman, E. G., Kolodny, A., & Weiser, U. C. (2013) · IEEE Transactions on Circuits and Systems I: Regular Papers · doi:10.1109/tcsi.2012.2215714 · Cited by 843
Self-citationBuilding high-level features using large scale unsupervised learning
Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 823
Switching dynamics in titanium dioxide memristive devices
Pickett, M. D., Strukov, D. B., Borghetti, J. L., Yang, J. J., Snider, G. S., Stewart, D. R., & Williams, R. S. (2009) · Journal of Applied Physics · doi:10.1063/1.3236506 · Cited by 764
Sub-nanosecond switching of a tantalum oxide memristor
Torrezan, A. C., Strachan, J. P., Medeiros-Ribeiro, G., & Williams, R. S. (2011) · Nanotechnology · doi:10.1088/0957-4484/22/48/485203 · Cited by 701
Memristor-Based Material Implication (IMPLY) Logic: Design Principles and Methodologies
Kvatinsky, S., Satat, G., Wald, N., Friedman, E. G., Kolodny, A., & Weiser, U. C. (2014) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2013.2282132 · Cited by 677
Self-citationArtificial neural networks in hardware: A survey of two decades of progress
Misra, J., & Saha, I. (2010) · Neurocomputing · doi:10.1016/j.neucom.2010.03.021 · Cited by 646
Pattern classification by memristive crossbar circuits using ex situ and in situ training
Alibart, F., Zamanidoost, E., & Strukov, D. B. (2013) · Nature Communications · doi:10.1038/ncomms3072 · Cited by 642
On Spike-Timing-Dependent-Plasticity, Memristive Devices, and Building a Self-Learning Visual Cortex
Zamarreño-Ramos, C., Camuñas-Mesa, L. A., Pérez-Carrasco, J. A., Masquelier, T., Serrano-Gotarredona, T., & Linares-Barranco, B. (2011) · Frontiers in Neuroscience · doi:10.3389/fnins.2011.00026 · Cited by 558
High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm
Alibart, F., Gao, L., Hoskins, B. D., & Strukov, D. B. (2012) · Nanotechnology · doi:10.1088/0957-4484/23/7/075201 · Cited by 557 · cited 1×
“The proposed memristor-based design should resolve the main obstacles of the above CMOS-based designs, and provide a compact nonvolatile [71] circuit.”
STDP and STDP variations with memristors for spiking neuromorphic learning systems
Serrano-Gotarredona, T., Masquelier, T., Prodromakis, T., Indiveri, G., & Linares-Barranco, B. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00002 · Cited by 533
Analog Versus Digital: Extrapolating from Electronics to Neurobiology
Sarpeshkar, R. (1998) · Neural Computation · doi:10.1162/089976698300017052 · Cited by 496 · cited 1×
“Usually, analog computation suffers from reduced robustness to noise as compared with digital computation [50].”
Synaptic behaviors and modeling of a metal oxide memristive device
Chang, T., Jo, S.-H., Kim, K.-H., Sheridan, P., Gaba, S., & Lu, W. (2011) · Applied Physics A · doi:10.1007/s00339-011-6296-1 · Cited by 432 · cited 2×
“For example, the values of the memristive timescales range from picoseconds [74] to milliseconds [55].”
Immunity to Device Variations in a Spiking Neural Network With Memristive Nanodevices
Querlioz, D., Bichler, O., Dollfus, P., & Gamrat, C. (2013) · IEEE Transactions on Nanotechnology · doi:10.1109/tnano.2013.2250995 · Cited by 420
Memristor Bridge Synapse-Based Neural Network and Its Learning
Adhikari, S. P., Yang, C., Kim, H., & Chua, L. O. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2204770 · Cited by 386
Zhang, G., & Shen, Y. (2014) · Neural Networks · doi:10.1016/j.neunet.2014.03.009 · Cited by 235
State Dynamics and Modeling of Tantalum Oxide Memristors
Strachan, J. P., Torrezan, A. C., Miao, F., Pickett, M. D., Yang, J. J., Yi, W., Medeiros-Ribeiro, G., & Williams, R. S. (2013) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2013.2264476 · Cited by 211
What Can a Neuron Learn with Spike-Timing-Dependent Plasticity?
Legenstein, R., Naeger, C., & Maass, W. (2005) · Neural Computation · doi:10.1162/0899766054796888 · Cited by 207
Adaptive Synchronization of Memristor-Based Neural Networks with Time-Varying Delays
Wang, L., Shen, Y., Yin, Q., & Zhang, G. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2361776 · Cited by 200
Simulation of a memristor-based spiking neural network immune to device variations
Querlioz, D., Bichler, O., & Gamrat, C. (2011) · The 2011 International Joint Conference on Neural Networks · doi:10.1109/ijcnn.2011.6033439 · Cited by 198
Soudry, D., Hubara, I., & Meir, R. (2014) · Cited by 176
Self-citationA Boundary Condition-Based Approach to the Modeling of Memristor Nanostructures
Corinto, F., & Ascoli, A. (2012) · IEEE Transactions on Circuits and Systems I: Regular Papers · doi:10.1109/tcsi.2012.2190563 · Cited by 163
An all-analog expandable neural network LSI with on-chip backpropagation learning
Morie, T., & Amemiya, Y. (1994) · IEEE Journal of Solid-State Circuits · doi:10.1109/4.309904 · Cited by 115
Passivity and Passification of Memristor-Based Recurrent Neural Networks With Time-Varying Delays
Guo, Z., Wang, J., & Yan, Z. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2305440 · Cited by 113
Massimiliano Versace, A., & Chandler, B. (2010) · IEEE Spectrum · doi:10.1109/mspec.2010.5644776 · Cited by 91
Enabling back propagation training of memristor crossbar neuromorphic processors
Hasan, R., & Taha, T. M. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889893 · Cited by 89
Robust neural logic block (NLB) based on memristor crossbar array
Chabi, D., Zhao, W. S., Querlioz, D., & Klein, J. (2011) · IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH) · doi:10.1109/nanoarch.2011.5941495 · Cited by 82
Neuro chips with on-chip back-propagation and/or Hebbian learning
Shima, T., Kimura, T., Kamatani, Y., Itakura, T., Fujita, Y., & Iida, T. (1992) · IEEE Journal of Solid-State Circuits · doi:10.1109/4.173117 · Cited by 76
A digital neuromorphic VLSI architecture with memristor crossbar synaptic array for machine learning
Kim, Y., Zhang, Y., & Li, P. (2012) · IEEE International SOC Conference (SOCC) · doi:10.1109/socc.2012.6398336 · Cited by 69
Memristor-Based Neural Logic Blocks for Nonlinearly Separable Functions
Soltiz, M., Kudithipudi, D., Merkel, C. E., Rose, G. S., & Pino, R. E. (2013) · IEEE Transactions on Computers · doi:10.1109/tc.2013.75 · Cited by 51
Analog implementation of ann with inherent quadratic nonlinearity of the synapses
Milev, M., & Hristov, M. (2003) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2003.816369 · Cited by 37
Survey of neural network hardware
Lindsey, C. S., & Lindblad, T. (1995) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.205116 · Cited by 37
A Neuromorphic Architecture for Object Recognition and Motion Anticipation Using Burst-STDP
Nere, A., Olcese, U., Balduzzi, D., & Tononi, G. (2012) · PLoS ONE · doi:10.1371/journal.pone.0036958 · Cited by 31
Analysis and verification of an analog VLSI incremental outer-product learning system
Cauwenberghs, G., Neugebauer, C. F., & Yariv, A. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.129421 · Cited by 30
Memristor-Based Multithreading
Kvatinsky, S., Nacson, Y. H., Etsion, Y., Friedman, E. G., Kolodny, A., & Weiser, U. C. (2013) · IEEE Computer Architecture Letters · doi:10.1109/l-ca.2013.3 · Cited by 28
Self-citationManem, H., Rajendran, J., & Rose, G. S. (2012) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2012.2190665 · Cited by 28
Neuroelectronics: Smart connections
Waldrop, M. M. (2013) · Nature · doi:10.1038/503022a · Cited by 28
Neural network LSI chip with on-chip learning
Eguchi, H., Furuta, T., Horiguchi, H., Oteki, S., & Kitaguchi, T. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.155220 · Cited by 26
Memristor-based synaptic networks and logical operations using in-situ computing
Kavehei, O., Al-Sarawi, S. F., Cho, K.-R., Iannella, N. L., Kim, S., Eshraghian, K., & Abbott, D. (2011) · International Conference on Intelligent Sensors, Sensor Networks and Information Processing · doi:10.1109/issnip.2011.6146610 · Cited by 17
An On-Chip BP Learning Neural Network with Ideal Neuron Characteristics and Learning Rate Adaptation
Lu, C., Shi, B., & Chen, L. (2002) · Analog Integrated Circuits and Signal Processing · doi:10.1023/a:1014476806076 · Cited by 16
Letter to the Editor: NEUROCOMPUTERS: A DEAD END?
Omondi, A. R. (2000) · International Journal of Neural Systems · doi:10.1142/s0129065700000375 · Cited by 14
Vasilkoski, Z., Ames, H., Chandler, B., Anatoli Gorchetchnikov, A., Léveillé, J., Livitz, G., Mingolla, E., & Massimiliano Versace, A. (2011) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2011.6033553 · Cited by 14
A Learning Analog Neural Network Chip with Continuous-Time Recurrent Dynamics
Cauwenberghs, G. (1993) · CaltechAUTHORS (California Institute of Technology) · Cited by 14
Hebbian plasticity in mos synapses
CARD, H. C., Schneider, C., & Moore, W. R. (1991) · IEE Proceedings F Radar and Signal Processing · doi:10.1049/ip-f-2.1991.0003 · Cited by 12
Spike timing dependent plasticity with memristive synapse in neuromorphic systems
Chan, W., & Lohn, J. D. (2012) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252822 · Cited by 9
Learning capacitive weights in analog CMOS neural networks
CARD, H. C., Schneider, C., & Schneider, R. S. (1994) · The Journal of VLSI Signal Processing Systems for Signal Image and Video Technology · doi:10.1007/bf02106447 · Cited by 8
Cited by
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106
Self-citationPhysics for neuromorphic computing
Marković, D., Mizrahi, A., Querlioz, D., & Grollier, J. (2020) · Nature Reviews Physics · doi:10.1038/s42254-020-0208-2 · Cited by 1,176
Efficient and self-adaptive in-situ learning in multilayer memristor neural networks
Li, C., Belkin, D., Li, Y., Yan, P., Hu, M., Ge, N., Jiang, H., Montgomery, E., Lin, P., Wang, Z., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-04484-2 · Cited by 931
Gokmen, T., & Vlasov, Y. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00333 · Cited by 491
A Survey of Neuromorphic Computing and Neural Networks in Hardware
Schuman, C. D., Potok, T. E., Patton, R. M., Birdwell, J. D., Dean, M. E., Rose, G. S., & Plank, J. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06963 · Cited by 352
Metadata
- Authors
- Soudry, D., Di Castro, D., Gal, A., Kolodny, A., & Kvatinsky, S.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2015-10
- Volume
- 26
- Issue
- 10
- Pages
- 2408-2421
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W1964288235
- Cited by
- 271
- References
- 70
- FWCI
- 14.43
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.