Mixed-precision architecture based on computational memory for training deep neural networks
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Deep neural networks (DNN) have revolutionized the field of machine learning by providing unprecedented human-like performance in solving many real-world problems such as image or speech recognition. Training of large DNNs, however, is a computationally intensive task, and this necessitates the development of novel computing architectures targeting this application. A computational memory unit where resistive memory devices are organized in crossbar arrays can be used to store the synaptic weights in their conductance states. The expensive multiply accumulate operations can be performed in place using Kirchhoff's circuit laws in a non-von Neumann manner. However, a key challenge remains the inability to alter the conductance states of the devices in a reliable manner during the weight update process. We propose a mixed-precision architecture that combines a computational memory unit storing the synaptic weights with a digital processing unit and an additional memory unit that stores the accumulated weight updates in high precision. The new architecture delivers classification accuracies comparable to those of floating-point implementations without being constrained by challenges associated with the non-ideal weight update characteristics of emerging resistive memories. The computational memory unit in a two layer neural network realized using nonlinear stochastic models of phase-change memory achieves a test accuracy of 97.40% in the MNIST digit classification problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Resistance switching memories are memristors
Chua, L. (2011) · Applied Physics A · doi:10.1007/s00339-011-6264-9 · Cited by 1,434 · cited 1×
“The devices store information in their conductance states [6], [7], which can be used to represent the synaptic weights.”
BackgroundDeep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 1×
“In parallel, there are some key developments taking place at the algorithmic front with respect to training DNNs using digital arithmetic with reduced precision [9]–[12].”
BackgroundBurr, G. W., Shelby, R. M., Sidler, S., di Nolfo, C., Jang, J., Boybat, I., Shenoy, R. S., Narayanan, P., Virwani, K., Giacometti, E. U., et al. (2015) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2015.2439635 · Cited by 936
Self-citationNeuromorphic computing with multi-memristive synapses
Boybat, I., Le Gallo, M., Nandakumar, S. R., Moraitis, T., Parnell, T., Tuma, T., Rajendran, B., Leblebici, Y., Sebastian, A., & Eleftheriou, E. (2018) · Nature Communications · doi:10.1038/s41467-018-04933-y · Cited by 899 · cited 1×
“Note that, with improved PCM devices and synaptic architectures these numbers are likely to increase [28], [29].”
BackgroundSelf-citationSparse coding with memristor networks
Sheridan, P. M., Cai, F., Du, C., Ma, W., Zhang, Z., & Lu, W. D. (2017) · Nature Nanotechnology · doi:10.1038/nnano.2017.83 · Cited by 619
Gokmen, T., & Vlasov, Y. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00333 · Cited by 491
Mixed-precision in-memory computing
Le Gallo, M., Sebastian, A., Mathis, R., Manica, M., Giefers, H., Tuma, T., Bekas, C., Curioni, A., & Eleftheriou, E. (2018) · Nature Electronics · doi:10.1038/s41928-018-0054-8 · Cited by 485 · cited 2×
“Building on this insight and on our recent work on mixed-precision memcomputing [13], we present a mixedprecision architecture based on computational memory to train DNNs.”
BackgroundMethodSelf-citationRecent Progress in Phase-ChangeMemory Technology
Burr, G. W., BrightSky, M. J., Sebastian, A., Cheng, H.-Y., Wu, J.-Y., Kim, S., Sosa, N. E., Papandreou, N., Lung, H.-L., Pozidis, H., et al. (2016) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2016.2547718 · Cited by 393 · cited 1×
“Phase-change memory (PCM) is a relatively mature resistive memory technology that has found applications in the space of storage-class memory [17] and novel computing paradigms such as neuromorphic computing [18]–[20] and computational memory [13], [21]–[23].”
BackgroundSelf-citationSuri, M., Bichler, O., Querlioz, D., Cueto, O., Perniola, L., Sousa, V., Vuillaume, D., Gamrat, C., & DeSalvo, B. (2011) · 2011 International Electron Devices Meeting · doi:10.1109/iedm.2011.6131488 · Cited by 314 · cited 1×
“Two PCM devices in differential configuration with weight refresh [2], [27] are used for the network weight simulation.”
MethodTemporal correlation detection using computational phase-change memory
Sebastian, A., Tuma, T., Papandreou, N., Le Gallo, M., Kull, L., Parnell, T., & Eleftheriou, E. (2017) · Nature Communications · doi:10.1038/s41467-017-01481-9 · Cited by 241
Self-citationCompressed sensing recovery using computational memory
Le Gallo, M., Sebastian, A., Cherubini, G., Giefers, H., & Eleftheriou, E. (2017) · 2017 IEEE International Electron Devices Meeting (IEDM) · doi:10.1109/iedm.2017.8268469 · Cited by 28 · cited 1×
“Phase-change memory (PCM) is a relatively mature resistive memory technology that has found applications in the space of storage-class memory [17] and novel computing paradigms such as neuromorphic computing [18]–[20] and computational memory [13], [21]–[23].”
BackgroundSelf-citationBoybat, I., Le Gallo, M., Moraitis, T., Leblebici, Y., Sebastian, A., & Eleftheriou, E. (2017) · 2017 13th Conference on Ph.D. Research in Microelectronics and Electronics (PRIME) · doi:10.1109/prime.2017.7974095 · Cited by 20
Self-citationSupervised learning in spiking neural networks with MLC PCM synapses
Nandakumar, S. R., Boybat, I., Le Gallo, M., Sebastian, A., Rajendran, B., & Eleftheriou, E. (2017) · 2017 75th Annual Device Research Conference (DRC) · doi:10.1109/drc.2017.7999481 · Cited by 13 · cited 1×
“The model is created based on characterization data from approximately 10,000 devices integrated in 90nm CMOS technology [26].”
MethodSelf-citation
Cited by
Memory devices and applications for in-memory computing
Sebastian, A., Le Gallo, M., Khaddam-Aljameh, R., & Eleftheriou, E. (2020) · Nature Nanotechnology · doi:10.1038/s41565-020-0655-z · Cited by 2,289
Self-citationAccurate deep neural network inference using computational phase-change memory
Joshi, V., Le Gallo, M., Haefeli, S., Boybat, I., Nandakumar, S. R., Piveteau, C., Dazzi, M., Rajendran, B., Sebastian, A., & Eleftheriou, E. (2020) · Nature Communications · doi:10.1038/s41467-020-16108-9 · Cited by 490
Self-citationAccurate deep neural network inference using computational phase-change memory
Joshi, V., Gallo, M. L., Haefeli, S., Boybat, I., Nandakumar, S. R., Piveteau, C., Dazzi, M., Rajendran, B., Sebastian, A., & Eleftheriou, E. S. (2020) · RePEc: Research Papers in Economics · doi:10.3929/ethz-b-000416797 · Cited by 379
Self-citationReinforcement learning with analogue memristor arrays
Wang, Z., Li, C., Song, W., Rao, M., Belkin, D., Li, Y., Yan, P., Jiang, H., Lin, P., Hu, M., et al. (2019) · Nature Electronics · doi:10.1038/s41928-019-0221-6 · Cited by 370
Mixed-Precision Deep Learning Based on Computational Memory
Nandakumar, S. R., Le Gallo, M., Piveteau, C., Joshi, V., Mariani, G., Boybat, I., Karunaratne, G., Khaddam-Aljameh, R., Egger, U., Petropoulos, A., et al. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00406 · Cited by 120
Self-citation
Metadata
- Authors
- Nandakumar, S. R., Le Gallo, M., Boybat, I., Rajendran, B., Sebastian, A., & Eleftheriou, E.
- Venue
- 2018 IEEE International Symposium on Circuits and Systems (ISCAS)
- Publisher
- IEEE
- Published
- 2018
- Pages
- 1-5
- ISSN
- 0271-4302 (ISSN-L)
- OpenAlex
- W2801035805
- Cited by
- 67
- References
- 15
- FWCI
- 18.90
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.