The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
RAPA-ConvNets: Modified Convolutional Networks for Accelerated Training on Architectures With Analog Arrays
Open accessAbstract
Analog arrays are a promising emerging hardware technology with the potential to drastically speed up deep learning. Their main advantage is that they employ analog circuitry to compute matrix-vector products in constant time, irrespective of the size of the matrix. However, ConvNets map very unfavorably onto analog arrays when done in a straight-forward manner, because kernel matrices are typically small and the constant time operation needs to be sequentially iterated a large number of times. Here, we propose to parallelize the training by replicating the kernel matrix of a convolution layer on distinct analog arrays, and randomly divide parts of the compute among them. With this modification, analog arrays execute ConvNets with a large acceleration factor that is proportional to the number of kernel matrices used per layer (here tested 16-1024). Despite having more free parameters, we show analytically and in numerical experiments that this new convolution architecture is self-regularizing and implicitly learns similar filters across arrays. We also report superior performance on a number of datasets and increased robustness to adversarial attacks. Our investigation suggests to revise the notion that emerging hardware architectures that feature analog arrays for fast matrix-vector multiplication are not suitable for ConvNets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Shallower but wider layers should generally be preferable for AIMC because higher utilization of large matrices stored on crossbar arrays does not significantly change runtime and helps improve energy efficiency.
“For instance, because of the sequential nature of layers of a deep network, shallower but wider layers should generally be preferable for AIMC, since higher utilization of large matrices stored on the crossbar arrays does not significantly change the runtime52,63 and helps improve energy efficiency.”
From Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators (2023) · cites this paper as [52]
The Introduction and §1.2 state that analog-array matrix-vector operations take time independent of the size of the stored matrix, supporting the runtime rationale. The paper does not establish that shallower, wider networks are generally preferable or that higher matrix utilization improves energy efficiency; its proposed acceleration instead uses parallel replicas of convolutional kernel matrices. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCNNs are already less well-suited for resistive crossbar arrays due to the uneven temporal reuse between layers and spatial under-utilization of the large analog tiles by the small kernel matrices.
“In terms of AIMC workload execution latency and system mapping51, CNNs are already less well-suited for resistive crossbar arrays due to the uneven temporal reuse between layers and spatial under-utilization of the large analog tiles by the small kernel matrices (see Table 1), although some optimization and mapping tricks52 are available.”
From Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators (2023) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Convolutional neural networks at constrained time cost
He, K., & Sun, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299173 · Cited by 1,558
Equivalent-accuracy accelerated neural-network training using analogue memory
Ambrogio, S., Narayanan, P., Tsai, H., Shelby, R. M., Boybat, I., di Nolfo, C., Sidler, S., Giordano, M., Bodini, M., Farinha, N. C. P., et al. (2018) · Nature · doi:10.1038/s41586-018-0180-5 · Cited by 1,151
Accelerating machine learning with Non-Volatile Memory: Exploring device and circuit tradeoffs
Fumarola, A., Narayanan, P., Sanches, L. L., Sidler, S., Jang, J., Moon, K., Shelby, R. M., Hwang, H., & Burr, G. W. (2016) · 2016 IEEE International Conference on Rebooting Computing (ICRC) · doi:10.1109/icrc.2016.7738684 · Cited by 37
Flagged first, then by citations
Research works printed but not linked: 27
- Ben-Nun, 2018, Demystifying parallel and distributed deep learning: an in-depth concurrency analysis, CoRR abs/1802.09941. · arXiv
- Bishop, 1995, Training with noise is equivalent to tikhonov regularization, Neural Comput., 7, 108, 10.1162/neco.1995.7.1.108 · DOI · Paper page
- Burr, 2017, Neuromorphic computing using non-volatile memory, Adv. Phys., 2, 89, 10.1080/23746149.2016.1259585 · DOI
- Chetlur, 2014, cuDNN: efficient primitives for deep learning, CoRR abs/1410.0759. · arXiv
- Cisse, 2017, Parseval networks: improving robustness to adversarial examples, 854
- Figurnov, 2016, Perforatedcnns: acceleration through elimination of redundant convolutions, 947
- Gokmen, 2017, Training deep convolutional neural networks with resistive cross-point devices, Front. Neurosci., 11, 538, 10.3389/fnins.2017.00538 · DOI · Paper page
- Gokmen, 2018, Training lstm networks with resistive cross-point devices, Front. Neurosci., 12, 745, 10.3389/fnins.2018.00745 · DOI
- Gokmen, 2016, Acceleration of deep neural network training with resistive cross-point devices: design considerations, Front. Neurosci., 10, 333, 10.3389/fnins.2016.00333 · DOI · Paper page
- Goodfellow, 2014, Explaining and harnessing adversarial examples, arXiv e-prints arXiv:1412.6572. · arXiv
- Graham, 2014, Fractional max-pooling, CoRR abs/1412.6071. · arXiv
- Gu, 2018, Recent advances in convolutional neural networks, Pattern Recogn., 77, 354, 10.1016/j.patcog.2017.10.013 · DOI · Paper page
- Haensch, 2019, The next generation of deep learning hardware: analog computing, Proc. IEEE, 107, 108, 10.1109/JPROC.2018.2871057 · DOI
- Jia, 2014, Caffe: convolutional architecture for fast feature embedding, arXiv e-prints arXiv:1408.5093 · arXiv
- Krizhevsky, 2009, Learning multiple layers of features from tiny images
- Martí, 2016, Energy-efficient neuromorphic classifiers, Neural Comput., 28, 2011, 10.1162/NECO_a_00882 · DOI
- Netzer, 2011, Reading digits in natural images with unsupervised feature learning
- Ngiam, 2010, Tiled convolutional neural networks, 1279
- Perez, 2017, The effectiveness of data augmentation in image classification using deep learning, CoRR abs/1712.04621. · arXiv
- Rasch, 2019, Training large-scale ANNs on simulated resistive crossbar arrays, CoRR abs/1906.02698. · arXiv
- Rifai, 2011, Adding noise to the input of a model trained with a regularized objective, CoRR abs/1104.3250. · arXiv
- Wager, 2013, Dropout training as adaptive regularization, 351
- Xie, 2017, Mitigating adversarial effects through randomization, CoRR abs/1711.01991. · arXiv
- Yang, 2013, Memristive devices for computing, Nat. Nanotechnol., 8, 13, 10.1038/nnano.2012.240 · DOI · Paper page
- You, 2017, Imagenet training in minutes, CoRR abs/1709.05011 · arXiv
- Yu, 2014, Mixed pooling for convolutional neural networks, 364, 10.1007/978-3-319-11740-9_34 · DOI
- Zhang, 2018, Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning, CoRR abs/1803.01927. · arXiv
Cited by
Sorted by citations
Rasch, M. J., Mackin, C., Le Gallo, M., Chen, A., Fasoli, A., Odermatt, F., Li, N., Nandakumar, S. R., Narayanan, P., Tsai, H., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-40770-4 · Cited by 213
“Our results here indicate that AIMC noise-robustness issues will pose additional challenges when implementing CNNs onto AIMC systems.”
BackgroundSelf-citationJain, S., Tsai, H., Chen, C.-T., Muralidhar, R., Boybat, I., Frank, M. M., Woźniak, S., Stanisavljevic, M., Adusumilli, P., Narayanan, P., et al. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3221390 · Cited by 76
Metadata
- Authors
- Rasch, M. J., Gokmen, T., Rigotti, M., & Haensch, W.
- Affiliations
- IBM (United States) (US)
- Corresponding author
- Rasch, Malte J.
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2019-07-30
- Volume
- 13
- Pages
- 753-753
- ISSN
- 1662-453X (ISSN-L)
- PMC
- PMC6682637
- OpenAlex
- W2956306874
- Cited by
- 14
- References
- 4
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 2 · 2022: 2 · 2021: 1 · 2020: 2 · 2019: 4
- FWCI
- 1.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Neuroscience (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- convolutional neural networks, matrix-vector multiplication, self-regularization, adversarial robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
