The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices
Open accessAbstract
Recent advances in deep learning have been driven by ever-increasing model sizes, with networks growing to millions or even billions of parameters. Such enormous models call for fast and energy-efficient hardware accelerators. We study the potential of Analog AI accelerators based on Non-Volatile Memory, in particular Phase Change Memory (PCM), for software-equivalent accurate inference of natural language processing applications. We demonstrate a path to software-equivalent accuracy for the GLUE benchmark on BERT (Bidirectional Encoder Representations from Transformers), by combining noise-aware training to combat inherent PCM drift and noise sources, together with reduced-precision digital attention-block computation down to INT6.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Noise-aware training-simple injection of noise onto weights or activations during DNN training-can make DNNs more robust for AIMC deployment.
“A number of previous studies have shown that noise-aware training —simple injection of noise onto weights or activations during DNN training—can make DNNs more robust for AIMC deployment33,38–42.”
From Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators (2023) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 1×
“While current models are generally trained and run on general-purpose digital processors such as CPUs and GPUs, the rapid growth in both size and scope of these networks has fostered novel hardware architectures aiming to optimize speed and energy-efficiency, specifically targeting either neural network training or in…”
BackgroundA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Equivalent-accuracy accelerated neural-network training using analogue memory
Ambrogio, S., Narayanan, P., Tsai, H., Shelby, R. M., Boybat, I., di Nolfo, C., Sidler, S., Giordano, M., Bodini, M., Farinha, N. C. P., et al. (2018) · Nature · doi:10.1038/s41586-018-0180-5 · Cited by 1,151
Self-citationBurr, G. W., Shelby, R. M., Sidler, S., di Nolfo, C., Jang, J., Boybat, I., Shenoy, R. S., Narayanan, P., Virwani, K., Giacometti, E. U., et al. (2015) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2015.2439635 · Cited by 936 · cited 1×
“This architecture can significantly mitigate the Von-Neumann bottleneck caused by communication between the processor and memory, and is particularly efficient for fully-connected neural network layers (Burr et al., 2015).”
BackgroundSelf-citationAccurate deep neural network inference using computational phase-change memory
Joshi, V., Le Gallo, M., Haefeli, S., Boybat, I., Nandakumar, S. R., Piveteau, C., Dazzi, M., Rajendran, B., Sebastian, A., & Eleftheriou, E. (2020) · Nature Communications · doi:10.1038/s41467-020-16108-9 · Cited by 490
A Flexible and Fast PyTorch Toolkit for Simulating Training and Inference on Analog Crossbar Arrays
Rasch, M. J., Moreda, D., Gokmen, T., Le Gallo, M., Carta, F., Goldberg, C., El Maghraoui, K., Sebastian, A., & Narayanan, V. (2021) · 2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas51828.2021.9458494 · Cited by 160
Self-citationAmbrogio, S., Gallot, M., Spoon, K., Tsai, H., Mackin, C., Wesson, M., Kariyappa, S., Narayanan, P., Liu, C.-C., Kumar, A., et al. (2019) · 2019 IEEE International Electron Devices Meeting (IEDM) · doi:10.1109/iedm19573.2019.8993482 · Cited by 76 · cited 2×
“However, NVM devices exhibit many conductance instabilities [conductance drift (Ambrogio et al., 2019), programming and read noise (Tsai et al.”
BackgroundSelf-citationTsai, H., Ambrogio, S., Mackin, C., Narayanan, P., Shelby, R. M., Rocki, K., Chen, A., & Burr, G. W. (2019) · 2019 Symposium on VLSI Technology · doi:10.23919/vlsit.2019.8776519 · Cited by 47
Self-citationThe marriage of training and inference for scaled deep learning analog hardware
Gokmen, T., Rasch, M. J., & Haensch, W. (2019) · 2019 IEEE International Electron Devices Meeting (IEDM) · doi:10.1109/iedm19573.2019.8993573 · Cited by 37
Self-citation
Flagged first, then by citations
Research works printed but not linked: 21
- Agirre, 2007, Proceedings Fourth International Workshop on Semantic Evaluations (SemEval), 10.3115/1621474 · DOI
- Bar-Haim, 2006, The second PASCAL recognising textual entailment challenge,
- Bentivogli, 2009, The fifth PASCAL recognizing textual entailment challenge,
- Burr, 2016, Recent progress in PCM technology, IEEE J. Emerg. Sel. Top. Circ. Sys., 6, 146, 10.1109/JETCAS.2016.2547718 · DOI · Paper page
- Burr, 2017, Neuromorphic computing using non-volatile memory, Adv. Phys. X, 2, 89, 10.1080/23746149.2016.1259585 · DOI
- Chang, 2019, Ai hardware acceleration with analog memory: microarchitectures for low energy at high speed, IBM J. Res. Dev., 8, 1, 10.1147/JRD.2019.2934050 · DOI · Paper page
- Dagan, 2006, The PASCAL recognising textual entailment challenge,, 177, 10.1007/11736790_9 · DOI
- Dolan, 2005, Automatically constructing a corpus of sentential paraphrases,
- Giampiccolo, 2007, The third PASCAL recognizing textual entailment challenge,, Proceedings ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, 1
- Giannopoulos, 2018, 8-bit precision in-memory multiplication with projected phase-change memory,, 27.7.1, 10.1109/IEDM.2018.8614558 · DOI
- Ielmini, 2007, Recovery and drift dynamics of resistance and threshold voltages in phase-change memories, IEEE Trans. Electron Dev., 54, 308, 10.1109/TED.2006.888752 · DOI
- Lan, 2020, ALBERT: a lite BERT for self-supervised learning of language representations
- Matsukura, 2015, Control of magnetism by electric fields, Nat. Nanotechnol., 10, 209, 10.1038/nnano.2015.22 · DOI
- Paszke, 2019, Pytorch: an imperative style, high-performance deep learning library, NIPS, 32, 8026
- Rajpurkar, 2016, SQuAD: 100,000+ questions for machine comprehension of text,, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2383, 10.18653/v1/D16-1264 · DOI · Paper page
- Socher, 2013, Recursive deep models for semantic compositionality over a sentiment treebank,, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1631
- Vaswani, 2017, Attention is all you need,
- Wang, 2019, GLUE:a multi-task benchmark and analysis platform for natural language understanding,
- Warstadt, 2018, Neural network acceptability judgments, arXiv preprint 1805.12471 · arXiv
- Wolf, 2020, Transformers: State-of-the-art natural language processing,, 38
- Wong, 2012, Metal-Oxide RRAM, Proc. IEEE, 100, 1951, 10.1109/JPROC.2012.2190369 · DOI · Paper page
Cited by
Sorted by citations
An analog-AI chip for energy-efficient speech recognition and transcription
Ambrogio, S., Narayanan, P., Okazaki, A., Fasoli, A., Mackin, C., Hosokawa, K., Nomura, A., Yasuda, T., Chen, A., Friz, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06337-5 · Cited by 269
Self-citationRasch, M. J., Mackin, C., Le Gallo, M., Chen, A., Fasoli, A., Odermatt, F., Li, N., Nandakumar, S. R., Narayanan, P., Tsai, H., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-40770-4 · Cited by 213
“A number of previous studies have shown that noise-aware training — simple injection of noise onto weights or activations during DNN training — can make DNNs more robust for AIMC deployment 33,38 – 42 .”
BackgroundSelf-citationNarayanan, P., Ambrogio, S., Okazaki, A., Hosokawa, K., Tsai, H., Nomura, A., Yasuda, T., Mackin, C., Lewis, S. C., Friz, A., et al. (2021) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2021.3115993 · Cited by 85
Self-citationJain, S., Tsai, H., Chen, C.-T., Muralidhar, R., Boybat, I., Frank, M. M., Woźniak, S., Stanisavljevic, M., Adusumilli, P., Narayanan, P., et al. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3221390 · Cited by 76
Self-citation
Metadata
- Authors
- Spoon, K., Tsai, H., Chen, A., Rasch, M. J., Ambrogio, S., Mackin, C., Fasoli, A., Friz, A. M., Narayanan, P., Stanisavljevic, M., & Burr, G. W.
- Affiliations
- IBM Research - Almaden (US)
- IBM (United States) (US)
- Corresponding author
- Spoon, Katie
- ORCID
- Venue
- Frontiers in Computational Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2021-07-05
- Volume
- 15
- Pages
- 675741-675741
- ISSN
- 1662-5188 (ISSN-L)
- PMC
- PMC8287521
- OpenAlex
- W3184497370
- Cited by
- 26
- References
- 12
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 2 · 2023: 11 · 2022: 5 · 2021: 1
- FWCI
- 1.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Computational Neuroscience (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- phase-change memory, non-volatile memory, BERT, GLUE benchmark, noise-aware training, INT6, attention blocks, energy-efficient hardware
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
