CoChat

Study of Quantized Hardware Deep Neural Networks Based on Resistive Switching Devices, Conventional versus Convolutional Approaches

Open access
Romero-Zaliz, Rocío, Pérez, Eduardo, Jiménez-Molinos, Francisco, Wenger, Christian, Roldán, Juan B. · 2021 · Electronics

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.

Abstract

A comprehensive analysis of two types of artificial neural networks (ANN) is performed to assess the influence of quantization on the synaptic weights. Conventional multilayer-perceptron (MLP) and convolutional neural networks (CNN) have been considered by changing their features in the training and inference contexts, such as number of levels in the quantization process, the number of hidden layers on the network topology, the number of neurons per hidden layer, the image databases, the number of convolutional layers, etc. A reference technology based on 1T1R structures with bipolar memristors including HfO2 dielectrics was employed, accounting for different multilevel schemes and the corresponding conductance quantization algorithms. The accuracy of the image recognition processes was studied in depth. This type of studies are essential prior to hardware implementation of neural networks. The obtained results support the use of CNNs for image domains. This is linked to the role played by convolutional layers at extracting image features and reducing the data complexity. In this case, the number of synaptic weights can be reduced in comparison to MLPs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/electronics10030346

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Circuit schematic of the 1T1R cells (G: gate terminal; S: source terminal; and TE: top electrode terminal) and cross-sectional TEM image of the metal-insulator-metal (MIM) stack.
Figure 1 Circuit schematic of the 1T1R cells (G: gate terminal; S: source terminal; and TE: top electrode terminal) and cross-sectional TEM image of the metal-insulator-metal (MIM) stack.CC-BY · From the PDF
Figure 2 Cumulative distribution functions (CDFs) of the read-out currents measured for each set of conductive levels, namely, 2 levels ( a ), 3 levels ( b ), 4 levels ( c ), and 5 levels ( d ). The conductance levels were obtained by means of the Multilevel Incremental Step Pulse with Verify Algorithm (M-ISPVA) algorithm [35]. LRS stands for low resistance state, this means that the memristors have a low internal resistance value, and HRS stands for high resistance state. These current levels, and therefore, the device conductance, are obtained by means of different sets of algorithm parameters. In this respect, the corresponding neural network weight quantization strategies can be linked to the algorithms chosen to electrically operate the devices.
Figure 2 Cumulative distribution functions (CDFs) of the read-out currents measured for each set of conductive levels, namely, 2 levels ( a ), 3 levels ( b ), 4 levels ( c ), and 5 levels ( d ). The conductance levels were obtained by means of the Multilevel Incremental Step Pulse with Verify Algorithm (M-ISPVA) algorithm [35]. LRS stands for low resistance state, this means that the memristors have a low internal resistance value, and HRS stands for high resistance state. These current levels, and therefore, the device conductance, are obtained by means of different sets of algorithm parameters. In this respect, the corresponding neural network weight quantization strategies can be linked to the algorithms chosen to electrically operate the devices.CC-BY · From the PDF
Figure 3 CDFs of the read-out currents measured for eight levels. The conductance levels were obtained by means of the Incremental Gate Voltage and Verify Algorithm (IGVVA) algorithm [36]. LRS stands for low resistance state, this means that the memristors has a low internal resistance value, and HRS stands for high resistance state.
Figure 3 CDFs of the read-out currents measured for eight levels. The conductance levels were obtained by means of the Incremental Gate Voltage and Verify Algorithm (IGVVA) algorithm [36]. LRS stands for low resistance state, this means that the memristors has a low internal resistance value, and HRS stands for high resistance state.CC-BY · From the PDF
Figure 4 CNNarchitecture. The input data are introduced into a convolutional layer (accounting for different convolutional filters to generate feature maps, that accentuate the unique features of the original image) followed by a pooling layer (that reduces the size of the image combining neighboring pixels of a certain image area into a single representative value). Multiple sets of convolutional + pooling layers can be applied as pre-processing. Finally, all outputs of the last pooling layer are flattened (from a matrix to an array) and used as inputs of a MLP.
Figure 4 CNNarchitecture. The input data are introduced into a convolutional layer (accounting for different convolutional filters to generate feature maps, that accentuate the unique features of the original image) followed by a pooling layer (that reduces the size of the image combining neighboring pixels of a certain image area into a single representative value). Multiple sets of convolutional + pooling layers can be applied as pre-processing. Finally, all outputs of the last pooling layer are flattened (from a matrix to an array) and used as inputs of a MLP.CC-BY · From the PDF
Figure 5 Distribution of total number of synapses for each type of artificial neural network (ANN). The boxplots show a rectangle where its bottom line corresponds to the 25th percentile (or first quartile), the top line to the 75th percentile (or third quartile) and the middle line to 50th percentile or median (the middle value). The vertical line below the box extends until the 0th percentile (the lowest data point excluding outliers), while the vertical line above the box extends until 100th percentile (the largest data point excluding outliers). In this figure, outliers were removed for visualization enhancement.
Figure 5 Distribution of total number of synapses for each type of artificial neural network (ANN). The boxplots show a rectangle where its bottom line corresponds to the 25th percentile (or first quartile), the top line to the 75th percentile (or third quartile) and the middle line to 50th percentile or median (the middle value). The vertical line below the box extends until the 0th percentile (the lowest data point excluding outliers), while the vertical line above the box extends until 100th percentile (the largest data point excluding outliers). In this figure, outliers were removed for visualization enhancement.CC-BY · From the PDF
Figure 6 Four examples the 28 × 28 pixel images of the MNIST dataset, labelled as '4', '1', '9' and '2' from left to right.
Figure 6 Four examples the 28 × 28 pixel images of the MNIST dataset, labelled as '4', '1', '9' and '2' from left to right.CC-BY · From the PDF
Figure 7 Five examples the 28 × 28 grayscale pixel images of the Fashion MNIST dataset, labelled as 'Ankle boot', 'T-shirt/top', 'T-shirt/top', 'Dress' and 'Dress' from left to right.
Figure 7 Five examples the 28 × 28 grayscale pixel images of the Fashion MNIST dataset, labelled as 'Ankle boot', 'T-shirt/top', 'T-shirt/top', 'Dress' and 'Dress' from left to right.CC-BY · From the PDF
Figure 8 Categorical accuracy for the MNIST dataset. Experiments using 1 to 4 hidden layers, shown in each of the boxes, were used. For each hidden layer, hidden units ranging from 8 to 128 were tested ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization.
Figure 8 Categorical accuracy for the MNIST dataset. Experiments using 1 to 4 hidden layers, shown in each of the boxes, were used. For each hidden layer, hidden units ranging from 8 to 128 were tested ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization.CC-BY · From the PDF
Figure 9 Categorical accuracy for the Fashion MNIST dataset. Experiments using 1 to 4 hidden layers, shown in each of the boxes, were used. For each hidden layer, hidden units ranging from 8 to 128 were tested ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization.
Figure 9 Categorical accuracy for the Fashion MNIST dataset. Experiments using 1 to 4 hidden layers, shown in each of the boxes, were used. For each hidden layer, hidden units ranging from 8 to 128 were tested ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization.CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Figure 11 Categorical accuracy for the MNIST dataset using 2-CNN (i.e., one convolutional layer, one intermediate pooling layer, another convolutional layer and a final pooling layer). Only one hidden layer was used, with hidden units ranging from 8 to 128 ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization. Each box in the figure shows a particular experiment: convolutional layer with 8, 16 and 32 filters (the top number of the box) and pooling with 2 × 2, 3 × 3 and 4 × 4 matrices (the bottom number of the box).
Figure 11 Categorical accuracy for the MNIST dataset using 2-CNN (i.e., one convolutional layer, one intermediate pooling layer, another convolutional layer and a final pooling layer). Only one hidden layer was used, with hidden units ranging from 8 to 128 ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization. Each box in the figure shows a particular experiment: convolutional layer with 8, 16 and 32 filters (the top number of the box) and pooling with 2 × 2, 3 × 3 and 4 × 4 matrices (the bottom number of the box).CC-BY · From the PDF
Figure 12 Categorical accuracy for the Fashion MNIST dataset using 1-CNN (i.e., one convolutional layer + one pooling layer). Only one hidden layer was used, with hidden units ranging from 8 to 128 ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization. Each box in the figure shows a particular experiment: convolutional layer with 8, 16 and 32 filters (top number of the box) and pooling with 2 × 2, 3 × 3 and 4 × 4 matrices (bottom number of the box).
Figure 12 Categorical accuracy for the Fashion MNIST dataset using 1-CNN (i.e., one convolutional layer + one pooling layer). Only one hidden layer was used, with hidden units ranging from 8 to 128 ( x axis). Each dashed line in the plots shows the accuracy for different quantization levels, while solid lines indicate no quantization. Each box in the figure shows a particular experiment: convolutional layer with 8, 16 and 32 filters (top number of the box) and pooling with 2 × 2, 3 × 3 and 4 × 4 matrices (bottom number of the box).CC-BY · From the PDF
Image 13
CC-BY · From the PDF
Image 14
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 28

Metadata

Authors
Romero-Zaliz, R., Pérez, E., Jiménez-Molinos, F., Wenger, C., & Roldán, J. B.
Affiliations
  • Andalusian Research Institute on Data Science and Computational Intelligence (DaSCI), University of Granada, 18071 Granada, Spain
  • Leibniz Institute for High Performance Microelectronics (DE)
  • Universidad de Granada (ES)
  • Brandenburg University of Technology Cottbus-Senftenberg (DE)
Corresponding author
Roldán, Juan B.
Venue
Electronics
Publisher
MDPI AG
Published
2021-02-01
Volume
10
Issue
3
Pages
346
ISSN
2079-9292 (ISSN-L)
OpenAlex
W3126884064
Cited by
28
References
47
Citations by year
2025: 3 · 2024: 4 · 2023: 5 · 2022: 6 · 2021: 10
FWCI
1.84
Type
journal article
Language
en
Open access
Locations
  • Electronics (publishedVersion, cc-by)
  • Institutional Repository of the University of Granada (University of Granada) (submittedVersion, cc-by)
  • LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Ministerio de Ciencia e Innovación (TEC2017-84321-C4-3-R, RTI2018-098983-B-I00)
  • Consejería de Conocimiento, Investigación y Universidad, Junta de Andalucía (A-TIC-117-UGR18)
  • German Research Foundation (FOR2093)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
neural network hardware, resistive switching devices, memristors, multilevel quantization, convolutional neural networks, multilayer perceptron, HfO2 dielectric, image recognition, energy efficiency, hardware implementation of neural networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.