CoChat
Sign Up
Cited by 838Advanced Neural Network ApplicationsParallel Computing and Optimization TechniquesDomain Adaptation and Few-Shot Learning

Benchmark analysis of representative deep neural network architectures

Open access
Bianco, Simone, Cadene, Remi, Celona, Luigi, Napoletano, Paolo·2018·IEEE Access
DOI 10.1109/access.2018.2877890
Open this paper in CoChat — free account

Abstract

This paper presents an in-depth analysis of the majority of the deep neural networks (DNNs) proposed in the state of the art for image recognition. For each DNN, multiple performance indices are observed, such as recognition accuracy, model complexity, computational complexity, memory usage, and inference time. The behavior of such performance indices and some combinations of them are analyzed and discussed. To measure the indices, we experiment the use of DNNs on two different computer architectures, a workstation equipped with a NVIDIA Titan X Pascal, and an embedded system based on a NVIDIA Jetson TX1 board. This experimentation allows a direct comparison between DNNs running on machines with very different computational capacities. This paper is useful for researchers to have a complete view of what solutions have been explored so far and in which research directions are worth exploring in the future, and for practitioners to select the DNN architecture(s) that better fit the resource constraints of practical deployments and applications. To complete this work, all the DNNs, as well as the software used for the analysis, are available online.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2018.2877890

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The size of neural networks is growing exponentially.

    “Many studies report that the size of neural networks is growing exponentially [Xu et al., 2018, Bianco et al., 2018].”

    From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Bianco et al., 2018

    The Introduction (p. 1) says researchers have developed both more accurate and more computationally efficient networks. The Model Complexity section (p. 2) measures parameter-file sizes across architectures, but neither passage establishes that network size grows exponentially over time. Searches for exponential growth and changes in network size over time found no supporting passage in our copy. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 15

  1. [4][4] A. Canziani, A. Paszke, and E. Culurciello, 'An analysis of deep neural network models for practical applications,' arXiv preprint arXiv:1605.07678, 2016. - · arXiv
  2. [5][5] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama et al., 'Speed/accuracy trade-offs for modern convolutional object detectors,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 73107311. -
  3. [8][8] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' arXiv preprint arXiv:1409.1556, 2014. - · arXiv
  4. [10][10] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, 'Going deeper with convolutions,' in Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015. -
  5. [11][11] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, 'Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size,' arXiv preprint arXiv:1602.07360, 2016. - · arXiv
  6. [13][13] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, 'Rethinking the inception architecture for computer vision,' in Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016, pp. 2818- 85 IEEEAccess <!-- image --> FIGURE 3: Top-1 accuracy vs. number of images processed per second (with batch size 1) using the Titan Xp (a) and Jetson TX1 (b). <!-- image --> TABLE 2: Memory consumption of the different DNN models considered on the Titan Xp for different batch sizes. FIGURE 4: Plot of the initial static allocation of the model parameters (i.e. the model complexity) and the total memory utilization with batch size 1 on the Titan Xp. <!-- image --> 2826. -
  7. [15][15] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten, 'Densely connected convolutional networks,' in Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, no. 2. IEEE, 2017, p. 3. -
  8. [16][16] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, 'Aggregated residual transformations for deep neural networks,' in Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017, pp. 5987-5995. -
  9. [18][18] Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng, 'Dual path networks,' in Advances in Neural Information Processing Systems (NIPS), 2017, pp. 4467-4475. -
  10. [19][19] J. Hu, L. Shen, and G. Sun, 'Squeeze-and-excitation networks,' in Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, <!-- image --> IEEEAccesS TABLE 3: Top 5 models (sorted in decreasing Top-1 accuracy) satisfying memory consumption ( ≤ 0.7GB, ≤ 1.0GB, ≤ 1.4GB) and inference speed ( ≥ 15FPS, ≥ 30FPS, ≥ 60FPS) constraints on the Titan Xp (a) and Jetson TX1 (b). 2018. -
  11. [20][20] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, 'Learning transferable architectures for scalable image recognition,' in Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018. -
  12. [21][21] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, 'Mobilenets: Efficient convolutional neural networks for mobile vision applications,' arXiv preprint arXiv:1704.04861, 2017. - · arXiv
  13. [22][22] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, 'Mobilenetv2: Inverted residuals and linear bottlenecks,' in Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018, pp. 4510-4520. -
  14. [23][23] X. Zhang, X. Zhou, M. Lin, and J. Sun, 'Shufflenet: An extremely efficient convolutional neural network for mobile devices,' in Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018. -
  15. [24][24] S. Han, H. Mao, and W. J. Dally, 'Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,' arXiv preprint arXiv:1510.00149, 2015. <!-- image --> SIMONE BIANCO is assistant professor of Computer Science at Department of Informatics, Systems and Communication of the University of Milano-Bicocca, Italy. He obtained the PhD in Computer Science from the University of MilanoBicocca, in 2010. He obtained the BSc and the MSc degree in Mathematics from the University of Milano-Bicocca, respectively in 2003 and 2006. His current research interests include computer vision, machine learning, optimization algorithms, LUIGI CELONA is currently a postdoctoral fellow at DISCo (Department of Informatics, Systems and Communication) of the University of Milano-Bicocca, Italy. In 2018 and 2014, he obtained respectively the PhD and the MSc degree in Computer Science at DISCo. In 2011, he obtained the BSc degree in Computer Science from the University of Messina. His current research interests focus on image analysis and classification, machine learning and face analysis. <!-- image --> and color imaging. <!-- image --> <!-- image --> PAOLO NAPOLETANO is assistant professor of Computer Science at Department of Informatics, Systems and Communication of the University of Milano-Bicocca. In 2007, he received a PhD in Information Engineering from the University of Salerno. In 2003, he received a Master's degree in Telecommunications Engineering from the University of Naples Federico II. His current research interests include machine learning for multi-modal data classification and understanding. REMI CADENE is a PhD student and teaching assistant at LIP6 (Computer Science laboratory) of Sorbonne University, France. In 2016, he received a MSc degree in Computer Science at Sorbonne University. His primary research interests are in the fields of Machine Learning, Computer Vision and Natural Language Processing. He is currently focusing on Neural Networks, Multimodal Learning and Weakly Supervised Learning. · arXiv

Cited by

Sorted by citations

Authors
Bianco, S., Cadene, R., Celona, L., & Napoletano, P.
Affiliations
University of Milano-Bicocca (IT)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2018
Volume
6
Pages
64270-64277
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2018.2877890
OpenAlex
W2893813411
Cited by
838
References
7
Citations by year
2026: 35 · 2025: 60 · 2024: 89 · 2023: 125 · 2022: 175 · 2021: 161 · 2020: 139 · 2019: 53 · 2018: 1
FWCI
44.26
Type
journal article
Language
en
Open access
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/OAPA.html (vor)
Locations
  • IEEE Access (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
Agence Nationale de la Recherche (ANR-11-LABX-0065)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
deep learning architectures, image recognition, benchmarking, model complexity, computational complexity, memory usage, inference latency, embedded systems, NVIDIA Jetson TX1, resource-constrained deployment, recognition accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review