Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers
Abstract
Artificial intelligence and machine learning are experiencing widespread adoption in the industry, academia, and even public consciousness. This has been driven by the rapid advances in the applications and accuracy of AI through increasingly complex algorithms and models; this, in turn, has spurred research into developing specialized hardware AI accelerators. The rapid pace of the advances makes it easy to miss the forest for the trees: they are often developed and evaluated in a vacuum without considering the full application environment in which they must eventually operate. In this paper, we deploy and characterize Face Recognition, an AI-centric edge video analytics application built using open source and widely adopted infrastructure and ML tools. We evaluate its holistic, end-to-end behavior in a production-size edge data center and reveal the "AI tax" for all the processing that is involved. Even though the application is built around state-of-the-art AI and ML algorithms, it relies heavily on pre-and post-processing code which must be executed on a general-purpose CPU. As AI-centric applications start to reap the acceleration promised by so many accelerators, we find they impose stresses on the underlying software infrastructure and the data center's capabilities: storage and network bandwidth become major bottlenecks with increasing AI acceleration. By not having to serve a wide variety of applications, we show that a purpose-built edge data center can be designed to accommodate the stresses of accelerated AI at 15% lower TCO than one derived from homogeneous servers and infrastructure. We also discuss how our conclusions generalize beyond Face Recognition as many AI-centric applications at the edge rely upon the same underlying software and hardware infrastructure.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In a system utilizing publish/subscribe messaging services, 33% of application latency is a result of internal communication frameworks.
“In a system utilizing publish/subscribe messaging services, [44] observed that 33% of application latency is a result of internal communication frameworks, and by experimentation, found that the messaging system becomes congested when faced with an eightfold increase in inference performance.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [44]
The messaging system becomes congested when faced with an eightfold increase in inference performance.
“by experimentation, found that the messaging system becomes congested when faced with an eightfold increase in inference performance.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [44]
Data pre- and post-processing code in streaming ML systems could soon become another performance bottleneck.
“[44] warns that the data pre-and post-processing code in streaming ML systems could soon become another performance bottleneck.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [44]
Increasing the number of broker node instances used in the messaging system clears up the congestion issue.
“The authors found that a solution to the problem could be to increase the number of broker node instances used in the messaging system, which clears up the congestion issue.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [44]
Docker is by far the most frequently reported solution for packaging ML models in containers.
“Docker 17 is by far the most frequently reported solution [12,16,20,26,28,40,44].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [44]
Having multiple containers deployed often necessitates a container orchestration tool such as Kubernetes or Apache Mesos.
“Having multiple containers deployed will often necessitate the use of a container orchestration tool such as Kubernetes 20 [29,40,43,44] or Apache Mesos 21 [16].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [44]
Apache Kafka is a commonly reported messaging and communications framework, used for logging predictions, notifying model hosts of updates, and general inter-container communication.
“A commonly reported messaging and communications framework is Apache Kafka 28 , used for logging predictions [28], notifying model hosts of updates [16,29] and general inter-container communication [44].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [44]
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · see citing works
Fat-trees: Universal networks for hardware-efficient supercomputing
Leiserson, C. E. (1985) · IEEE Transactions on Computers · doi:10.1109/tc.1985.6312192 · Cited by 1,330 · see citing works
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2541940.2541967 · Cited by 1,314 · see citing works
Reddi, V. J., Cheng, C., Kanter, D. E., Mattson, P. R., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M. B., Charlebois, M., Chou, W., et al. (2020) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca45697.2020.00045 · Cited by 634 · see citing works
Exploiting virtual synchrony in distributed systems
Birman, K., & Joseph, T. (1987) · ACM SIGOPS Operating Systems Review · doi:10.1145/37499.37515 · Cited by 602 · see citing works
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590 · see citing works
Profiling a warehouse-scale computer
Kanev, S., Darago, J. P., Hazelwood, K., Ranganathan, P., Moseley, T., Wei, G.-Y., & Brooks, D. M. (2015) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2749469.2750392 · Cited by 409 · see citing works
Serving DNNs in Real Time at Datacenter Scale with Project Brainwave
Chung, E. S., Fowers, J., Ovtcharov, K., Papamichael, M. K., Caulfield, A. M., Massengill, T., Liu, M., Lo, D., Alkalay, S., Haselman, M., et al. (2018) · IEEE Micro · doi:10.1109/mm.2018.022071131 · Cited by 329 · see citing works
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · ACM SIGPLAN Notices · doi:10.1145/2644865.2541967 · Cited by 281 · see citing works
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation
Gupta, U., Wu, C.-J., Wang, X., Naumov, M., Reagen, B., Brooks, D. M., Cottel, B., Hazelwood, K., Hempstead, M., Jia, B., et al. (2020) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00047 · Cited by 279 · see citing works
Li, J., Sharma, N. K., Ports, D. R. K., & Gribble, S. D. (2014) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2670979.2670988 · Cited by 235 · see citing works
The mystery machine: end-to-end performance analysis of large-scale internet services
Chow, M., Meisner, D., Flinn, J., Peek, D., & Wenisch, T. F. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685066 · Cited by 182 · see citing works
TABLA: A unified template-based framework for accelerating statistical machine learning
Mahajan, D., Park, J., Amaro, E., Sharma, H., Yazdanbakhsh, A., Kim, J. K., & Esmaeilzadeh, H. (2016) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2016.7446050 · Cited by 156 · see citing works
Exploiting Virtual Synchrony in Distributed Systems
Birman, K., & Joseph, T. A. (1987) · doi:10.21236/ada177091 · Cited by 97 · see citing works
Heterogeneity in “Homogeneous” Warehouse-Scale Computers: A Performance Opportunity
Mars, J., Tang, L., & Hundt, R. (2011) · IEEE Computer Architecture Letters · doi:10.1109/l-ca.2011.14 · Cited by 85 · see citing works
Video Analytics for Surveillance: Theory and Practice [From the Guest Editors
Regazzoni, C. S., Cavallaro, A., Wu, Y., et al. (2010) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.937451 · Cited by 77 · see citing works
Exploiting heterogeneity for tail latency and energy efficiency
Haque, M. E., He, Y., Elnikety, S., Nguyen, T. D., Bianchini, R., & McKinley, K. S. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3123956 · Cited by 55 · see citing works
Long, D. R. (1986) · Foreign Language Annals · doi:10.1111/j.1944-9720.1986.tb02838.x · Cited by 46 · see citing works
MLPerf Inference BenchmarkPreprint
Reddi, V. J., Cheng, C., Kanter, D., et al. (2019) · arXiv (Cornell University) · doi:10.4230/oasics.parma-ditam.2021.4 · Cited by 35 · see citing works
Docker : build, ship, and run any app, anywhere
Kek, P. Q. (2017) · DR-NTU (Nanyang Technological University) · Cited by 31 · see citing works
Hardware Architectures for Image Processing Acceleration
Lindoso, A., & Entre, L. (2009) · InTech eBooks · doi:10.5772/7066 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Cores
Kao, S.-C., & Krishna, T. (2022) · 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00065 · Cited by 58 · see citing works
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
- Authors
- Richins, D., Doshi, D., Blackmore, M., Thulaseedharan Nair, A., Pathapati, N., Patel, A., Daguman, B., Dobrijalowski, D., Illikkal, R., Long, K., Zimmerman, D., & Janapa Reddi, V.
- Affiliations
- The University of Texas at Austin (US)
- Intel (United States) (US)
- Venue
- 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)
- Publisher
- IEEE
- Published
- 2020-02
- Pages
- 515-528
- ISSN
- 1530-0897 (ISSN-L)
- OpenAlex
- W3016892718
- Cited by
- 23
- References
- 26
- Citations by year
- 2026: 2 · 2024: 6 · 2023: 2 · 2022: 3 · 2021: 7 · 2020: 2 · 2019: 1
- FWCI
- 2.66
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE International Symposium on High-Performance Computer Architecture (HPCA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Keywords
- AI acceleration, face recognition, storage bottlenecks, network bandwidth, total cost of ownership
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
