OpenEI: An Open Framework for Edge Intelligence

“It achieves AlexNet-level [39] accuracy with 50 times fewer parameters on ImageNet data set (510 times smaller than AlexNet).”

“ESE [59] used FPGAs to accelerate the LSTM model on mobile devices, which adopted the load-balance-aware pruning method to ensure high hardware utilization and the partitioned compressed LSTM model on multiple PEs to process LSTM data flow in parallel.”

“The edge is usually resource-constrained compared to the cloud data center, which is not a good fit for executing DNN represented AI algorithms since DNN requires a large footprint on both storage (as big as 500MB for VGG-16 Model [8]) and computing power (as high as 15300 MMA for executing VGG-16 model [9]).”

“MobileNets are generated mainly from depthwise separable convolutions, which were first introduced in the work of [33] and subsequently employed in Inception models [34].”

“The autonomous driving scenario has conducted many classic computer vision and deep learning algorithms [66], [67].”

“Last, with the support of OpenEI , the edge will need to handle multiple tasks which raises the problem of how to execute multiple tasks on a package in the meantime.”

“Last, with the support of OpenEI , the edge will need to handle multiple tasks which raises the problem of how to execute multiple tasks on a package in the meantime.”

“Since the algorithms will be deployed on the vehicle, which is a resource-constrained and real-time EC system, the algorithm should consider not only precision but also latency, as the end-to-end deep learning algorithm YOLOv3 [68].”

“Similar to TensorFlow Lite [15], package manager is a lightweight deep learning package which has been optimized to run AI algorithms on the edge platform, which guarantees the low power consumption and low memory footprint. package manager is installed on the operating system of edge hardware and provides a running e…”

“published KITTI benchmark datasets [69], which provide a large quantity of camera and LiDAR data for various autonomous driving applications.”

“Edge Computing (EC) guarantees quality of service when dealing with a massive amount of data for cloud computing [1].”

“The autonomous driving scenario has conducted many classic computer vision and deep learning algorithms [66], [67].”

“The values of the parameters are adjusted using the standard backpropagation method [23] during training.”

“1) Compression: Compression techniques are roughly categorized into three groups: Parameter sharing and pruning methods, low-rank approximation methods, and knowledge transfer methods [18], [19].”

“They are deployed on the highperformance platforms, such as GPU, CPU, FPGA, and ASIC (TPU [45]) clusters.”

“is to handle concurrency intensive operations with small physical size and low power consumption [50].”

“In industry, many leaders have published some dedicated hardware modules to accelerate EI applications; for example, IBM TrueNorth [57] and Intel Loihi [58] are both the neuromorphic processors.”

“It is very important and urgent to develop a lightweight, efficient and highly-scalable framework to support AI algorithms on edges.”

“EIE [56] is an efficient hardware design for compressed DNN inference.”

“Taking image classification as an example, more than 10 AI models (AlexNet, Vgg, ResNet, MobileNet, to name a few), 5 packages (TensorFlow, PyTorch, MXNet, to name a few), and 10 edge hardware platforms (NVIDIA Jetson TX2, Intel Movidius, Mobile Phone, to name a few) need to be considered.”

“[24] pruned redundant connections using a three-step method: First, the network learns which connections are important, and then they prune the unimportant connections.”

“TensorRT is a platform for high-performance deep learning inference, not training and will be deployed on the cloud and edge platforms.”

“[21] employed the k-means clustering algorithm to quantize the weights of fully connected layers, which could achieve up to 24 times the compression of the network with only 1% loss of classification accuracy for the CNN network in the ImageNet challenge.”

“Users are allowed to integrate the trained machine learning model into Apple products, such as Siri, Camera, and QuickType.”

“ShiDianNao [55] first proposed that the artificial intelligence processor should be deployed next to the camera sensors.”

“1) Compression: Compression techniques are roughly categorized into three groups: Parameter sharing and pruning methods, low-rank approximation methods, and knowledge transfer methods [18], [19].”

“For example, DDNN [17] is a distributed deep neural network architecture across the cloud and edge.”

“Based on the idea of function-preserving transformations, the work in [32] instantaneously transfers the knowledge from a teacher network to each new deeper or wider network.”

“Sainath [27] uses a low-rank matrix factorization on the final weight layer of a DNN for acoustic modeling.”

“This section will illustrate the typical application scenarios and discuss how to leverage OpenEI to support these applications.”

“OpenVDAP [52], Autoware [73], and Baidu Apollo [74] are open-source software frameworks for autonomous driving, which provide interfaces for developers to build and customize autonomous driving vehicles.”

“[26] employ low-rank approximation to compress the weights of different layers and reduce the number of dynamic parameters.”

“explored the hardware computing platform design of autonomous vehicles [70].”

“Health and biomedicine are entering a data-driven epoch [81], [82].”

“The work in [30] trained a parametric student model to estimate a Monte Carlo teacher model.”

“Flattened networks [35] are designed for fast feedforward execution.”

“[31] use the neurons in the hidden layer to generate more compact models and preserve as much of the label information as possible.”

“They consist of a consecutive sequence of one-dimensional filters that span every direction of three-dimensional space to achieve comparable performance as conventional convolutional networks [36].”

“Then, they developed EMI-RNN [42] and FastGRNN [43] in 2018.”

“The idea of knowledge transfer is to adopt a teacher-student strategy and use a pre-trained network to train a compact network for the same task [28].”

“Health and biomedicine are entering a data-driven epoch [81], [82].”

“Recently, heterogeneous hardware platforms present the potential to accelerate specific deep learning algorithms while reducing the processing time and energy consumption [53], [54].”

“In 2018, [11] discussed the challenges and the opportunities that EI created by presenting a use-case showing that the careful design of the convolutional neural networks (CNN) for object detection would lead to real-time performance on embedded edge devices.”

“OpenVDAP [52] is an edge based data analysis platform for Connected and Autonomous Vehicles(CAVs).”

“made a comprehensive performance comparison of several state-of-the-art deep learning frameworks on the edges and evaluated the latency, memory footprint, and energy of these frameworks with two popular deep learning models on different edge devices [48].”

“To evaluate the performance of the computing platform designed for CAVs, Wang et al. first proposed CAVBench [72], which takes six diverse on-vehicle applications as evaluation workloads and provides the matching factor between the workload and the computing platform.”

“MUVR is proposed in this scenario to boost the multi-user gaming experience with the edge caching mechanism [80].”

“In [62], an edge based real-time video analysis system for public safety is proposed to distribute the computing workload in both the edge node and the cloud in an optimized way.”

“A3 [63] is an edge based amber alert application which support distributed collaborative execution on the edge.”

“SafeShareRide [64] is an edge based detection platform which enables a smartphone to conduct real-time detection including video analysis for both passengers and drivers in ridesharing services.”

“proposed a CNN model running on edge devices in a smart home to recognize activity with promising results [12].”

“We also hope that this paper provides helpful information to researchers and practitioners from various disciplines when designing new technologies and building new applications for EI.”

“deployed an artificial neural network on wearable sensors to detect emotion [84].”

“Because the video stream occupies a considerable storage space and transmission bandwidth, it is almost impossible to upload every frame recorded from a surveillance system to the cloud for further processing, especially for high resolution videos [79].”

“The US government has promoted the sharing of personal health information since 2009 [85], but it turns out that although doctors’ usage of electronic medical records has improved, interoperability is missing, and the large amount of medical data gathered together does not produce real value.”

“found that a smart home will benefit from EI to achieve energy efficiency [77].”

“OpenEI [183] defines each DL algorithm as a four-element tuple ¡Accuracy, Latency, Energy, Memory Footprint¿ to evaluate the Edge DL capability of the target hardware platform.”

“[11] defined edge intelligence as the capability to enable edges to execute AI algorithms.”

“OpenEI [70] is a lightweight software platform to equip edges with intelligent processing and data sharing capability.”

“This article defines edge AI as Zhang et al. [9] does: “the capability to enable edges to execute AI algorithms.””

“This greatly enhances both the efficiency and speed of data processing while enhancing data privacy [117], [118], [119].”

“To address these challenges, OpenEI is proposed as an Open Framework for Edge Intelligence [66].”

“[201] introduced an Open Framework for Edge Intelligence (OpenEI), a lightweight software platform for EI applications.”

“OpenEI [82], an edge intelligence framework that was proposed by Xingzhou Zhang et.al.”

“Edge Intelligence [39] significantly reduces communication costs and simultaneously plays data consumer and data generator by executing multitasking such as caching, managing, offloading, and service delivery.”

“OpenEI is an open lightweight framework for edge intelligence [46].”

“[5] introduce an Open Framework for Edge Intelligence (OpenEI) which is a lightweight software platform to equip the edge with intelligent processing and data sharing capability.”

“Current datasets vary in logging approaches which brings challenges to model or cross-data transformation which can also create a bias on the developed AI algorithm.”

“OpenEI [46] is a framework with a lightweight deep learning Package Manager similar to TensorFlow Lite, optimized to run AI algorithms at the edge, which guarantees low power consumption and low memory footprint.”

“• OpenEI : This framework, developed by Zhang et al. [198], is an open-source lightweight software platform to add intelligent processing and data sharing capabilities to the network edge.”

“Various studies in the literature have proposed different implementations of edge computing-based architectures with specialized application domains [51,67,73,170].”

“For overcoming this challenge and enabling edge intelligence, both Zhou et al. [10] and Zhang et al. [15] summarize some technologies that enable edge intelligence model inference, including model compression, model partition, conditional computation, etc.”

“Zhang et al. [32] believed that the scope of edge intelligence should not be limited to running Artificial Intelligence (AI) models on edge servers or devices but in a collaborative way of edge and cloud.”

“…typical of distributed computing and, hence, of the IoT [61], such as scalability [53,55], security and privacy [12,50,52], ethical issues [7,53], pervasiveness and ubiquity [7,14], resource optimization [48,52,54], heterogeneity [14,15,50,54], data scarcity and consistency [35,47,54], etc.”

“To the author’s knowledge, Open EI [283] is the only framework that provides a generic approach to facilitate the development of applications for a wide range of problem domains (computer vision, natural language processing, etc.”

“Accordingly, it is natural to rely on collaboration in edge learning, whereby IoT devices work together to accomplish computation-intensive tasks [3].”

“From enabling real-time object recognition in autonomous vehicles to facilitating personalized healthcare monitoring on wearable devices, Edge AI stands ready to transform how we harness the power of artificial intelligence in our daily lives [5].”

“Thus, for essential applications, latency becomes insignificant at a certain point of edge computing [43].”

“It has also been identified that the solution provided by the Edge AI is better than the cloud [6,7] AI.”

“This approach is particularly advantageous when handling numerous distributed data sources where cloud computing may not be feasible [12].”

“Edge intelligence was defined as the capability to enable edges to execute artificial intelligence algorithms in [11].”

“Some research has explored edge computing for AI [45], while other research has explored AI for edge computing [40].”

“extensively used in a wide range of computer vision applications and hardware platforms [1], their deployment on resource-limited edge devices is not a trivial process, as they are normally both compute and memory intensive [2].”

“Moreover, often customers wanted to keep their security and privacy, and thus do not want to expose their data to the external clouds [33], [20], [37], [11].”

“A large portion of jobs executed by robots are related to computer vision, which exploit cameras and machine learning models to make the robots more intelligent and autonomous [51].”

“Edge Computing is another key technology that can be implemented within IoD and blockchain infrastructures for data processing and storage at the utilized network’s edge [182], including the AI known as “edge AI” [183].”

“Having dynamic and rational task allocation methods for edge AI systems has an urgent demand [45, 46].”

“In [12], EI is defined as enabling edges to execute AI algorithms.”

“To perform tasks consistently in real time often requires high memory usage; although, memory itself is limited at the edge [10].”

“Zhang et al. [52] utilized edge-AI for autonomous vehicles to process real-time AI tasks like pedestrian detection, object detection, and traffic detection using Intel Movidius.”

“Therefore, the new paradigm of decentralizing computing and storage to the edge has gradually become an important research interest [1–3].”

“environmental data through multiple gateways and computing environment to a centrally located controller (generally in a cloud environment), which in turn is responsible for implementing and ensuring the enforcement of the higher level of automation logic [1], [2] , as shown in Figure 1.”

“In the second stage, we also provide excellent image classification models [14] and image segmentation models [15], among which the classification model includes Resnet50 and DenseNet, and the segmentation model includes Unet.”

“As described in Reference [155], the first step towards having intelligence at the edge is to collect the data from various sources.”

“Many efforts have been devoted to enabling edge intelligence [12], [13], in both hardware (e.”

“A large portion of jobs executed by AMRs is related to computer vision, which exploits cameras and machine learning models to make the robots more intelligent and autonomous [26].”

“This classification is based on recent research that identified different layers and tasks [15], [6].”

“Zhang et al. [169] further devise a framework combined with application-level optimization, such as adaptive model selection.”

“The performance of DNN models on edge devices was investigated in [57], in which an open-source framework was employed to determine whether edge devices meet the performance requirements of intelligent video analytics.”

“Edge intelligence for service deployment is discussed in Zhang [37].”

“Edge computing has been identified as a potential solution to address some of these limitations by bringing computation closer to the data source [3].”

“In [5], a framework to aid system designers in selecting a model for edge deployment is proposed.”

“The emergence of advanced novel applications have demanded low network latency [2].”

“Edge AI is currently confronted with a slew of challenges in the academic community [4].”

“Edge computing has the potential to overcome these limitations by allocating the computation closer to the network edge and data sources especially in real-time applications [14].”

“It is anticipated that edge intelligence will facilitate almost every aspect of our daily lives by provisioning various applications such as smart healthcare [1], autonomous driving [2], and advanced video analytics [3].”

“Meanwhile, collaboration among various edge terminals provide more potentiality for improving system performance [3].”

“Since the resources of computation and memory on edge devices are limited, two categories of solutions proposed for the limitation [72, 73] have emerged: lightweight network and model compression [74].”

“To the author’s knowledge, Open EI [283] is the only framework that provides a generic approach to facilitate the development of applications for a wide range of problem domains (computer vision, natural language processing, etc.”

“We compare our system with AIfES [25], InfiniEdge AI 1 , and OpenEI [26].”

“By adding the module that implements the matching library functionality, software like OpenCV[9,10] may perform GPU acceleration.”

“With regards to purely Edge based AI, Zhang et al. developed an AI platform called “OpenEI” [8].”

“Recently, there are high interests in investigating the capabilities, potential architectures, frameworks, techniques, and applications of edge intelligence [5][6][7][8][9][10][11] [12][13].”

“Another framework for edge AI is provided in [10] without providing detailed solution though.”

“Zhang et al. [23] proposed the “ OpenEI ” open-edge intelligent framework to “ marginalize ” the model training process.”

“The second thrust is to design an edge computing system that is viable to provide intelligence for various applications [2].”

“To address these challenges, researchers have developed specialized feature extraction techniques that are tailored for edge computing environments, leveraging the unique characteristics of edge devices and the data they generate [34].”

“Several Edge Intelligence applications require the deployment of CNNs on resourceconstrained devices [20].”

“A collaborative sensors movement algorithm is proposed on a basis of target learning to minimize the energy consumption [4].”

“The shift to edge AI introduces significant challenges, primarily inference latency and memory usage [6] [7] [39].”

“However, edge learning lacks the same support; solutions such as Azure IoT Edge 2 and OpenEI [5] generally only enable edge intelligence, and in limited cases edge learning, whereas most recent studies focus on edge learning specifically through FL, or are aimed at specialized use cases that only…”

“Similar to U2Net [23], our EADB-Net constructs two output branches, where each branch generates six probability maps, Mask i Edge i i ∈ [1 , 6] , from the last downsampling layer and all upsampling layers through a 3 × 3 convolutional layer and a sigmoid function.”

“• Distributed edge learning [58]: Several edge nodes can cooperate to train large datasets.”

“Currently, edge artificial intelligence has made significant progress (X. Zhang et al. 2019).”

“The framework of deep learning model training and reasoning at edge by mobile devices and edge collaboration has been discussed [6]–[9], but the service placement and migration of multi edge clouds are not considered in ultar-intensive mobile edge cloud and user mobility.”

“Wearable device data, diagnostic findings, sleep tests, and electronic health records may all be readily linked [16-18].”

“In TensorFlow frameworks, the model’s data is quantized in three ways: i) converting 32-bit floats Accuracy [8] Accuracy [9] Compression [10] Partition [11] DAG Partitioning [12] Clustering [13] Hardware - in-memory [14] Grouping [15] Tuning [16] Caching, Offloading [17] Service Placement [18] Edge…”

“[8] present an intelligent open framework for the edge that intelligently provides edge devices with the ability to process data and share data.”

“To address this issue, [1] introduces a lightweight software framework that provides edge devices capacity to interpret and share data intelligently.”

“In recent years, with the development of the Internet of Vehicles and edge computing technologies [14-17], the issue of how to plan efficient distribution routes has become more and more worthy of consideration [18-20].”

“Precog [10], CCO[19], BLOT[20], OpenEI[21], AMiSS[22] and deep learning-based approach [23] offloaded computation task to edge server in MEC or IoT scenarios.”

“The second focuses on model adaptation through lightweight architectures or compression techniques such as pruning and quantization [8]–[11].”

“MEC literature focuses on lowering latency by performing low-level data processing near the user, and improvements in performance have been demonstrated when analyzing edge-centric services [7].”

“EdgeML based event detectors have also been investigated for in sensor decision making [13], but they often entail high computational overhead or rely on complex training pipelines not suitable for embedded WuRx integration.”

“However, the issues of edge deployment and privacy are not sufficiently addressed, because cloud-based AI systems transmit data, privacy concerns are raised [8].”