Edge Assisted Real-time Object Detection for Mobile Augmented Reality
The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Most existing Augmented Reality (AR) and Mixed Reality (MR) systems are able to understand the 3D geometry of the surroundings but lack the ability to detect and classify complex objects in the real world. Such capabilities can be enabled with deep Convolutional Neural Networks (CNN), but it remains difficult to execute large networks on mobile devices. Offloading object detection to the edge or cloud is also very challenging due to the stringent requirements on high detection accuracy and low end-to-end latency. The long latency of existing offloading techniques can significantly reduce the detection accuracy due to changes in the user's view. To address the problem, we design a system that enables high accuracy object detection for commodity AR/MR system running at 60fps. The system employs low latency offloading techniques, decouples the rendering pipeline from the offloading pipeline, and uses a fast object tracking method to maintain detection accuracy. The result shows that the system can improve the detection accuracy by 20.2%-34.8% for the object detection and human keypoint detection tasks, and only requires 2.24ms latency for object tracking on the AR device. Thus, the system leaves more time and computational resources to render virtual elements for the next frame and enables higher quality AR/MR experiences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the frame regions containing small objects need a higher resolution to guarantee object detection accuracy according to the conclusion in [19]”
From Orchestrating Joint Offloading and Scheduling for Low-Latency Edge SLAM (2025) · cites this paper as [19]
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“Huang et al. [29] compare the speed and accuracy trade-offs for modern CNN object detection models, including Faster R-CNN [30], R-FCN [31] and SSD [32].”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · cited 1×
“Huang et al. [29] compare the speed and accuracy trade-offs for modern CNN object detection models, including Faster R-CNN [30], R-FCN [31] and SSD [32].”
Method(2016) · Cited by 31,590
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128 · cited 1×
“Huang et al. [29] compare the speed and accuracy trade-offs for modern CNN object detection models, including Faster R-CNN [30], R-FCN [31] and SSD [32].”
MethodMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · cited 1×
“There have been extensive works on how to efficiently run these CNN models on mobile devices [35–42].”
BackgroundOverview of the H.264/AVC video coding standard
Wiegand, T., Sullivan, G. J., Bjontegaard, G., & Luthra, A. (2003) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2003.815165 · Cited by 8,184 · cited 1×
“To reduce the bandwidth consumption and streaming latency t stream , the raw frames are compressed to H.264 video streams on the device and decoded in the edge cloud [10].”
MethodHe, K., Gkioxari, G., Dollar, P., & Girshick, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2844175 · Cited by 3,457
R-FCN: Object Detection via Region-based Fully Convolutional Networks
Dai, J., Li, Y., He, K., & Sun, J. Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06409 · Cited by 3,149 · cited 2×
“Huang et al. [29] compare the speed and accuracy trade-offs for modern CNN object detection models, including Faster R-CNN [30], R-FCN [31] and SSD [32].”
MethodSpeed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.351 · Cited by 2,743 · cited 1×
“Huang et al. [29] compare the speed and accuracy trade-offs for modern CNN object detection models, including Faster R-CNN [30], R-FCN [31] and SSD [32].”
MethodNeural Adaptive Video Streaming with Pensieve
Mao, H., Netravali, R., & Alizadeh, M. (2017) · Proceedings of the Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3098822.3098843 · Cited by 1,357 · cited 1×
“Other video adaptation techniques [56–59] are also complementary to our work.”
ResultRanjan, R., Patel, V. M., & Chellappa, R. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2781233 · Cited by 1,333
Mask R-CNNPreprint
He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017) · arXiv (Cornell University) · Cited by 1,252
Quantized Convolutional Neural Networks for Mobile Devices
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.521 · Cited by 1,212
Chameleon: scalable adaptation of video analytics
Jiang, J., Ananthanarayanan, G., Bodik, P., Sen, S., & Stoica, I. (2018) · Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3230543.3230574 · Cited by 467 · cited 1×
“VideoStorm [45] and Chameleon [46] achieve higher accuracy video analytics with the same amount of computational resources on the cloud by adapting the video configurations.”
BackgroundDeepDecision: A Mobile Deep Learning Framework for Edge Video Analytics
Ran, X., Chen, H., Zhu, X., Liu, Z., & Chen, J. (2018) · IEEE INFOCOM 2018 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2018.8485905 · Cited by 450 · cited 2×
“DeepDeci-sion [9] designs a framework to decide whether to offload the object detection task to the edge cloud or do local inference based on current network conditions.”
BackgroundData Distillation: Towards Omni-Supervised Learning
Radosavovic, I., Dollar, P., Girshick, R., Gkioxari, G., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00433 · Cited by 421
CS2P: Improving Video Bitrate Selection and Adaptation with Data-Driven Throughput Prediction
Sun, Y., Yin, X., Jiang, J., Sekar, V., Lin, F., Wang, N., Liu, T., & Sinopoli, B. (2016) · Proceedings of the 2016 ACM SIGCOMM Conference · doi:10.1145/2934872.2934898 · Cited by 407
He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06870 · Cited by 406 · cited 1×
“Thanks to the idea of multitask learning, current CNN can further reuse the deep features inside the object bounding box for more fine-grained detection, such as instance segmentation [33], human key points detection [33], facial landmark detection [34], etc.”
BackgroundOptimizing 360 video delivery over cellular networks
Qian, F., Ji, L., Han, B., & Gopalakrishnan, V. (2016) · Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges · doi:10.1145/2980055.2980056 · Cited by 402
Han, S., Shen, H., Philipose, M., Agarwal, S., Wolman, A., & Krishnamurthy, A. (2016) · Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906396 · Cited by 367
Glimpse: Continuous, Real-Time Object Recognition on Mobile Devices
Chen, T. Y.-H., Ravindranath, L., Deng, S., Bahl, P., & Balakrishnan, H. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809711 · Cited by 358 · cited 2×
“Glimpse [8] is a continuous object recognition system on the mobile device designed for cloud offload under challenging network conditions.”
Background(2017) · Networked Systems Design and Implementation · Cited by 329
DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications
Huynh, L. N., Lee, Y., & Balan, R. K. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081360 · Cited by 329
LAVEA: latency-aware video analytics on edge computing platform
Yi, S., Hao, Z., Zhang, Q., Zhang, Q., Shi, W., & Li, Q. (2017) · Proceedings of the Second ACM/IEEE Symposium on Edge Computing · doi:10.1145/3132211.3134459 · Cited by 244 · cited 1×
“Lavea [44] offloads computation between clients and nearby edge nodes to provide low-latency video analytics.”
BackgroundOn-Demand Deep Model Compression for Mobile Devices: A Usage-Driven Model Selection Framework
Liu, S., Lin, Y., Zhou, Z., Nan, K., Liu, H., & Du, J. (2018) · Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3210240.3210337 · Cited by 217 · cited 1×
“There have been extensive works on how to efficiently run these CNN models on mobile devices [35–42].”
BackgroundChen, Z., Hu, W., Wang, J., Zhao, S., Amos, B., Wu, G., Ha, K., Elgazzar, K., Pillai, P., Klatzky, R., et al. (2017) · Proceedings of the Second ACM/IEEE Symposium on Edge Computing · doi:10.1145/3132211.3134458 · Cited by 213
RedEye: analog ConvNet image sensor architecture for continuous mobile vision
LiKamWa, R., Hou, Y., Gao, J., Polansky, M., & Zhong, L. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001164 · Cited by 189
Outatime: Using Speculation to Enable Low-Latency Continuous Interaction for Mobile Cloud Gaming
Lee, K., Chu, D., Cuervo, E., Kopf, J., Degtyarev, Y., Grizan, S., Wolman, A., & Flinn, J. (2015) · Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2742647.2742656 · Cited by 175 · cited 1×
“Adaptive video streaming techniques have also been adopted by mobile gaming [51, 52] and virtual reality system [53–55] to achieve high quality experience on mobile thin clients.”
BackgroundMP-DASH: Adaptive Video Streaming Over Preference-Aware Multipath
Han, B., Qian, F., Ji, L., & Gopalakrishnan, V. (2016) · Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies · doi:10.1145/2999572.2999606 · Cited by 153
FlashBack: Immersive Virtual Reality on Mobile Devices via Rendering Memoization
Boos, K., Chu, D., & Cuervo, E. (2016) · Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906418 · Cited by 135 · cited 1×
“In addition, as mixed reality graphics approach the complexity of VR, one can also expect them to require less than 20ms motion-to-photon latency, which has been found to be necessary to avoid causing user motion sickness in VR applications [7].”
BackgroundRubiks: Practical 360-Degree Streaming for Smartphones
He, J., Qureshi, M. A., Qiu, L., Li, J., Li, F., & Han, L. (2018) · Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3210240.3210323 · Cited by 130
Mathur, A., Lane, N. D., Bhattacharya, S., Boran, A., Forlivesi, C., & Kawsar, F. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081359 · Cited by 128
AVR: Augmented Vehicular Reality
Qiu, H., Ahmad, F., Bai, F., Gruteser, M., & Govindan, R. (2018) · Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3210240.3210319 · Cited by 124 · cited 1×
“Other mobile AR work [26–28] also provide useful insights for us.”
BackgroundSelf-citationOverLay: Practical Mobile Augmented Reality
Jain, P., Manweiler, J., & Roy Choudhury, R. (2015) · Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2742647.2742666 · Cited by 116 · cited 1×
“Overlay [24] uses sensor data from the mobile device to reduce the number of candidate objects.”
MethodLiu, L., Zhong, R., Zhang, W., Liu, Y., Zhang, J., Zhang, L., & Gruteser, M. (2018) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3210240.3210313 · Cited by 115
Self-citationKahawai: High-Quality Mobile Gaming Using GPU Offload
Cuervo, E., Wolman, A., Cox, L. P., Lebeck, K., Razeen, A., Saroiu, S., & Musuvathi, M. (2015) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2742647.2742657 · Cited by 93 · cited 1×
“Adaptive video streaming techniques have also been adopted by mobile gaming [51, 52] and virtual reality system [53–55] to achieve high quality experience on mobile thin clients.”
BackgroundOn the Networking Challenges of Mobile Augmented Reality
Zhang, W., Han, B., & Hui, P. (2017) · Proceedings of the Workshop on Virtual Reality and Augmented Reality Network · doi:10.1145/3097895.3097900 · Cited by 82 · cited 1×
“Other mobile AR work [26–28] also provide useful insights for us.”
Background360° Innovations for Panoramic Video Streaming
Liu, X., Xiao, Q., Gopalakrishnan, V., Han, B., Qian, F., & Varvello, M. (2017) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3152434.3152443 · Cited by 70 · cited 1×
“Several 360-degree video streaming works [47–50] also adopt the idea of RoI encoding to reduce the latency and bandwidth consumption of the streaming process.”
BackgroundCuervo, E., Chintalapudi, K. K., & Kotaru, M. (2018) · IEEE Workshop on Mobile Computing Systems and Applications (WMCSA) · doi:10.1145/3177102.3177115 · Cited by 65
RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision
LiKamWa, R., Hou, Y., Gao, Y., Polansky, M., & Zhong, L. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.31 · Cited by 61
CARS: Collaborative Augmented Reality for Socialization
Zhang, W., Han, B., Hui, P., Gopalakrishnan, V., Zavesky, E., & Qian, F. (2018) · IEEE Workshop on Mobile Computing Systems and Applications (WMCSA) · doi:10.1145/3177102.3177107 · Cited by 59
Low Bandwidth Offload for Mobile AR
Jain, P. K., Manweiler, J., & Choudhury, R. R. (2016) · ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT) · doi:10.1145/2999572.2999587 · Cited by 57 · cited 1×
“VisualPrint [25] aims to reduce the bandwidth consumption of image offloading by only transmit the extracted feature points to the cloud.”
BackgroundXie, X., & Zhang, X. (2017) · ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT) · doi:10.1145/3143361.3143381 · Cited by 55
How will Deep Learning Change Internet Video Delivery?
Yeo, H., Do, S., & Han, D. (2017) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3152434.3152440 · Cited by 50
FlashBack: Immersive Virtual Reality on Mobile Devices via Rendering Memoization
Boos, K., Chu, D., & Cuervo, E. (2017) · GetMobile: Mobile Computing and Communications · doi:10.1145/3081016.3081026 · Cited by 47
On Building a Programmable Wireless High-Quality Virtual Reality System Using Commodity Hardware
Zhong, R., Wang, M., Chen, Z., Liu, L., Liu, Y., Zhang, J., Zhang, L., & Moscibroda, T. (2017) · Asia-Pacific Workshop on Systems · doi:10.1145/3124680.3124723 · Cited by 23 · cited 1×
“Adaptive video streaming techniques have also been adopted by mobile gaming [51, 52] and virtual reality system [53–55] to achieve high quality experience on mobile thin clients.”
BackgroundSelf-citation
Research works printed but not linked: 2
- Kaiming He , Georgia Gkioxari , Piotr Dollár , and Ross Girshick . Maskr-cnn. In Computer Vision (ICCV) , 2017 IEEE International Conferenceon , pages 2980 -- 2988 . IEEE, 2017 . Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Maskr-cnn. In Computer Vision (ICCV), 2017 IEEE International Conferenceon, pages 2980--2988. IEEE, 2017.
- 10.5555/3154630.3154661 · doi:10.5555/3154630.3154661
Cited by
Deep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540
“, as used in [33]), with Nvidia-provided SDKs used to program the devices.”
BackgroundMethodConvergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
“Interest (RoI) encoding [140] to focus on target objects in video frames can further reduce the bandwidth consumption and data transmission delay.”
BackgroundSiriwardhana, Y., Porambage, P., Liyanage, M., & Ylianttila, M. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3061981 · Cited by 674
“Multi-access Edge Computing (MEC) brings the computations closer to the end devices, minimizing the latency of cloud offloading and facilitating the latency-critical applications [42]–[45].”
BackgroundXu, M., Ng, W. C., Lim, W. Y. B., Kang, J., Xiong, Z., Niyato, D., Yang, Q., Shen, X., & Miao, C. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3221119 · Cited by 564
“However, resources on the edge networks are limited, while the customization and communication resources used for AR adaptation are enormous [97], [98], [99].”
BackgroundJoint Task Offloading and Resource Allocation for Energy-Constrained Mobile Edge Computing
Jiang, H., Dai, X., Xiao, Z., & Iyengar, A. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3150432 · Cited by 411
“Based on this, MEC drives the development of many emerging applications, such as augmented reality (AR), natural language processing and realtime video analytics [1].”
Background
Metadata
- Authors
- Liu, L., Li, H., & Gruteser, M.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- Venue
- The 25th Annual International Conference on Mobile Computing and Networking
- Publisher
- ACM
- Published
- 2019-08-05
- Pages
- 1-16
- ISSN
- 1543-5679 (ISSN-L)
- OpenAlex
- W2965289829
- Cited by
- 552
- References
- 47
- Citations by year
- 2026: 33 · 2025: 72 · 2024: 93 · 2023: 92 · 2022: 119 · 2021: 70 · 2020: 62 · 2019: 11
- FWCI
- 21.35
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Annual International Conference on Mobile Computing and Networking (MobiCom) (publishedVersion)
- Funders
- National Science Foundation (1329939, 1827923)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Augmented Reality Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- mobile augmented reality, mixed reality, real-time object detection, edge computing, computation offloading, low latency, object tracking, human keypoint detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.