NoScope: optimizing neural network queries over video at scale
Open accessThe DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index.
Abstract
Recent advances in computer vision---in the form of deep neural networks---have made it possible to query increasing volumes of video data with high accuracy. However, neural network inference is computationally expensive at scale: applying a state-of-the-art object detector in real time (i.e., 30+ frames per second) to a single video requires a $4000 GPU. In response, we present N o S cope , a system for querying videos that can reduce the cost of neural network video analysis by up to three orders of magnitude via inference-optimized model search. Given a target video, object to detect, and reference neural network, N o S cope automatically searches for and trains a sequence, or cascade, of models that preserves the accuracy of the reference network but is specialized to the target video and are therefore far less computationally expensive. N o S cope cascades two types of models: specialized models that forego the full generality of the reference model but faithfully mimic its behavior for the target video and object; and difference detectors that highlight temporal differences across frames. We show that the optimal cascade architecture differs across videos and objects, so N o S cope uses an efficient cost-based optimizer to search across models and cascades. With this approach, N o S cope achieves two to three order of magnitude speed-ups (265-15,500x real-time) on binary classification tasks over fixed-angle webcam and surveillance video while maintaining accuracy within 1--5% of state-of-the-art neural networks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“In recent times, however, deep neural networks (NNs) [40, 44, 58, 66, 67] have largely displaced classical computer”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“NOSCOPE implements specialized NNs based on the AlexNet architecture [56] (filter doubling, dropout), using ReLU units for all the hidden layers and a softmax unit at the end to return a confidence for the class we are querying.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · cited 19×
“However, even in this video, NOSCOPE can offer an 30× speedup over YOLOv2 with no loss in accuracy.”
BackgroundInfluentialMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 4×
“geNet 2012 [82]), a variety of SIFT-based bag-of-words classifiers (varying the SIFT parameters), and selective search with a SIFTbased classifier [63] (which performed well on PASCAL VOC [32]).”
BackgroundMethodInfluentialYOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · cited 2×
“We used YOLOv2 [80] as our reference model.”
MethodRapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300 · cited 2×
“One of the first cascades, the Viola-Jones detector [90], cascades traditional image processing features; if any classifier is confident about the output, the Viola-Jones cascade short-circuits evaluation, improving execution speed.”
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · cited 4×
“geNet 2012 [82]), a variety of SIFT-based bag-of-words classifiers (varying the SIFT parameters), and selective search with a SIFTbased classifier [63] (which performed well on PASCAL VOC [32]).”
BackgroundMethodInfluentialAre we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 1×
“Therefore, with few exceptions [37], accelerated methods are not favored by the most prestigious competitions in the field.”
BackgroundDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 3×
“Shallow NNs have been shown to be effective in other compression routines [45], are efficient at inference time, and naturally output a confidence in their classification.”
Background(2016) · Cited by 11,679
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853 · cited 1×
“More recent and advanced methods such as HOG [100], deformable parts model [34] and selective search [88] are among the most sophisticated of these classic approaches.”
MethodFace recognition: A literature survey
Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2003) · ACM Computing Surveys · doi:10.1145/954339.954342 · Cited by 6,163 · cited 1×
“Classic techniques [64, 86, 99] combine machine learning methods such as classification and clustering with image-specific featurization techniques such as SIFT [63].”
MethodSelective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 1×
“Now, these “deep” models (with millions to billions of parameters) have become not only feasible but also the preferred method for computer vision tasks spanning image classification [82], pedestrian detection [98], and automatic captioning [29].”
MethodYilmaz, A., Javed, O., & Shah, M. (2006) · ACM Computing Surveys · doi:10.1145/1177352.1177355 · Cited by 4,792 · cited 1×
“Beyond object detection, object tracking refers to the task of tracking an object through a video and is a vital tool in video analysis [93].”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 1×
“Another strategy is to modify the network weights directly, either via hashing [22] or pruning and literal compression of the weights themselves [41, 77].”
BackgroundOnline Object Tracking: A Benchmark
Wu, Y., Lim, J., & Yang, M.-H. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.312 · Cited by 4,138 · cited 1×
“2k fps [92], and the above methods run much slower.”
MethodNeural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974 · cited 1×
“In this way, NOSCOPE acts as a system for model search [14, 23, 102] for efficient inference and is inspired by conventional database cost optimization [83] and related efforts on self-adapting query processing engines [20, 47].”
MethodComputer vision: algorithms and applications
(2011) · Choice Reviews Online · doi:10.5860/choice.48-5140 · Cited by 3,911
A survey of image classification methods and techniques for improving classification performance
Lu, D., & Weng, Q. (2007) · International Journal of Remote Sensing · doi:10.1080/01431160600746456 · Cited by 3,402 · cited 1×
“Classic techniques [64, 86, 99] combine machine learning methods such as classification and clustering with image-specific featurization techniques such as SIFT [63].”
MethodPedestrian Detection: An Evaluation of the State of the Art
Dollar, P., Wojek, C., Schiele, B., & Perona, P. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.155 · Cited by 3,190 · cited 1×
“One of the most popular video-based object detection tasks is pedestrian detection [15, 28], with a range of NN-based approaches (e.”
BackgroundSpringer Science+Business Media
Технология (2013) · Betascript Publishing eBooks · Cited by 3,087
Query by image and video content: the QBIC system
Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petkovic, D., et al. (1995) · Computer · doi:10.1109/2.410146 · Cited by 3,060 · cited 3×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
BackgroundObject retrieval with large vocabularies and fast spatial matching
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383172 · Cited by 2,929 · cited 1×
“, this image is a photograph of a specific building on Stanford’s campus [76]).”
BackgroundMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579 · cited 3×
“To evaluate this hypothesis, we trained three deep NNs for binary classification on the classes of objects we evaluate NOSCOPE on: people, buses, and cars using the more general MS-COCO dataset [62], a recent high-quality object detection dataset.”
MethodInfluentialAccess path selection in a relational database management system
Selinger, P. G., Astrahan, M. M., Chamberlin, D. D., Lorie, R. A., & Price, T. G. (1979) · Proceedings of the 1979 ACM SIGMOD international conference on Management of data - SIGMOD '79 · doi:10.1145/582095.582099 · Cited by 2,188 · cited 1×
“In this way, NOSCOPE acts as a system for model search [14, 23, 102] for efficient inference and is inspired by conventional database cost optimization [83] and related efforts on self-adapting query processing engines [20, 47].”
MethodBuciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · cited 1×
“model. One strategy for model compression is to use the behavior of a larger model (e.g., its output confidences) to train a smaller model, either distilling an ensemble of models into a smaller model [14], or distilling one large network into one smaller network [7,40]. One can think of this form of distillation as an…”
MethodRobust object tracking with online multiple instance learning
Babenko, B., Yang, M.-H., & Belongie, S. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.226 · Cited by 2,007 · cited 1×
“license plate detection [5], counting cars [97], tracking pedestrians or cars [10]).”
BackgroundBeyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Danelljan, M., Robinson, A., Shahbaz Khan, F., & Felsberg, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_29 · Cited by 1,853 · cited 1×
“There are many such methods for tracking, including NN-based methods [24, 71] and traditional methods [27, 30], that all performed well on the VOT2016 challenge [55].”
MethodA Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection
Cai, Z., Fan, Q., Feris, R. S., & Vasconcelos, N. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_22 · Cited by 1,605
Fast Human Detection Using a Cascade of Histograms of Oriented Gradients
Zhu, Q., Yeh, M.-C., Cheng, K.-T., & Avidan, S. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.119 · Cited by 1,399
A convolutional neural network cascade for face detection
Li, H., Lin, Z., Shen, X., Brandt, J., & Hua, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299170 · Cited by 1,390 · cited 1×
“is confident about the output, the cascade short-circuits evaluation, thus improving execution speed. Recent work has focused on learning cascades for pedestrian detection [16] and facial recognition [48,70] image tasks. In NOSCOPE, we design a cascade architecture for use in video, specifically adapted to account for t…”
MethodDeep Convolutional Network Cascade for Facial Point Detection
Sun, Y., Wang, X., & Tang, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.446 · Cited by 1,382 · cited 1×
“is confident about the output, the cascade short-circuits evaluation, thus improving execution speed. Recent work has focused on learning cascades for pedestrian detection [16] and facial recognition [48,70] image tasks. In NOSCOPE, we design a cascade architecture for use in video, specifically adapted to account for t…”
MethodDo Deep Nets Really Need to be Deep?
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,335 · cited 2×
“, its output confidences) to train a smaller model, either distilling an ensemble of models into a smaller model [17], or distilling one large network into one smaller network [8, 45].”
BackgroundThe design of the borealis stream processing engine
Abadi, D. J., Ahmad, Y., Bałazińska, M., Cherniack, M., Hwang, J.-H., Lindner, W., Maskey, A. S., Rasin, A., Ryvkina, E., Tatbul, N., et al. (2005) · Cited by 1,255
EIE: Efficient Inference Engine on Compressed Deep Neural Network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.30 · Cited by 1,077
Compressing Neural Networks with the Hashing Trick
Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2015) · arXiv · doi:10.48550/arxiv.1504.04788 · Cited by 974 · cited 1×
“Another strategy is to modify the network weights directly, either via hashing [22] or pruning and literal compression of the weights themselves [41, 77].”
BackgroundYu, K.-H., Zhang, C., Berry, G. J., Altman, R. B., Ré, C., Rubin, D. L., & Snyder, M. (2016) · Nature Communications · doi:10.1038/ncomms12474 · Cited by 972 · cited 1×
“vision methods due to their incredible accuracy—often rivaling or exceeding human capabilities—in visual analyses ranging from object classification [82] to image-based cancer diagnosis [31, 95].”
BackgroundTelegraphCQ: Continuous Dataflow Processing for an Uncertain World.
Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M. J. M., Hellerstein, J. M., Hong, W., Krishnamurthy, S., Madden, S. R., Raman, V., Reiss, F. R., et al. (2003) · Conference on Innovative Data Systems Research · Cited by 947
ImageNet classification with deep convolutional neural networks
Krizhevsky (2012) · Medical Entomology and Zoology · Cited by 799
Pedestrian Detection with Unsupervised Multi-stage Feature Learning
Sermanet, P., Kavukcuoglu, K., Chintala, S., & Lecun, Y. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.465 · Cited by 742 · cited 1×
“els designed for detecting specific classes of objects in video. One of the most popular video-based object detection tasks is pedestrian detection [13,25], with a range of CNN-based approaches (e.g., [69]). In NOSCOPE, our goal is to specialize to a range of inputs automatically, using a large general CNN as guidance.…”
MethodGigascope: a stream database for network applications
Cranor, C., Johnson, T., Spataschek, O., & Shkapenyuk, V. (2003) · Proceedings of the 2003 ACM SIGMOD international conference on Management of data · doi:10.1145/872757.872838 · Cited by 687 · cited 1×
“Similarly, a range of systems from academia and industry provide stream processing functionality [4, 19, 26, 39]; here, we specifically focus on video stream processing.”
BackgroundThe Visual Object Tracking VOT2016 Challenge Results
Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Čehovin, L., Vojír̃, T., Häger, G., Lukežič, A., Fernández, G., et al. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-48881-3_54 · Cited by 679 · cited 1×
“There are many such methods for tracking, including NN-based methods [24, 71] and traditional methods [27, 30], that all performed well on the VOT2016 challenge [55].”
MethodChabot: retrieval from a relational database of images
Ogle, V. E., & Stonebraker, M. (1995) · Computer · doi:10.1109/2.410150 · Cited by 650 · cited 2×
“Chabot: Retrieval from a relational database of images.”
BackgroundLicense Plate Recognition From Still Images and Video Sequences: A Survey
Anagnostopoulos, C.-N. E., Anagnostopoulos, I. E., Psoroulas, I. D., Loumos, V., & Kayafas, E. (2008) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2008.922938 · Cited by 645 · cited 1×
“license plate detection [5], counting cars [97], tracking pedestrians or cars [10]).”
BackgroundDeep learning of binary hash codes for fast image retrieval
Lin, K., Yang, H.-F., Hsiao, J.-H., & Chen, C.-S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301269 · Cited by 614 · cited 1×
“The second class of techniques allows retrieval of similar images from a large corpus that are similar to a query image [61,96] (e.”
BackgroundSpatiotemporal Residual Networks for Video Action Recognition
Feichtenhofer, C., Pinz, A. J., & Wildes, R. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02155 · Cited by 597 · cited 1×
“Similar to NOSCOPE, several NNs specialized for video take multiple frames as input to improve accuracy [33, 91], but again, these NNs focus on accuracy, not inference speed.”
ResultT-CNN: Tubelets With Convolutional Neural Networks for Object Detection From Videos
Kang, K., Li, H., Yan, J., Zeng, X., Yang, B., Xiao, T., Zhang, C., Wang, Z., Wang, R., Wang, X., et al. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2736553 · Cited by 554 · cited 1×
“State-ofthe-art methods [43, 50, 51] follow a common pattern: run still image”
MethodA survey of video processing techniques for traffic applications
Kastrinaki, V, Zervakis, M., & Kalaitzakis, K. (2003) · Image and Vision Computing · doi:10.1016/s0262-8856(03)00004-0 · Cited by 527
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2016) · arXiv · doi:10.48550/arxiv.1612.08242 · Cited by 488 · cited 2×
“We used YOLOv2 [80] as our reference model.”
MethodHow Far are We from Solving Pedestrian Detection?
Zhang, S., Benenson, R., Omran, M., Hosang, J., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.141 · Cited by 454 · cited 1×
“Now, these “deep” models (with millions to billions of parameters) have become not only feasible but also the preferred method for computer vision tasks spanning image classification [82], pedestrian detection [98], and automatic captioning [29].”
MethodMinerva: enabling low-power, highly-accurate deep neural network accelerators
Reagen, B., Whatmough, P., Adolf, R., Rama, S., Lee, H., Lee, S. K., Hernández-Lobato, J. M., Wei, G.-Y., & Brooks, D. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001165 · Cited by 446 · cited 1×
“These networks are almost always more amenable to hardware acceleration—a related line of research [42, 78].”
BackgroundObject Detection from Video Tubelets with Convolutional Neural Networks
Kang, K., Ouyang, W., Li, H., & Wang, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.95 · Cited by 386 · cited 1×
“State-ofthe-art methods [43, 50, 51] follow a common pattern: run still image”
MethodMotion tracking with an active camera
Murray, D., & Basu, A. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.291452 · Cited by 352 · cited 1×
“The video processing community has also developed techniques to track objects when the camera is moving [70], so these techniques present a promising approach to extend NOSCOPE to moving cameras.”
MethodA survey on content-based retrieval for multimedia databases
Yoshitaka, A., & Ichikawa, T. (1999) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.755617 · Cited by 334 · cited 2×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
Background(2017) · Networked Systems Design and Implementation · Cited by 329
Learning Complexity-Aware Cascades for Deep Pedestrian Detection
Cai, Z., Saberian, M., & Vasconcelos, N. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.384 · Cited by 326 · cited 1×
“Recent work has focused on learning cascades for pedestrian detection [18] and facial recognition [60, 85] image tasks.”
BackgroundMLbase: A Distributed Machine-learning System
Kraska, T., Talwalkar, A., & Duchi, J. C. (2013) · Cited by 324
An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections
Cheng, Y., Yu, F. X., Feris, R. S., Kumar, S., Choudhary, A., & Chang, S.-F. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.327 · Cited by 303 · cited 1×
“In this way, NOSCOPE acts as a system for model search [14, 23, 102] for efficient inference and is inspired by conventional database cost optimization [83] and related efforts on self-adapting query processing engines [20, 47].”
MethodDo Deep Nets Really Need to be DeepPreprint
Ba, L. J., & Caruana, R. (2013) · arXiv (Cornell University) · Cited by 282
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural NetworksPreprint
Rastegari, M., Ordóñez, V., Redmon, J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.05279 · Cited by 279 · cited 1×
“Another strategy is to modify the network weights directly, either via hashing [22] or pruning and literal compression of the weights themselves [41, 77].”
BackgroundLearning to Track for Spatio-Temporal Action Localization
Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.362 · Cited by 278 · cited 1×
“Similar to NOSCOPE, several NNs specialized for video take multiple frames as input to improve accuracy [33, 91], but again, these NNs focus on accuracy, not inference speed.”
ResultModeling and Propagating CNNs in a Tree Structure for Visual Tracking
Nam, H., Baek, M., & Han, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07242 · Cited by 278 · cited 1×
“There are many such methods for tracking, including NN-based methods [24, 71] and traditional methods [27, 30], that all performed well on the VOT2016 challenge [55].”
MethodSelf-tuning database systems: a decade of progress
Chaudhuri, S., & Narasayya, V. R. (2007) · Cited by 249
Minerva: Enabling Low-Power, Highly-Accurate Deep Neural Network Accelerators
Reagen, B., Whatmough, P., Adolf, R., Rama, S., Lee, H., Lee, S. K., Hernandez-Lobato, J. M., Wei, G.-Y., & Brooks, D. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.32 · Cited by 246
Adaptive ordering of pipelined stream filters
Babu, S., Motwani, R., Munagala, K., Nishizawa, I., & Widom, J. (2004) · Proceedings of the 2004 ACM SIGMOD international conference on Management of data · doi:10.1145/1007568.1007615 · Cited by 229 · cited 2×
“[11]): one specialized model may complement one difference detector but not another, and the viable combinations of thresholds for each pair of models will differ across scenes.”
BackgroundMethodA Survey on the Automatic Indexing of Video Data,
Brunelli, R., Mich, O., & Modena, C. M. (1999) · Journal of Visual Communication and Image Representation · doi:10.1006/jvci.1997.0404 · Cited by 222
Image processing on compressed data for large video databases
Arman, F., Hsu, A., & Chiu, M.-Y. (1993) · Proceedings of the first ACM international conference on Multimedia - MULTIMEDIA '93 · doi:10.1145/166266.166297 · Cited by 206 · cited 2×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
BackgroundOptimization of queries with user-defined predicates
Chaudhuri, S., & Shim, K. (1999) · ACM Transactions on Database Systems · doi:10.1145/320248.320249 · Cited by 186 · cited 1×
“This problem is similar to traditional cost-based estimation for userdefined functions [21] and streaming filter processing [9, 11], but NOSCOPE must also perform cost estimation of each candidate model and set model-specific parameters for each together to achieve a target accuracy goal.”
MethodTowards a unified architecture for in-RDBMS analytics
Feng, X., Kumar, A., Recht, B., & Ré, C. (2012) · Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213874 · Cited by 182 · cited 1×
“Like Bismarck [35], NOSCOPE is structured as a sequence of dataflow operators, but for performing video classification.”
MethodOperator scheduling in data stream systems
Babcock, B., Babu, S., Datar, M., Motwani, R., & Thomas, D. (2004) · The VLDB Journal · doi:10.1007/s00778-004-0132-6 · Cited by 161
Ten Years of Pedestrian Detection, What Have We Learned?Preprint
Benenson, R., Omran, M., Hosang, J., & Schiele, B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4304 · Cited by 141 · cited 1×
“One of the most popular video-based object detection tasks is pedestrian detection [15, 28], with a range of NN-based approaches (e.”
BackgroundMacroBase: Prioritizing Attention in Fast Data
Bailis, P. D., Gan, E., Madden, S. R., Narayanan, D., Rong, K., & Suri, S. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3035928 · Cited by 135 · cited 3×
“Velox [25] provides a high-performance platform for serving personalized models at scale, while MLBase [54] provides a declarative interface for specifying modeling tasks, and MacroBase [12, 13] provides a platform for executing classification and data aggregation tasks on fast data.”
BackgroundSelf-citationEfficient region-based motion segmentation for a video monitoring system
Kim, J. B., & Kim, H. J. (2002) · Pattern Recognition Letters · doi:10.1016/s0167-8655(02)00194-0 · Cited by 131 · cited 1×
“Video monitoring captures a variety of tasks [52], ranging from object detection [53] to vehicle tracking [87].”
BackgroundVideo data mining: semantic indexing and event detection from the association perspective
Xingquan Zhu, A., Wu, X., Elmagarmid, A. K., Feng, Z., & Wu, L. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.83 · Cited by 129 · cited 2×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
BackgroundJain, R., & Hampapur, A. (1994) · SGMD · doi:10.1145/190627.190638 · Cited by 126 · cited 2×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
BackgroundDual Deep Network for Visual Tracking
Chi, Z., Li, H., Lu, H., & Yang, M. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2669880 · Cited by 125 · cited 1×
“There are many such methods for tracking, including NN-based methods [24, 71] and traditional methods [27, 30], that all performed well on the VOT2016 challenge [55].”
MethodSeq-NMS for Video Object Detection
Han, W., Khorrami, P., Paine, T. L., Ramachandran, P., Babaeizadeh, M., Shi, H., Li, J., Yan, S., & Huang, T. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08465 · Cited by 123 · cited 1×
“State-ofthe-art methods [43, 50, 51] follow a common pattern: run still image”
MethodVideo processing techniques for traffic flow monitoring: A survey
Tian, B., Yao, Q., Gu, Y., Wang, K., & Li, Y. (2011) · 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2011.6083125 · Cited by 106 · cited 1×
“Video monitoring captures a variety of tasks [52], ranging from object detection [53] to vehicle tracking [87].”
BackgroundBello, M. G. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.165589 · Cited by 98 · cited 1×
“paredtoworkinthecomputervision literature that uses the same cascade of filters (or that optimize solely foraccuracy,notcomputationalcost;cf.[89]). Inthisway, NOSCOPE acts as a system for model search [14,23,102] for efficient inference and is inspired by conventional database cost optimization [83] and related efforts…”
MethodExploiting Local Features from Deep Networks for Image RetrievalPreprint
Ng, J. Y.-H., Yang, F., & Davis, L. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.05133 · Cited by 95 · cited 1×
“The second class of techniques allows retrieval of similar images from a large corpus that are similar to a query image [61,96] (e.”
BackgroundJACOB: just a content-based query system for video databases
Cascia, M. L., & Ardizzone, E. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543585 · Cited by 86
Crankshaw, D., Bailis, P. D., Gonzalez, J. E., Li, H., Zhang, Z., Franklin, M. J. M., Ghodsi, A., & Jordan, M. I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.3809 · Cited by 84 · cited 1×
“Velox [25] provides a high-performance platform for serving personalized models at scale, while MLBase [54] provides a declarative interface for specifying modeling tasks, and MacroBase [12, 13] provides a platform for executing classification and data aggregation tasks on fast data.”
BackgroundSelf-citationScene change detection techniques for video database systems
Jiang, H., Helal, S., Elmagarmid, A. K., & Joshi, A. (1998) · Multimedia Systems · doi:10.1007/s005300050087 · Cited by 82
Optimization of continuous queries with shared expensive filters
Munagala, K., Srivastava, U., & Widom, J. (2007) · ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/1265530.1265561 · Cited by 65 · cited 1×
“Alternatively, a system monitoring 100 streams might be able to batch together data from different streams, applying different NN models to each one, allowing a range of optimizations [69] we have not yet explored in the context of streaming video.”
BackgroundSTRG-Index: spatio-temporal region graph indexing for large video databases
Lee, J., Oh, J., & Hwang, S. (2005) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1066157.1066239 · Cited by 64 · cited 2×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
BackgroundOnline Deformable Object Tracking Based on Structure-Aware Hyper-Graph
Du, D., Qi, H., Li, W., Wen, L., Huang, Q., & Lyu, S. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2570556 · Cited by 58 · cited 1×
“There are many such methods for tracking, including NN-based methods [24, 71] and traditional methods [27, 30], that all performed well on the VOT2016 challenge [55].”
MethodAudio/video databases: an object-oriented approach
Gibbs, S. J., Breiteneder, C. J., & Tsichritzis, D. (2002) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.1993.344043 · Cited by 54 · cited 2×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
BackgroundVerschae, R., Ruíz-del-Solar, J., & Correa, M. (2007) · Machine Vision and Applications · doi:10.1007/s00138-007-0084-0 · Cited by 49
Papert, S. A. (1966) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 44
Efficient and cost-effective techniques for browsing and indexing large video databases
Oh, J., & Hua, K. A. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335436 · Cited by 39
WaveScope: a signal-oriented data stream management system
Girod, L., Jamieson, K., Mei, Y., Newton, R., Rost, S., Thiagarajan, A., Balakrishnan, H., & Madden, S. R. (2006) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/1182807.1182886 · Cited by 25 · cited 1×
“Similarly, a range of systems from academia and industry provide stream processing functionality [4, 19, 26, 39]; here, we specifically focus on video stream processing.”
BackgroundTechnical Perspective: What led computer vision to deep learning?
Malik, J. (2017) · Communications of the ACM · doi:10.1145/3065384 · Cited by 24 · cited 1×
“In recent times, however, deep neural networks (NNs) [40, 44, 58, 66, 67] have largely displaced classical computer”
BackgroundVideo query processing in the VDBMS testbed for video database research
Aref, W. G., Hammad, M. A., Catlin, A. C., Ilyas, I. F., Ghanem, T. M., Elmagarmid, A. K., & Marzouk, M. (2003) · ACM International Workshop On Multimedia Databases · doi:10.1145/951676.951682 · Cited by 21 · cited 2×
“Data management researchers have long recognized the potential of computer vision techniques for organizing and querying visual data—starting with early systems such as Chabot [72] and QBIC [36], and followed by a range of “multimedia” databases for storing [7, 59], querying [6, 57, 73, 101], and managing [38, 48, 94]…”
BackgroundFaster R-CNN Scene Specialization with a Sequential Monte-Carlo Framework
Mhalla, A., Maâmatou, H., Château, T., Gazzah, S., & Amara, N. E. B. (2016) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2016.7797042 · Cited by 19 · cited 1×
“However, related strategies for specialization have been shown to boost accuracy (but not runtime) [65, 68], and there are a range of models designed for detecting specific classes of objects in video.”
BackgroundSequential Monte Carlo filter based on multiple strategies for a scene specialization classifier
Maâmatou, H., Château, T., Gazzah, S., Goyat, Y., & Amara, N. E. B. (2016) · EURASIP Journal on Image and Video Processing · doi:10.1186/s13640-016-0143-4 · Cited by 7 · cited 1×
“However, related strategies for specialization have been shown to boost accuracy (but not runtime) [65, 68], and there are a range of models designed for detecting specific classes of objects in video.”
BackgroundPrioritizing Attention in Analytic Monitoring.
Bailis, P. D., Gan, E., Rong, K., & Suri, S. (2017) · Conference on Innovative Data Systems Research · Cited by 5
Self-citation
Research works printed but not linked: 24
- Abadi D. J., 2005, CIDR
- Ba J., 2014, NIPS
- Bailis P., 2017, CIDR
- Benenson R., 2014, ECCV
- Chandrasekaran S., 2003, CIDR
- Chaudhuri S., 2007, VLDB
- Chen W., 2015, ICML
- Feichtenhofer C., 2016, NIPS
- 10.5555/2354409.2354978 · doi:10.5555/2354409.2354978
- Gibbs S., 1993, ICDE
- Goodfellow I., 2016, MIT Press
- Han S., 2016, ICLR
- Hinton G., 2014, NIPS
- Idreos S., 2007, CIDR
- Kraska T., 2013, CIDR
- Krizhevsky A., 2012, NIPS
- 10.5555/850924.851523 · doi:10.5555/850924.851523
- Metz C., 2017, Wired
- Patait A., 2016, 2016 GPU Technology Conference (https://goo.gl/Bdjdgm)
- Ren S., 2015, NIPS
- Yue-Hei Ng J., 2015, CVPR, 53
- Zhang H., 2017, NSDI
- Zhanget S., 2016, CVPR
- Zoph B., 2017, ICLR
Cited by
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
“NoScope [144] leverages two types of models rather than the standard model (such as YOLO”
MethodChameleon: scalable adaptation of video analytics
Jiang, J., Ananthanarayanan, G., Bodik, P., Sen, S., & Stoica, I. (2018) · Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3230543.3230574 · Cited by 467
“Instead, we use an expensive golden configuration to provide the “ground truth”, following prior work [22, 24, 32].”
BackgroundMethodEfficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
SPINN: synergistic progressive inference of neural networks over device and cloud
Laskaridis, S., Venieris, S. I., Almeida, M., Leontiadis, I., & Lane, N. D. (2020) · Proceedings of the 26th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3372224.3419194 · Cited by 310
Metadata
- Authors
- Kang, D., Emmons, J., Abuzaid, F., Bailis, P., & Zaharia, M.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2017-08
- Volume
- 10
- Issue
- 11
- Pages
- 1586-1597
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W2752236330
- Cited by
- 367
- References
- 97
- Citations by year
- 2026: 23 · 2025: 33 · 2024: 35 · 2023: 40 · 2022: 60 · 2021: 49 · 2020: 45 · 2019: 53 · 2018: 28 · 2017: 1
- FWCI
- 14.42
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- Locations
- Proceedings of the VLDB Endowment (publishedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Keywords
- object detection, inference optimization, model search, cascade model, cost-based optimization, real-time video analysis, surveillance video, webcam video, binary classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.