Flow-Guided Feature Aggregation for Video Object Detection
Open accessAbstract
Extending state-of-the-art object detectors from image to video is challenging. The accuracy of detection suffers from degenerated object appearances in videos, e.g., motion blur, video defocus, rare poses, etc. Existing work attempts to exploit temporal information on box level, but such methods are not trained end-to-end. We present flow-guided feature aggregation, an accurate and end-to-end learning framework for video object detection. It leverages temporal coherence on feature level instead. It improves the per-frame features by aggregation of nearby features along the motion paths, and thus improves the video recognition accuracy. Our method significantly improves upon strong singleframe baselines in ImageNet VID [33], especially for more challenging fast moving objects. Our framework is principled, and on par with the best engineered systems winning the ImageNet VID challenges 2016, without additional bells-and-whistles. The code would be released.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Overlooking spatial and temporal correlation of frames, motion blur, occlusion, and posture changes results in a decrease in object detection performance in videos.
“Indeed, overlooking these factors results in a decrease in the object detection performance in videos.”
From A Systematic Review of Drone Based Road Traffic Monitoring System (2022) · cites this paper as [31]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 11× · see citing works
“The ResNet-50, ResNet-101, and the Aligned-Inception-ResNet models are all pre-trained on ImageNet classification.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 3× · see citing works
“State-of-the-art methods for general object detection [10, 30, 26, 5] are mainly based on deep CNNs [22, 36, 40, 14].”
BackgroundMethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · cited 3× · see citing works
“State-of-the-art methods for general object detection [10, 30, 26, 5] are mainly based on deep CNNs [22, 36, 40, 14].”
BackgroundMethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · cited 5× · see citing works
“In [11], a multistage pipeline called Regions with Convolutional Neural Networks (R-CNN) is proposed for training deep CNN to classify region proposals for object detection.”
BackgroundMethodInfluentialGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · cited 3× · see citing works
“State-of-the-art methods for general object detection [10, 30, 26, 5] are mainly based on deep CNNs [22, 36, 40, 14].”
BackgroundMethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 12× · see citing works
“Our approach is evaluated on the large-scale ImageNet VID dataset [33].”
MethodInfluential(2014) · Cited by 33,419 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · cited 3× · see citing works
“Recent years have witnessed significant progress in object detection [11].”
BackgroundMethod(2016) · Cited by 31,590 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389824 · Cited by 11,971 · see citing works
Horn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 2× · see citing works
“On the other hand, exhaustive spatial-temporal convolution is used to directly extract spatial-temporal features [38, 21, 41, 42].”
MethodDeformable Convolutional Networks
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.89 · Cited by 7,513 · cited 1× · see citing works
“To resolve feature misalignment issue and make it proper for object detection, We utilize a modified version dubbed as “Aligned-Inception-ResNet”, which is described in [6].”
MethodSelf-citationHorn, B. K. P., & Schunck, B. G. (1981) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.965761 · Cited by 7,443 · cited 1× · see citing works
“Traditional methods are dominated by variational approaches [2, 15], which mainly address small displacements [43].”
MethodLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 2× · see citing works
“On the other hand, exhaustive spatial-temporal convolution is used to directly extract spatial-temporal features [38, 21, 41, 42].”
MethodLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
R-FCN: Object Detection via Region-based Fully Convolutional NetworksPreprint
Dai, J., Li, Y., He, K., & Sun, J. Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06409 · Cited by 5,135 · cited 9× · see citing works
“To increase the feature resolution, following the practice in [4, 5], the effective stride of the last block is changed from 32 to 16.”
BackgroundMethodInfluentialSelf-citationInception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · cited 1× · see citing works
“We adopt the state-of-the-art ResNet (-50 and -101) [14] and Inception-Resnet [39] as the feature network.”
MethodFlownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · cited 8× · see citing works
“It takes the feature and the detection sub-networks from the single-frame baseline (a), and the pre-trained off-the-shelf FlowNet.”
BackgroundMethodInfluentialSemantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213 · cited 1× · see citing works
“To increase the feature resolution, following the practice in [4, 5], the effective stride of the last block is changed from 32 to 16.”
MethodFlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.179 · Cited by 3,177 · see citing works
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10578-9_23 · Cited by 3,034 · cited 2× · see citing works
“A shallow detection-specific network [13, 10, 30, 26, 5] then generates the detection results from the feature maps.”
MethodA Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,879 · cited 1× · see citing works
“Equation (2) is similar to the formula of attention models [32], where varying weights are assigned to the features in the memory buffer.”
BackgroundLearning Multi-domain Convolutional Neural Networks for Visual Tracking
Nam, H., & Han, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.465 · Cited by 2,806 · cited 1× · see citing works
“Recently, deep CNNs have been used for object tracking [25, 16] and achieved impressive tracking accuracy.”
MethodHigh Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634 · cited 1× · see citing works
“Traditional methods are dominated by variational approaches [2, 15], which mainly address small displacements [43].”
MethodBeyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,121 · cited 2× · see citing works
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundMethodOptical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.291 · Cited by 1,379 · cited 1× · see citing works
“, FlowNet [8] and its successors [28, 17]) have been exploited for optical flow recently.”
BackgroundLarge Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
Brox, T., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.143 · Cited by 1,295 · cited 1× · see citing works
“The recent focus is on large displacements [3], and combinatorial matching (e.”
BackgroundDescribing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035 · cited 2× · see citing works
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundMethodDeepFlow: Large Displacement Optical Flow with Deep Matching
Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.175 · Cited by 1,019 · see citing works
Visual Tracking with Fully Convolutional Networks
Wang, L., Ouyang, W., Wang, X., & Lu, H. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.357 · Cited by 1,001 · cited 1× · see citing works
“Recently, deep CNNs have been used for object tracking [25, 16] and achieved impressive tracking accuracy.”
MethodEpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298720 · Cited by 757 · cited 2× · see citing works
“, DeepFlow [44], EpicFlow [31]) has been integrated into the variational approach.”
MethodDelving Deeper into Convolutional Networks for Learning Video RepresentationsPreprint
Ballas, N., Yao, L., Pal, C., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06432 · Cited by 715 · cited 2× · see citing works
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundMethodDeep Feature Flow for Video Recognition
Zhu, X., Xiong, Y., Dai, J., Yuan, L., & Wei, Y. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.441 · Cited by 678 · cited 4× · see citing works
“As motivated by [49], given a reference frame Ii and a neighbor frame Ij , a flow field Mi→j = F(Ii, Ij) is estimated by a flow network F (e.”
BackgroundMethodInfluentialSelf-citationAction Recognition using Visual AttentionPreprint
Sharma, S., Kiros, R., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04119 · Cited by 593 · cited 2× · see citing works
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundMethodHuman Action Recognition Using Factorized Spatio-Temporal Convolutional Networks
Sun, L., Jia, K., Yeung, D., & Shi, B. E. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.522 · Cited by 580 · cited 2× · see citing works
“On the other hand, exhaustive spatial-temporal convolution is used to directly extract spatial-temporal features [38, 21, 41, 42].”
MethodYouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video
Real, E., Shlens, J., Mazzocchi, S., Pan, X., & Vanhoucke, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.789 · Cited by 580 · see citing works
T-CNN: Tubelets With Convolutional Neural Networks for Object Detection From Videos
Kang, K., Li, H., Yan, J., Zeng, X., Yang, B., Xiao, T., Zhang, C., Wang, Z., Wang, R., Wang, X., et al. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2736553 · Cited by 554 · cited 8× · see citing works
“Following the protocols in [18, 23], model training and evaluation are performed on the 3,862 video snippets from the training set and the 555 snippets from the validation set, respectively.”
BackgroundMethodInfluentialVideoLSTM convolves, attends and flows for action recognition
Li, Z., Gavrilyuk, K., Gavves, E., Jain, M., & Snoek, C. G. M. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.10.011 · Cited by 485 · cited 2× · see citing works
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundMethodObject Detection from Video Tubelets with Convolutional Neural Networks
Kang, K., Ouyang, W., Li, H., & Wang, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.95 · Cited by 386 · cited 2× · see citing works
“Such temporal information is exploited in existing video object detection methods [18, 19, 12, 23] in a simple way.”
BackgroundMethodSeq-NMS for Video Object DetectionPreprint
Han, W., Khorrami, P., Paine, T. L., Ramachandran, P., Babaeizadeh, M., Shi, H., Li, J., Yan, S., & Huang, T. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08465 · Cited by 305 · cited 9× · see citing works
“In particular, we tested three prevalent techniques, namely, motion guided propagation (MGP) [18], Tubelet rescoring [18], and Seq-NMS [12].”
BackgroundMethodInfluentialCosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
Luo, C., Zhan, J., Xue, X., Wang, L., Ren, R., & Yang, Q. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01418-6_38 · Cited by 219 · cited 1× · see citing works
“Here, we use the cosine similarity metric [27] to measure the similarity between the warped features and the features extracted from the reference frame.”
MethodKar, A., Rai, N., Sikka, K., & Sharma, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.604 · Cited by 170 · cited 1× · see citing works
“Feature aggregation is widely used in action recognition [34, 20, 24, 47, 38, 1, 21, 41] and video description [7, 46].”
MethodMulti-class Multi-object Tracking Using Changing Point Detection
Lee, B., Erdenee, E., Jin, S., Nam, M. Y., Jung, Y. G., & Rhee, P. K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-48881-3_6 · Cited by 151 · cited 4× · see citing works
“Following the protocols in [18, 23], model training and evaluation are performed on the 3,862 video snippets from the training set and the 555 snippets from the validation set, respectively.”
BackgroundMethodInfluentialDeep End2End Voxel2Voxel Prediction
Tran, D., Bourdev, L. D., Fergus, R., Torresani, L., & Paluri, M. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.57 · Cited by 108 · cited 1× · see citing works
“On the other hand, exhaustive spatial-temporal convolution is used to directly extract spatial-temporal features [38, 21, 41, 42].”
MethodConvolutional gated recurrent networks for video segmentation
Siam, M., Valipour, S., Jägersand, M., & Ray, N. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296851 · Cited by 107 · cited 1× · see citing works
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundA Survey on Variational Optic Flow Methods for Small Displacements
Weickert, J., Bruhn, A., Brox, T., et al. (2006) · Mathematics in industry · doi:10.1007/978-3-540-34767-5_5 · Cited by 91 · see citing works
STFCN: Spatio-Temporal FCN for Semantic Video SegmentationPreprint
Fayyaz, M., Saffar, M. H., Sabokrou, M., Fathy, M., Klette, R., & Huang, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.05971 · Cited by 45 · cited 1× · see citing works
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundOn The Stability of Video Detection and TrackingPreprint
Zhang, H., & Wang, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06467 · Cited by 16 · cited 1× · see citing works
“Comparison with state-of-the-art systems Unlike image object detection, the area of video object detection lacks principled metrics [48] and guidelines for evaluation and comparison.”
Background
Flagged first, then by citations
3 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- yang, 2016, Efficient object detection from videos
- tran, 2016, Deep end2end voxe12voxel prediction, CVPR Workshop
Cited by
Sorted by citations
Simple Baselines for Human Pose Estimation and Tracking
Xiao, B., Wu, H., & Wei, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_29 · Cited by 2,233 · see citing works
“d occlusion introduced by video frames. As shown in Fig. 2(c), the detector misses the left black person due to fast motion. Temporal information is often leveraged to generate more robust detections [36,35]. We propose to generate boxes for the processing frame from nearby frames using temporal information expressed…”
BackgroundSelf-citationTSM: Temporal Shift Module for Efficient Video Understanding
Lin, J., Gan, C., & Han, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00718 · Cited by 2,152 · see citing works
“We also compare to a strong baseline FGFA [60] that uses optical flow to aggregate the temporal information from 21 frames (past 10 frames and future 10 frames) for offline video detection.”
MethodZhou, X., Koltun, V., & Krähenbühl, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_28 · Cited by 1,428 · see citing works
“posals using bounding box regression. In video object detection, Kang et al. [16,17] feed stacked consecutive frames into the network and do object detection for a whole video segment. And Zhu et al. [59] use flow to warp intermediate features from previous frames to accelerate inference. Our method belongs to this cat…”
MethodVideo Enhancement with Task-Oriented Flow
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-01144-2 · Cited by 1,401 · see citing works
“…(Liu et al 2017), light-field interpolation (Wang et al 2017), novel view synthesis (Zhou et al 2016), eye gaze manipulation (Ganin et al 2016), object detection (Zhu et al 2017), denoising (Wen et al 2017), and super-resolution (Caballero et al 2017; Tao et al 2017; Makansi et al 2017).”
A Survey of Deep Learning-Based Object Detection
Jiao, L., Zhang, F., Liu, F., Yang, S., Li, L., Feng, Z., & Qu, R. (2019) · IEEE Access · doi:10.1109/access.2019.2939201 · Cited by 1,392 · see citing works
“Video detectors [266][267][268][269][270][271][272][273][274][275] exploit temporal contexts to meet this challenge.”
BackgroundDetection and tracking meet drones challenge
Zhu, P., Wen, L., Du, D., Bian, X., Fan, H., Hu, Q., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3119563 · Cited by 1,336 · see citing works
Video Enhancement with Task-Oriented FlowPreprint
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1,163 · see citing works
A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit
Padilla, R., Passos, W. L., Dias, T. L. B., Netto, S. L., & da Silva, E. A. B. (2021) · Electronics · doi:10.3390/electronics10030279 · Cited by 735 · see citing works
“Many works in the literature use the mAP as a metric to evaluate the performance of object detection models in video sequences [92–98].”
BackgroundComputer Vision for Autonomous Vehicles
Janai, J., Güney, F., Behl, A., & Geiger, A. (2020) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000079 · Cited by 664 · see citing works
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
Chan, K. C. K., Wang, X., Yu, K., Dong, C., & Loy, C. C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00491 · Cited by 626 · see citing works
“In this work, we conduct extensive analysis and present a simple, strong and versatile baseline, BasicVSR, which can serve as a back-bone with abundant flexibilities in design.”
MethodA review of object detection based on deep learning
Xiao, Y., Tian, Z., Yu, J., Zhang, Y., Liu, S., Du, S., & Lan, X. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-08976-6 · Cited by 584 · see citing works
“achieved good results by using optical flow to improve the speed and accuracy of video recognition [292, 293].”
MethodYang, L., Fan, Y., & Xu, N. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00529 · Cited by 505 · see citing works
“While object identity information are often utilized for improve robustness of detection algorithms [9, 36, 33], the evaluation metric is limited to per-frame detection and does not require joint object detection and tracking.”
BackgroundA comprehensive review of object detection with deep learning
Kaur, R., & Singh, S. (2023) · Digital Signal Processing · doi:10.1016/j.dsp.2022.103812 · Cited by 539 · see citing works
An Empirical Study of Spatial Attention Mechanisms in Deep Networks
Zhu, X., Cheng, D., Zhang, Z., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00679 · Cited by 659 · see citing works
“Motivated by their success in NLP tasks [2, 28, 14, 6, 26, 30, 31, 36], attention mechanisms have also been employed in computer vision applications such as relational reasoning among objects [3, 33], image captioning [41], image generation [45, 42], image recognition [20, 38, 17, 22, 46, 13], and video recognition [4…”
BackgroundSelf-citationTransTrack: Multiple Object Tracking with TransformerPreprint
Sun, P., Cao, J., Jiang, Y., Zhang, R., Xie, E., Yuan, Z., Wang, C., & Luo, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15460 · Cited by 640 · see citing works
“In video object detection, FGFA [49] uses optical flow to warp intermediate features from the previous frames to accelerate inference.”
MethodDeep learning for unmanned aerial vehicle-based object detection and tracking: a survey
Wu, X., Li, W., Hong, D., Tao, R., & Du, Q. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2021.3115137 · Cited by 421 · see citing works
Self-Correction for Human Parsing
Li, P., Xu, Y., Wei, Y., & Yang, Y. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3048039 · Cited by 392 · see citing works
Self-citationMemory Enhanced Global-Local Aggregation for Video Object Detection
Chen, Y., Cao, Y., Hu, H., & Wang, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01035 · Cited by 366 · see citing works
Computer vision algorithms and hardware implementations: A survey
Feng, X., Jiang, Y., Yang, X., Du, M., & Li, X. S. (2019) · Integration · doi:10.1016/j.vlsi.2019.07.005 · Cited by 360 · see citing works
Channel Attention Is All You Need for Video Frame Interpolation
Choi, M., Kim, H., Han, B., Xu, N., & Lee, K. M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6693 · Cited by 338 · see citing works
“…classi-fication (Carreira and Zisserman 2017; Simonyan and Zis-serman 2014a; Xie et al. 2018), object detection and segmentation (Cheng et al. 2017; Zhu et al. 2018; 2017), video prediction (Li et al. 2018; Reda et al. 2018), and others (Chen et al. 2017; Lai et al. 2018)—frequently depend on…”
BackgroundOcclusion Aware Unsupervised Learning of Optical Flow
Wang, Y., Yang, Y., Yang, Z., Zhao, L., Wang, P., & Xu, W. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00513 · Cited by 331 · see citing works
“With the accurate optical flow prediction, one could estimate the 3D structure of a scene [18], segment moving objects based on motion cues [38], track objects in a complicated environment [11], and build important visual cues for many high level vision tasks such as video action recognition [45] and video object dete…”
Self-citationEnd-to-End Flow Correlation Tracking with Spatial-Temporal Attention
Zhu, Z. D., Wu, W., Zou, W., & Yan, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00064 · Cited by 321 · see citing works
“DFF [47] and FGFA [46] utilize flow information to speed up vision recognition (segmentation and video detection) and upgrade performance, respectively.”
BackgroundTowards High Performance Video Object Detection
Zhu, X., Dai, J., Yuan, L., & Wei, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00753 · Cited by 285 · see citing works
“[36] improves feature quality by adaptive aggregation, which increases computation.”
BackgroundMethodInfluentialSelf-citationSequence Level Semantics Aggregation for Video Object Detection
Wu, H., Chen, Y., Wang, N., & Zhang, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00931 · Cited by 282 · see citing works
New Generation Deep Learning for Video Object Detection: A Survey
Jiao, L., Zhang, R., Liu, F., Yang, S., Hou, B., Li, L., & Tang, X. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3053249 · Cited by 267 · see citing works
Object Detection in Video with Spatiotemporal Sampling Networks
Bertasius, G., Torresani, L., & Shi, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_21 · Cited by 258 · see citing works
“For example, we note that it would take about four days to train an optical flow CNN of FGFA [24] from scratch and then four additional days to train FGFA [24] for video object detection, making it eight days of total training time.”
BackgroundMethodInfluentialLi, D., Wang, R., Xie, C., Liu, L., Zhang, J., Li, R., Wang, F., Zhou, M., & Liu, W. (2020) · Sensors · doi:10.3390/s20030578 · Cited by 259 · see citing works
“…video detection is di ff erent from still-image detection in many ways, and often remains challenges such as video defocus, motion blur and part occlusion [29] The video frames of rice diseases and pests symptoms which contain video defocus, motion blur and part occlusion are shown in Figure 2.”
BackgroundRelation Distillation Networks for Video Object Detection
Deng, J., Pan, Y., Yao, T., Zhou, W., Li, H., & Mei, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00712 · Cited by 242 · see citing works
“Following the common protocols in [8, 46, 49, 54], we utilize both ImageNet VID and ImageNet object detection (DET) dataset to train our RDN.”
BackgroundMethodInfluentialCross View Fusion for 3D Human Pose Estimation
Qiu, H., Wang, C., Wang, J., Wang, N., & Zeng, W. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00444 · Cited by 241 · see citing works
“For example, in [34], Zhu et al . propose to warp the features of the neighboring frames (in a video sequence) to the current frame according to optical flow in order to robustly detect the objects.”
Instance-Aware, Context-Focused, and Memory-Efficient Weakly Supervised Object Detection
Ren, Z., Yu, Z., Yang, X., Liu, M.-Y., Lee, Y. J., Schwing, A. G., & Kautz, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01061 · Cited by 230 · see citing works
EEMEFN: Low-Light Image Enhancement via Edge-Enhanced Multi-Exposure Fusion Network
Zhu, M., Pan, P., Chen, W., et al. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7013 · Cited by 230 · see citing works
“The low-quality images may significantly reduce the performance of computer vision methods (Zhu et al. 2017; Long, Shelhamer, and Darrell 2015), which rely”
BackgroundRetinaTrack: Online Single Stage Joint Detection and Tracking
Lu, Z., Rathod, V., Votel, R., & Huang, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01468 · Cited by 254 · see citing works
Transferable Adversarial Attacks for Image and Video Object Detection
Wei, X., Liang, S., Chen, N., & Cao, X. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/134 · Cited by 219 · see citing works
“Compared with the image scenario, video object detection considers the temporal interactions between frames [31, 32].”
BackgroundMethodInfluentialRelationTrack: Relation-Aware Multiple Object Tracking With Decoupled Representation
Yu, E., Li, Z., Han, S., & Wang, H. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3150169 · Cited by 219 · see citing works
“Many attempts have been conducted to overcome the disadvantages brought by implementing two separate sub-models [2], [32], [33], [34], [35].”
BackgroundFully Motion-Aware Network for Video Object Detection
Wang, S., Zhou, Y., Yan, J., & Deng, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_33 · Cited by 213 · see citing works
“As contrast, FGFA [30] attempts to leverage temporal coherence on feature level by aggregating features of nearby frames along the motion paths.”
BackgroundMethodInfluentialTransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers
Zhou, Q., Li, X., He, L., Yang, Y., Cheng, G., Tong, Y., Ma, L., & Tao, D. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3223955 · Cited by 216 · see citing works
Video Object Detection with an Aligned Spatial-Temporal Memory
Xiao, F., & Lee, Y. J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_30 · Cited by 199 · see citing works
“However, [51,50] use optical flow, which needs to be computed either externally e.”
BackgroundMethodInfluentialSong, T., Sun, L., Xie, D., Sun, H., & Pu, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_33 · Cited by 193 · see citing works
“We also list the result of our TLL combined with FGFA [27].”
BackgroundMethodResultInfluentialTF-Blender: Temporal Feature Blender for Video Object Detection
Cui, Y., Yan, L., Cao, Z., & Liu, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00803 · Cited by 191 · see citing works
“Albeit being superior over the prior efforts, FGFA [42] and SELSA [36] encounter several obstacles from achieving optimal performance: 1) They focus on modeling the global relation for every neighboring frame while ignoring the preservation of the local spatial information for aggregation; 2) They…”
BackgroundMethodInfluentialFlow Guided Recurrent Neural Encoder for Video Salient Object Detection
Li, G., Xie, Y., Wei, T., Wang, K., & Lin, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00342 · Cited by 189 · see citing works
“Inspired by [45, 44], we incorporate optical flow to enable feature warping across frames and compensate for the change caused by object motion.”
BackgroundMethodMAST: A Memory-Augmented Self-Supervised Tracker
Lai, Z., Lu, E., & Xie, W. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00651 · Cited by 176 · see citing works
“Memory models have been used for many applications, including reading comprehension [51], summarization [50], tracking[69], video understanding[7], and image and video captioning [70, 74].”
MethodMeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition
Wu, C.-Y., Li, Y., Mangalam, K., Fan, H., Xiong, B., Malik, J., & Feichtenhofer, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01322 · Cited by 176 · see citing works
“, [57]) to parse a video frame-by-frame, to consider longer-term context, most existing works use causal convolutions [6, 10, 36], RNNs [17,46], or feature fusion [8,84].”
BackgroundAction-Stage Emphasized Spatiotemporal VLAD for Video Action Recognition
Tu, Z., Li, H., Zhang, D., Dauwels, J., Li, B., & Yuan, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2890749 · Cited by 168 · see citing works
“In each ActionS segment, after locating the key frame feature map fS Kk , we apply the flow-guided warping strategy of [45] to select the discriminative local features by comparing the similarity between fS Kk and the warped feature maps of other frames.”
BackgroundMethodKim, D., Woo, S., Lee, J., & Kweon, I. S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00988 · Cited by 166 · see citing works
“It combines several existing tasks: video object segmentation [1,3,4,27,30,35,36,39] and video object detection [9,42,43], and aims at simultaneous detection, segmentation, and tracking of instances in videos.”
MethodClassifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation
Bertasius, G., & Torresani, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00976 · Cited by 166 · see citing works
“Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames.”
BackgroundTAO: A Large-Scale Benchmark for Tracking Any Object
Dave, A., Khurana, T., Tokmakov, P., Schmid, C., & Ramanan, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_26 · Cited by 166 · see citing works
Mobile Video Object Detection with Temporally-Aware Feature Maps
Zhu, M., & Liu, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00596 · Cited by 195 · see citing works
“Flowguided Feature Aggregation [42] improves detection accuracy by warping and averaging features from nearby frames with adaptive weighting.”
BackgroundMoNet: Deep Motion Exploitation for Video Object Segmentation
Xiao, H., Feng, J., Lin, G., Liu, Y., & Zhang, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00125 · Cited by 151 · see citing works
“Inspired by the success in video object detection [37, 38], we propose to align the features from adjacent frames It−m and It+k to It by exploiting motion cues, before combining them.”
MethodTowards An End-to-End Framework for Flow-Guided Video Inpainting
Li, Z., Lu, C.-Z., Qin, J., Guo, C., & Cheng, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01704 · Cited by 154 · see citing works
“The motion information across frames well assists many video-related tasks, such as video understanding [3,31], video segmentation [11,47], video object detection [65], depth estimation [18,36], video super-resolution [4, 57], frame interpolation [22, 27], etc.”
BackgroundAnnotating Objects and Relations in User-Generated Videos
Shang, X., Di, D., Xiao, J., Cao, Y., Yang, X., & Chua, T. (2019) · International Conference on Multimedia Retrieval · doi:10.1145/3323873.3325056 · Cited by 149 · see citing works
3D Object Detection From Images for Autonomous Driving: A Survey
Ma, X., Ouyang, W., Simonelli, A., & Ricci, E. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3346386 · Cited by 155 · see citing works
“On the other hand, lots of works had proved the effectiveness of using video data in many tasks, including 2D detection [195], [196], depth estimation [197], [198], and LiDAR-based 3D detection [199], [200].”
3D Object Detection from Images for Autonomous Driving: A Survey
Ma, X., Ouyang, W., Simonelli, A., & Ricci, E. (2022) · arXiv · doi:10.48550/arxiv.2202.02980 · Cited by 147 · see citing works
Kinematic 3D Object Detection in Monocular Video
Brazil, G., Pons-Moll, G., Liu, X., & Schiele, B. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58592-1_9 · Cited by 188 · see citing works
“A common trend in video-based detection is to benefit the accuracy-efficiency trade-off via reducing the frame redundancy [1,26,42,47,48].”
BackgroundA Review of Video Object Detection: Datasets, Metrics and Methods
Zhu, H., Wei, H., Li, B., Yuan, X., & Kehtarnavaz, N. (2020) · Applied Sciences · doi:10.3390/app10217834 · Cited by 150 · see citing works
“Similar 366 to the FGFA method in [22], it relies on the idea that detection on a single frame would have 367 difficulties dealing with noise sources such as motion blur and video defocus.”
BackgroundMethodInfluentialEnable Deep Learning on Mobile Devices: Methods, Systems, and Applications
Cai, H., Lin, J., Lin, Y., Liu, Z., Tang, H., Wang, H., Zhu, L., & Han, S. (2022) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3486618 · Cited by 152 · see citing works
Vision Meets Drones: Past, Present and FuturePreprint
Zhu, P., Wen, L., Du, D., Bian, X., Hu, Q., & Ling, H. (2020) · arXiv (Cornell University) · Cited by 145 · see citing works
VisDrone-VID2019: The Vision Meets Drone Object Detection in Video Challenge Results
Zhu, P., Du, D., Wen, L., Bian, X., Ling, H., Hu, Q., Peng, T., Zheng, J., Wang, X., Zhang, Y., et al. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00031 · Cited by 142 · see citing works
“On the other hand, there are 4 more methods ( i.e. , DetKITSY (A.5), EODST++ (A.7), Libra-HBR (A.11) and Sniper+ (A.12)) that outperforms or are on par with the best baseline method FGFA [43].”
BackgroundMethodInfluentialSelf-citationContext R-CNN: Long Term Temporal Context for Per-Camera Object Detection
Beery, S., Wu, G., Rathod, V., Votel, R., & Huang, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01309 · Cited by 140 · see citing works
“Leading methods have used pixel level flow (or flow-like concepts) to aggregate features [7, 56–58] or used correlation [13] to densely relate features at the current timestep to an adjacent timestep.”
MethodObject Detection Using Deep Learning Methods in Traffic Scenarios
Boukerche, A., & Hou, Z. (2021) · ACM Computing Surveys · doi:10.1145/3434398 · Cited by 130 · see citing works
“Research using optical flow to predict changes between frames include References [53, 141, 165, 167, 168], where References [167, 168] are two basis and the others are improvements or extensions of them.”
BackgroundMethodEnd-to-End United Video Dehazing and Detection
Li, B., Peng, X., Wang, Z., Xu, J., & Feng, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12287 · Cited by 127 · see citing works
“In outdoor surveillance or autonomous driving, object detection from video (Kang et al. 2016; Tripathi et al. 2016; Zhu et al. 2017) is widely desirable, whose performance is known to heavily suffer from the existence of haze.”
BackgroundDoTA: Unsupervised Detection of Traffic Anomaly in Driving Videos
Yao, Y., Wang, X., Xu, M., Pu, Z., Wang, Y., Atkins, E. M., & Crandall, D. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3150763 · Cited by 129 · see citing works
“Other methods [61], [62], [63], [64], [65], [66] use local feature aggregation for detection in a”
MethodSelf-citationEfficient Uncertainty Estimation for Semantic Segmentation in Videos
Huang, P., Hsu, W., Chiu, C.-Y., Wu, T., & Sun, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_32 · Cited by 123 · see citing works
“To enhance the reference feature map, they utilize optical flow network the work of Zhu et al.[35] to estimate the motions between nearby frames and the reference frame.”
MethodFlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation
Shi, X., Huang, Z., Li, D., Zhang, M., Cheung, K. C., See, S. C. W., Qin, H., Dai, J., & Li, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00160 · Cited by 127 · see citing works
Self-citationA Systematic Review of Drone Based Road Traffic Monitoring System
Bisio, I., Garibotto, C., Haleem, H., Lavagetto, F., & Sciarrone, A. (2022) · IEEE Access · doi:10.1109/access.2022.3207282 · Cited by 121 · see citing works
“…the following concepts must be taken into account: the spatial and temporal correlation of frames to overcome feature extraction redundancies, and the effect of motion blur, occlusion, and posture changes, which can contribute to the low quality of some individual frames in a video sequence [31].”
BackgroundTemporal ROI Align for Video Object Recognition
Gong, T., Chen, K., Wang, X., Chu, Q., Zhu, F., Lin, D., Yu, N., & Feng, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16234 · Cited by 118 · see citing works
Optimizing Video Object Detection via a Scale-Time Lattice
Chen, K., Wang, J., Yang, S., Zhang, X., Xiong, Y., Loy, C. C., & Lin, D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00815 · Cited by 120 · see citing works
“Temporal propagation The idea of temporal propagation is previously explored in the video object detection literatures [16, 32, 33].”
BackgroundMethodInfluentialVideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution
Chen, Z., Chen, Y., Liu, J., Xu, X., Goel, V., Wang, Z., Shi, H., & Wang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00209 · Cited by 117 · see citing works
“This introduces challenges in recognition systems such as video object detection [53], and we are still struggling at learning to recognize motion and actions from discrete frames [4, 12].”
Real-time face & eye tracking and blink detection using event cameras
Ryan, C., O’Sullivan, B., Elrasad, A., Cahill, A., Lemley, J., Kielty, P., Posch, C., & Perot, E. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.019 · Cited by 118 · see citing works
“In contrast, feature-level methods integrate image features (Broad et al., 2018; Liu & Zhu, 2018; Zhu et al., 2017).”
MethodObject Detection in Videos by High Quality Object Linking
Tang, P., Wang, C., Wang, X., Liu, W., Zeng, W., & Wang, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2910529 · Cited by 114 · see citing works
“We evaluate the performance on slow, medium, and fast ones which are formed according to their speed as done in [18].”
BackgroundMethodInfluentialEnd-to-End Video Object Detection with Spatial-Temporal Transformers
He, L., Zhou, Q., Li, X., Niu, L., Cheng, G., Li, X., Liu, W., Tong, Y., Ma, L., & Zhang, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475285 · Cited by 111 · see citing works
“Since the ground truth of the official testing set is not publicly available, we follow the widely adopted setting in previous work [11, 50, 53, 60] where we train our models using a combination of ImageNet VID and DET datasets [40] and measure the performance on the validation set using the mean…”
BackgroundMethodInfluentialRecent Advances in Object Detection in the Age of Deep Convolutional Neural NetworksPreprint
Agarwal, S., du Terrail, J. O., & Jurie, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03193 · Cited by 133 · see citing works
“Deep feature flow [485, 486] ran the convolutional subnetwork only on sparse key frames, propagated deep feature maps to other frames via a flow field.”
BackgroundMethodInfluentialObject Guided External Memory Network for Video Object Detection
Deng, H., Yang, H., Song, T., Zhang, Z., Xue, Z., Ma, R., Robertson, N. M., & Guan, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00678 · Cited by 109 · see citing works
“For a fair comparison with the previous work [45, 44, 35, 2, 43], we”
BackgroundMethodResultInfluentialSpSequenceNet: Semantic Segmentation Network on 4D Point Clouds
Shi, H., Lin, G., Wang, H., Hung, T.-Y., & Wang, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00463 · Cited by 106 · see citing works
“Optic flow methods [22, 29] use the nearest pixel from two different frames to generate local optic flow and achieve significant performance.”
MethodSchindler, F., & Steinhage, V. (2021) · Ecological Informatics · doi:10.1016/j.ecoinf.2021.101215 · Cited by 103 · see citing works
A Spatial-Temporal Feature-Based Detection Framework for Infrared Dim Small Target
Du, J., Lu, H., Zhang, L., Hu, M., Chen, S. Q., Deng, Y., Shen, X., & Zhang, Y. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3117131 · Cited by 103 · see citing works
“In flow-guided feature aggregation (FGFA) [32], a framework for VOD is presented.”
BackgroundMethodProgressive Sparse Local Attention for Video Object Detection
Guo, C., Fan, B., Gu, J., Zhang, Q., Xiang, S., Véronique Prinet, A., & Pan, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00401 · Cited by 106 · see citing works
“To boost the performance, MANet [35] and FGFA [42] use the optical flow estimated by FlowNet [8] to aggregate the feature of multiple nearby frames.”
BackgroundMethodLeveraging Long-Range Temporal Relationships Between Proposals for Video Object Detection
Shvets, M., Liu, W., & Berg, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00985 · Cited by 102 · see citing works
“Current state-of-the-art methods for video object detection all leverage pixel-level features [1, 3, 19, 26].”
BackgroundMethodInfluentialVideo object detection for autonomous driving: Motion-aid feature calibration
Liu, D., Cui, Y., Chen, Y., Zhang, J., & Fan, B. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.05.027 · Cited by 95 · see citing works
YOLOV: Making Still Image Object Detectors Great at Video Object Detection
Shi, Y., Wang, N., & Guo, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25320 · Cited by 96 · see citing works
Mobile Video Object Detection with Temporally-Aware Feature MapsPreprint
Liu, M., & Zhu, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06368 · Cited by 92 · see citing works
“Flowguided Feature Aggregation [42] improves detection accuracy by warping and averaging features from nearby frames with adaptive weighting.”
BackgroundMultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning
Chennupati, S., Sistu, G., Yogamani, S. K., & Rawashdeh, S. A. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00159 · Cited by 94 · see citing works
“These features can be extracted from different CNNs operating on different input data [61, 37] or from a CNN operating on different resolutions of input [24].”
BackgroundMining Inter-Video Proposal Relations for Video Object Detection
Han, M., Wang, Y., Chang, X., & Qiao, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_26 · Cited by 89 · see citing works
Self-citationEVA²: Exploiting Temporal Redundancy in Live Computer Vision
Buckler, M., Bedoukian, P., Jayasuriya, S., & Sampson, A. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00051 · Cited by 88 · see citing works
“(2) DFF uses a fixed key frame rate calibrated to the “worst-case scenario” for scene motion.”
BackgroundMethodInfluentialDogfight: Detecting Drones from Drones Videos
Ashraf, M. W., Sultani, W., & Shah, M. A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00699 · Cited by 87 · see citing works
“, [48] proposed to include the feature representation of each frame by aggregation of nearby frames features using optical flow.”
BackgroundMethodResultInfluentialMPPNet: Multi-frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection
Chen, X., Shi, S., Zhu, B., Cheung, K. C., Xu, H., & Li, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_39 · Cited by 89 · see citing works
Chen, S., Ji, L., Zhu, J., et al. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3350024 · Cited by 86 · see citing works
“For the data-driven methods such as 3DCNN [48], DFF [57], FGDA [58], RDN [59], SELSA [16], MEGA [18], TROI [20], ACM [10], TSDF [47], RISTD [49], STFBD [19], ISNet [42], UIUNet [43], SANet [60], Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
MethodKFNet: Learning Temporal Camera Relocalization Using Kalman Filtering
Zhou, L., Luo, Z., Shen, T., Zhang, J., Zhen, M., Yao, Y., Fang, T., & Quan, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00497 · Cited by 84 · see citing works
“Although the flow-guided warping is employed, TPooler and SWeight only achieve marginal improvements over SCoordNet compared with KFNet, which justifies the advantage of the Kalman filtering system.”
BackgroundInfluentialTSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Devices
Lin, J., Gan, C., Wang, K., & Han, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3029799 · Cited by 85 · see citing works
“Other methods on video object detection [71] fuses information along temporal dimension after the 2D feature is extracted by the backbone, which results in high latency, and also loses low-level temporal cues.”
InfluentialVideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation
Shi, X., Huang, Z., Bian, W., Li, D., Zhang, M., Cheung, K. C., See, S. C. W., Qin, H., Dai, J., & Li, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01146 · Cited by 91 · see citing works
“It is widely adopted to tackle various downstream video problems, including video restoration [32, 66, 12, 34, 5, 50, 72, 36], video object detection [78, 61, 79], video synthesis [69, 18, 19, 63, 62] and action recognition [58, 47, 76], providing valuable information on motion and cross-frame correspondence.”
BackgroundSelf-citationVisDrone-VDT2018: The Vision Meets Drone Video Detection and Tracking Challenge Results
Zhu, P., Wen, L., Du, D., Bian, X., Ling, H., Hu, Q., Wu, H., Nie, Q., Cheng, H., Liu, C., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11021-5_29 · Cited by 80 · see citing works
Self-citationRekavandi, A. M., Xu, L., Boussaïd, F., Seghouane, A., Hoefs, S., & Bennamoun, M. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3530678 · Cited by 111 · see citing works
Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching
Huang, X., Xu, J., Tai, Y., & Tang, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00890 · Cited by 76 · see citing works
“However, most of them require optical flows to obtain pixel-wise alignment [64, 63] which can be quite computationally extensive.”
BackgroundTemporally Identity-Aware SSD With Attentional LSTM
Chen, X., Yu, J., & Wu, Z. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2894261 · Cited by 76 · see citing works
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context
Liu, C., Yang, J., Xu, K., Gong, G., & Mu, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01085 · Cited by 75 · see citing works
“Feature-level methods [55, 56, 57] utilizes temporal information via flow-guided feature propagation from previous frames.”
MethodLearning Where to Focus for Efficient Video Object Detection
Jiang, Z., Liu, Y., Yang, C., Liu, J., Gao, P., Zhang, Q., Xiang, S., & Pan, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_2 · Cited by 72 · see citing works
“To improve detection accuracy, different methods [4, 31, 37, 41] have been proposed to aggregate features across frames.”
BackgroundMethodInfluentialAutomated rip current detection with region based convolutional neural networks
de Silva, A., Mori, I., Dusek, G., Davis, J., & Pang, A. (2021) · Coastal Engineering · doi:10.1016/j.coastaleng.2021.103859 · Cited by 70 · see citing works
“These methods have been extended to detect objects in videos [26, 35, 54].”
Fast Object Detection in Compressed Video
Wang, S., Alibaba Group, Lu, H., & Deng, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00720 · Cited by 68 · see citing works
“Since frames may suffer from imaging-related degradations, most previous works [53, 44, 48, 21, 22] focus on improving frame-wise detection results.”
BackgroundMethodInfluentialJoint Anchor-Feature Refinement for Real-Time Accurate Object Detection in Images and Videos
Chen, X., Yu, J., Kong, S., Wu, Z., & Wen, L. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.2980876 · Cited by 69 · see citing works
“To the best of our knowledge, main ideas of temporal detection include i) post-processing [7], ii) tracking-based location [8], [9], iii) feature aggregation with motion information [8], [10]– [13], iv) RNN-based feature propagation [12], [14]–[16], and v) batch-frame processing (i.”
BackgroundA Multitask Benchmark Dataset for Satellite Video: Object Detection, Tracking, and Segmentation
Li, S., Zhou, Z., Zhao, M., Yang, J. B., Guo, W., Lv, Y., Kou, L., Wang, H., & Gu, Y. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3278075 · Cited by 68 · see citing works
“For the video object detection experiments, we selected four representative video object detection methods, namely, DFF [63], FGFA [50], SELSA [64], and Temporal ROI Align [65].”
MethodReal-time Object Detection for Streaming Perception
Yang, J., Liu, S., Li, Z., Li, X., & Sun, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00531 · Cited by 66 · see citing works
“Some recent methods [1, 7, 11, 62] employ attention mechanism, optical flow, and tracking method, aiming to aggregate rich features for the complex video variation, e.”
Method
- Authors
- Zhu, X., Wang, Y., Dai, J., Yuan, L., & Wei, Y.
- Affiliations
- University of Science and Technology of China (CN)
- Microsoft (United States) (US)
- Venue
- 2017 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2017-10
- Pages
- 408-417
- OpenAlex
- W2964286567
- Cited by
- 729
- References
- 53
- Citations by year
- 2026: 48 · 2025: 57 · 2024: 86 · 2023: 91 · 2022: 102 · 2021: 123 · 2020: 96 · 2019: 75 · 2018: 47 · 2017: 4
- FWCI
- 17.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- video object detection, optical flow, temporal feature aggregation, movement paths, end-to-end learning, motion blur, fast-moving objects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
