Attention-Based Deep Metric Learning for Near-Duplicate Video Retrieval
Abstract
Near-duplicate video retrieval (NDVR) is an important and challenging problem due to the increasing amount of videos uploaded to the Internet. In this paper, we propose an attention-based deep metric learning method for NDVR. Our method is based on well-established principles: We leverage two-stream networks to combine RGB and optical flow features, and incorporate an attention module to effectively deal with distractor frames commonly observed in near duplicate videos. We further aggregate the features corresponding to multiple video segments to enhance the discriminative power. The whole system is trained using a deep metric learning objective with a Siamese architecture. Our experiments show that the attention module helps eliminate redundant and noisy frames, while focusing on visually relevant frames for solving NVDR. We evaluate our approach on recent large-scale NDVR datasets, CC_WEB_VIDEO, VCDB, FIVR and SVD. To demonstrate the generalization ability of our approach, we report results in both within- and cross-dataset settings, and show that the proposed method significantly outperforms state-of-the-art approaches.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The proposed scheme achieves superior copy detection performance compared with existing schemes.
“Extensive experiments demonstrate that the proposed scheme achieves superior performance of copy detection compared with existing schemes and performs well even in the case of untrained manipulations.”
From Perceptual Robust Hashing for Video Copy Detection with Unsupervised Learning (2023) · cites this paper as [24]
The proposed scheme achieves the best overall performance with respect to perceptual robustness and discrimination among the compared schemes.
“As shown in Fig. 5, the ROC curve of our scheme is the closest to the upper left corner, which means our scheme achieves the best overall performance with respect to perceptual robustness and discrimination.”
From Perceptual Robust Hashing for Video Copy Detection with Unsupervised Learning (2023) · cites this paper as [24]
For most manipulations, the F1 scores of the proposed scheme are greater than those of the other compared schemes.
“We can find that, for most manipulations, the 𝐹 1 scores of our scheme are greater than those of other compared schemes.”
From Perceptual Robust Hashing for Video Copy Detection with Unsupervised Learning (2023) · cites this paper as [24]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · see citing works
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
A comparative study of texture measures with classification based on featured distributions
Ojala, T., Pietikäinen, M., & Harwood, D. (1996) · Pattern Recognition · doi:10.1016/0031-3203(95)00067-4 · Cited by 6,882 · see citing works
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,164 · see citing works
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.179 · Cited by 3,177 · see citing works
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842 · see citing works
Evaluation campaigns and TRECVid
Smeaton, A. F., Over, P., & Kraaij, W. (2006) · doi:10.1145/1178677.1178722 · Cited by 1,227 · see citing works
Description of interest regions with local binary patterns
Heikkilä, M., Pietikäinen, M., & Schmid, C. (2008) · Pattern Recognition · doi:10.1016/j.patcog.2008.08.014 · Cited by 1,213 · see citing works
Sampling Matters in Deep Embedding Learning
Wu, C.-Y., Manmatha, R., Smola, A. J., & Krähenbühl, P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.309 · Cited by 880 · see citing works
Large-Scale Image Retrieval with Attentive Deep Local Features
Noh, H., Araújo, A. D. M., Sim, J., Weyand, T., & Han, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.374 · Cited by 851 · see citing works
Unifying Deep Local and Global Features for Image Search
Cao, B., Araújo, A. D. M., & Sim, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58565-5_43 · Cited by 393 · see citing works
Practical elimination of near-duplicates from web video search
Wu, X., Hauptmann, A. G., & Ngo, C.-W. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291280 · Cited by 340 · see citing works
Multiple feature hashing for real-time large scale near-duplicate video retrieval
Song, J., Yang, Y., Huang, Z., Shen, H. T., & Hong, R. (2011) · Proceedings of the 19th ACM international conference on Multimedia · doi:10.1145/2072298.2072354 · Cited by 334 · see citing works
Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification
Long, X., Gan, C., de Melo, G., Wu, J., Liu, X., & Wen, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00817 · Cited by 233 · see citing works
An Image-Based Approach to Video Copy Detection With Spatio-Temporal Post-Filtering
Douze, M., Jeǵou, H., & Schmid, C. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2046265 · Cited by 165 · see citing works
A Robust and Fast Video Copy Detection System Using Content-Based Fingerprinting
Esmaeili, M. M., Fatourechi, M., & Ward, R. K. (2010) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2010.2097593 · Cited by 153 · see citing works
Scalable detection of partial near-duplicate videos by visual-temporal consistency
Tan, H.-K., Ngo, C.-W., Hong, R., & Chua, T.-S. (2009) · Proceedings of the 17th ACM international conference on Multimedia · doi:10.1145/1631272.1631295 · Cited by 110 · see citing works
Near-Duplicate Video Retrieval with Deep Metric Learning
Kordopatis-Zilos, G., Papadopoulos, S., Patras, I., & Kompatsiaris, Y. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.49 · Cited by 87 · see citing works
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
Kordopatis-Zilos, G., Papadopoulos, S., Patras, I., & Kompatsiaris, Y. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-51811-4_21 · Cited by 83 · see citing works
VCDB: A Large-Scale Database for Partial Copy Detection in Videos
Jiang, Y.-G., Jiang, Y., & Wang, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_24 · Cited by 82 · cited 7× · see citing works
“To demonstrate the generalization ability of our method to different datasets, we conduct cross-dataset evaluation, where we train a model on one dataset and test it on another. a) VCDB [3] : We train our model on FIVR-200K and test it on VCDB.”
BackgroundMethodResultInfluentialFIVR: Fine-Grained Incident Video Retrieval
Kordopatis-Zilos, G., Papadopoulos, S., Patras, I., & Kompatsiaris, I. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2905741 · Cited by 78 · cited 5× · see citing works
“[37] : We split the FIVR-200K dataset into train and test splits, according to [37], and train/test our model on them.”
MethodInfluentialPartial Copy Detection in Videos: A Benchmark and an Evaluation of Popular Methods
Jiang, Y.-G., & Wang, J. (2016) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2016.2530714 · Cited by 72 · see citing works
Frame Fusion for Video Copy Detection
Wei, S., Zhao, Y., Zhu, C., Xu, C., & Zhu, Z. (2011) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2011.2105554 · Cited by 68 · see citing works
Million-scale near-duplicate video retrieval system
Cai, Y., Yang, L., Ping, W., Wang, F., Mei, T., Hua, X., & Li, S. (2011) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2072298.2072484 · Cited by 54 · see citing works
LAMV: Learning to Align and Match Videos with Kernelized Temporal Layers
Baraldi, L., Douze, M., Cucchiara, R., & Jeǵou, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00814 · Cited by 54 · see citing works
Compact Video Description for Copy Detection with Precise Temporal Alignment
Douze, M., Jégou, H., Schmid, C., & Pérez, P. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_38 · Cited by 53 · see citing works
SVD: A Large-Scale Short Video Dataset for Near-Duplicate Video Retrieval
Jiang, Q.-Y., He, Y., Li, G., Lin, J., Li, L., & Li, W.-J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00538 · Cited by 49 · see citing works
Large-scale video copy retrieval with temporal-concentration SIFT
Zhu, Y., Huang, X., Huang, Q., & Tian, Q. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.09.114 · Cited by 45 · see citing works
Compact CNN Based Video Representation for Efficient Video Copy Detection
Wang, L. E., Bao, Y., Li, H., Fan, X., & Luo, Z. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-51811-4_47 · Cited by 37 · see citing works
IBM Research TRECVID-2010 Video Copy Detection and Multimedia Event Detection System
Hill, M., Hua, G., Natsev, A., Smith, J. R., Xie, L., Huang, B., Merler, M., Ouyang, H., & Zhou, M. (2010) · Cited by 27 · see citing works
Effective real-scenario Video Copy Detection
Zhang, Y., & Zhang, X. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7900252 · Cited by 22 · see citing works
Partial-copy detection of non-simulated videos using learning at decision level
Guzman-Zavaleta, Z. J., & Feregrino-Uribe, C. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6345-2 · Cited by 14 · see citing works
Multi-Scale Context Attention Network for Image Retrieval
Lou, Y., Bai, Y., Wang, S., & Duan, L. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240602 · Cited by 13 · see citing works
Fast and accurate content-based video copy detection using bag-of-global visual features
Uchida, Y., Takagi, K., & Sakazawa, S. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288061 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval
Kordopatis-Zilos, G., Tzelepis, C., Papadopoulos, S., Kompatsiaris, I. Y., & Patras, I. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01651-3 · Cited by 37 · see citing works
A multi-embedding neural model for incident video retrieval
Chiang, T.-H., Tseng, Y., & Tseng, Y. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108807 · Cited by 24 · see citing works
Self-Supervised Video Similarity Learning
Kordopatis-Zilos, G., Tolias, G., Tzelepis, C., Kompatsiaris, I. Y., Patras, I., & Papadopoulos, S. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00504 · Cited by 20 · see citing works
Content based video retrieval system using two stream convolutional neural network
Sowmyayani, S., & Rani, P. A. J. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14784-5 · Cited by 17 · see citing works
The 2023 video similarity dataset and challenge
Pizzi, E., Kordopatis-Zilos, G., Patel, H., Postelnicu, G. M., Ravindra, S. N., Gupta, A., Papadopoulos, S., Tolias, G., & Douze, M. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.103997 · Cited by 12 · see citing works
Cross-ViT: Cross-attention Vision Transformer for Image Duplicate Detection
Chandrasiri, M. D. N., & Talagala, P. D. (2023) · International Conference on Information Technology Research (ICITR) · doi:10.1109/icitr61062.2023.10382916 · Cited by 6 · see citing works
The 2023 Video Similarity Dataset and ChallengePreprint
Pizzi, E., Kordopatis-Zilos, G., Patel, H., Postelnicua, G., Ravindra, S. N., Gupta, A., Papadopoulos, S., Tolias, G., & Douze, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4597445 · Cited by 5 · see citing works
An Automatic Near-Duplicate Video Data Cleaning Method Based on a Consistent Feature Hash Ring
Qin, Y., Ye, O., & Fu, Y. (2024) · Electronics · doi:10.3390/electronics13081522 · Cited by 4 · see citing works
Learning Multi-scale Representations with Single-stream Network for Video Retrieval
Wang, C.-H., Tseng, Y., Chiang, T.-H., & Chen, Y.-A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00656 · Cited by 4 · see citing works
Self-citationiESTA: Instance-Enhanced Spatial–Temporal Alignment for Video Copy Localization
Ding, X., Lou, J., Luo, W., Liu, Y., Li, B., & Hu, W. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3517664 · Cited by 2 · see citing works
STAR-GNN: Spatial-Temporal Video Representation for Content-Based Retrieval
Zhao, G., Zhang, B., Zhang, M., Li, Y., Liu, J., & Wen, J.-R. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859598 · Cited by 2 · see citing works
Perceptual Robust Hashing for Video Copy Detection with Unsupervised Learning
Zhao, G., Qin, C., Luo, X., Zhang, X., & Chang, C.-C. (2023) · Proceedings of the 2023 ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3577163.3595110 · Cited by 1 · see citing works
“Note that, the lengths of hash code for [23], [24], [25] and [15] were defaultly set as 96 bits, 100 bits, 256 bits and 75 bits, respectively, and our scheme was 64 bits.”
MethodFrame by Familiar Frame: Understanding Replication in Video Diffusion Models
Rahman, A., Perera, M. V., & Patel, V. M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00274 · Cited by 5 · see citing works
Application of Perceptual Video Hashing for Near-duplicate Video Retrieval
Sandeep, R., & Prabin, B. K. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-16-9605-3_18 · Cited by 1 · see citing works
FAVT: A Feature-Anchored Video Data Trading Mechanism
Zhou, J., Wang, N., Liu, A., & Du, X. (2022) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc55868.2022.00044
Zhao, G., Li, F., Yao, H., & Qin, C. (2023) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2023.103913
Performance Characterization of 2D CNN Features for Partial Video Copy Detection
Le, V.-H., Delalandre, M., & Cardot, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44237-7_20
DRM-SN: Detecting Reused Multimedia Content on Social Networks
Li, W.-S., Lü, Y., Hsiao, W.-K., Tseng, Y.-Y., & Wang, M.-H. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00033
Balancing Efficiency and Accuracy: An Analysis of Sampling for Video Copy Detection
Chen, X., & Satoh, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2054-8_9
- Authors
- Wang, K.-H., Cheng, C.-C., Chen, Y.-L., Song, Y., & Lai, S.-H.
- Affiliations
- National Tsing Hua University (TW)
- Microsoft (United States) (US)
- Venue
- 2020 25th International Conference on Pattern Recognition (ICPR)
- Publisher
- IEEE
- Published
- 2021-01-10
- Pages
- 5360-5367
- ISSN
- 1041-3278 (ISSN-L)
- OpenAlex
- W3126370457
- Semantic Scholar
- 5b8e1808b36e1341c30f61b22d72d2bed28d0067
- Cited by
- 19
- References
- 40
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 8 · 2022: 5
- FWCI
- 1.39
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Keywords
- near-duplicate video retrieval, attention mechanism, deep metric learning, two-stream network, Siamese network, cross-dataset generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
