Partial Copy Detection in Videos: A Benchmark and an Evaluation of Popular Methods
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
The goal of partial video copy detection is to find one or more segments of a query video which have (transformed) copies in a large dataset. Previous related research in this field used either small-scale datasets or large datasets with simulated partial copies by imposing several pre-defined transformations (e.g., photometric changes) due to the extremely time-consuming annotation of real copies. It is still unknown how well the techniques developed on simulated datasets perform on real copies, which are much more challenging and too complex to be simulated. In this paper, we introduce a large-scale video copy database (VCDB) with over 100,000 videos, and more than 9,000 copy pairs found by manual annotation. A state-of-the-art system of video copy detection is evaluated on VCDB to show the limitations of existing techniques. We also evaluate deep learning features learned by two neural networks: one is independently trained on a different dataset and the other is tailored to deal with the copy detection task. Our evaluation suggests that all the existing techniques, including the deep learning features, are far from satisfactory in detecting complex real copies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“progress of deep learning on image classification [33], [34] and video analysis [35], [36].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Works dedicated to entire video-level copy detection used global features like color histogram and local features like the LBP [3], [4], [12].”
BackgroundDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 1×
“In this paper, we introduce a large-scale video copy database (VCDB) with over 100,000 videos, and more than 9,000 copy pairs found by manual annotation.”
MethodRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · cited 3×
“For each sampled video frame, a 4,096-dimensional feature vector is extracted using the Caffe [31] implementation of the AlexNet in [24].”
MethodInfluentialDimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · cited 1×
“The contrastive loss introduced in [32] is used as loss function in the training process.”
MethodProduct Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956 · cited 1×
“Wu et al. [3] created the popularly used CC_Web dataset, which contains 12,790 videos collected by searching videos on Google, YouTube and Yahoo!.”
MethodObject retrieval with large vocabularies and fast spatial matching
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383172 · Cited by 2,929 · cited 1×
“Videos which are created from two sources but contain the same contents, such as the same scene captured by two cameras, are near-duplicate videos by definition, but not copy videos.”
BackgroundAggregating local descriptors into a compact image representation
Jegou, H., Douze, M., Schmid, C., & Perez, P. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540039 · Cited by 2,692 · cited 1×
“Two kinds of queries are defined in the dataset: (1) ST1: entire video copy (normally between 5 minutes and 1 hour), where the videos may be slightly recoded and/or noised; (2) ST2: partial video copy, where two videos only share one or more short segments (between 1 second to 1 minute).”
BackgroundDeep Learning Face Representation from Predicting 10,000 Classes
Sun, Y., Wang, X., & Tang, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.244 · Cited by 1,997 · cited 8×
“The secondmethod is called temporal Hough transform proposed in [2].”
MethodInfluentialHamming Embedding and Weak Geometric Consistency for Large Scale Image Search
Jegou, H., Douze, M., & Schmid, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_24 · Cited by 1,901
Fisher Kernels on Visual Vocabularies for Image Categorization
Perronnin, F., & Dance, C. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383266 · Cited by 1,641 · cited 1×
“Two kinds of queries are defined in the dataset: (1) ST1: entire video copy (normally between 5 minutes and 1 hour), where the videos may be slightly recoded and/or noised; (2) ST2: partial video copy, where two videos only share one or more short segments (between 1 second to 1 minute).”
BackgroundLost in quantization: Improving particular object retrieval in large scale image databases
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587635 · Cited by 1,478 · cited 1×
“, more Voronoi cells) since the latter will cause more quantization error [30].”
BackgroundThree things everyone should know to improve object retrieval
Arandjelovic, R., & Zisserman, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248018 · Cited by 1,383 · cited 1×
“The frame-level matching by local descriptors is almost the same as the techniques used in image-based object retrieval, which has been extensively studied in recent years [9], [19], [20], [21], [22], [23].”
MethodLarge-scale image retrieval with compressed Fisher vectors
Perronnin, F., Liu, Y., Sanchez, J., & Poirier, H. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540009 · Cited by 761 · cited 1×
“Videos which are created from two sources but contain the same contents, such as the same scene captured by two cameras, are near-duplicate videos by definition, but not copy videos.”
BackgroundDiscriminative Deep Metric Learning for Face Verification in the Wild
Hu, J., Lu, J., & Tan, Y.-P. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.242 · Cited by 682 · cited 1×
“Similarly, bag-of-words representations of local features and inverted file indexing are also used in another system by Tan et al. [15].”
MethodModeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification
Wu, Z., Wang, X., Jiang, Y.-G., Ye, H., & Xue, X. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2806222 · Cited by 479 · cited 1×
“progress of deep learning on image classification [33], [34] and video analysis [35], [36].”
BackgroundSelf-citationPractical elimination of near-duplicates from web video search
Wu, X., Hauptmann, A. G., & Ngo, C.-W. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291280 · Cited by 340
Multiple feature hashing for real-time large scale near-duplicate video retrieval
Song, J., Yang, Y., Huang, Z., Shen, H. T., & Hong, R. (2011) · Proceedings of the 19th ACM international conference on Multimedia · doi:10.1145/2072298.2072354 · Cited by 334 · cited 3×
“Our evaluation suggests that all the existing techniques, including the deep learning features, are far from satisfactory in detecting complex real copies.”
BackgroundVideo copy detection: a comparative study
Law-To, J., Chen, L., Joly, A., Laptev, I., Buisson, O., Gouet-Brunet, V., Boujemaa, N., & Stentiford, F. (2007) · Proceedings of the 6th ACM international conference on Image and video retrieval · doi:10.1145/1282280.1282336 · Cited by 273 · cited 1×
“For instance, Indyk et al. [5] downloaded 2,000 clips of news, music videos and movie trailers with durations between 2 and 5 minutes.”
BackgroundDescriptor Matching with Convolutional Neural Networks: a Comparison to SIFT
Fischer, P., Dosovitskiy, A., & Brox, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.5769 · Cited by 226 · cited 1×
“Different from existing datasets mostly generated “artificially” by imposing a few pre-defined transformations, the complex content variations in VCDB pose new challenges to video copy detection research. features are extracted and quantized to the bag-of-visualwords (BoV) representations, which…”
BackgroundAn Image-Based Approach to Video Copy Detection With Spatio-Temporal Post-Filtering
Douze, M., Jegou, H., & Schmid, C. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2046265 · Cited by 165
Deep Metric Learning for Practical Person Re-IdentificationPreprint
Yi, D., Lei, Z., & Li, S. Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.4979 · Cited by 163 · cited 1×
“Similarly, bag-of-words representations of local features and inverted file indexing are also used in another system by Tan et al. [15].”
MethodEvaluating Two-Stream CNN for Video Classification
Ye, H., Wu, Z., Zhao, R.-W., Wang, X., Jiang, Y., & Xue, X. (2015) · International Conference on Multimedia Retrieval · doi:10.1145/2671188.2749406 · Cited by 140 · cited 1×
“progress of deep learning on image classification [33], [34] and video analysis [35], [36].”
BackgroundSelf-citationFlip-Invariant SIFT for Copy and Object Detection
Zhao, W.-L., & Ngo, C.-W. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2226043 · Cited by 138 · cited 1×
“Wu et al. [3] created the popularly used CC_Web dataset, which contains 12,790 videos collected by searching videos on Google, YouTube and Yahoo!.”
MethodEvent Retrieval in Large Video Collections with Circulant Temporal Encoding
Revaud, J., Douze, M., Schmid, C., & Jegou, H. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.318 · Cited by 133 · cited 1×
“Wu et al. [3] created the popularly used CC_Web dataset, which contains 12,790 videos collected by searching videos on Google, YouTube and Yahoo!.”
MethodScalable detection of partial near-duplicate videos by visual-temporal consistency
Tan, H.-K., Ngo, C.-W., Hong, R., & Chua, T.-S. (2009) · Proceedings of the 17th ACM international conference on Multimedia · doi:10.1145/1631272.1631295 · Cited by 110 · cited 7×
“The US National Institute of Standards and Technology included a task on content-based copy detection in 2008 in its annual TRECVID evaluation [8].”
MethodInfluentialSemantic-Aware Co-indexing for Image Retrieval
Zhang, S. S., Yang, M., Wang, X., Lin, Y., & Tian, Q. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.210 · Cited by 97
Hough Pyramid Matching: Speeded-Up Geometry Re-ranking for Large Scale Image Retrieval
Avrithis, Y., & Tolias, G. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0659-3 · Cited by 85
VCDB: A Large-Scale Database for Partial Copy Detection in Videos
Jiang, Y.-G., Jiang, Y., & Wang, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_24 · Cited by 82 · cited 1×
“Index Terms—Video copy detection, benchmark dataset, frame matching, temporal alignment, deep learning Ç”
BackgroundSelf-citationRobust Content-Based Video Copy Identification in a Large Reference Database
Joly, A., Frélicot, C., & Buisson, O. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45113-7_41 · Cited by 69 · cited 1×
“And thenwe review a few representative approaches and briefly discuss some popular deep learningmethods.”
BackgroundCompact Video Description for Copy Detection with Precise Temporal Alignment
Douze, M., Jégou, H., Schmid, C., & Pérez, P. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_38 · Cited by 53 · cited 4×
“And thenwe review a few representative approaches and briefly discuss some popular deep learningmethods.”
BackgroundMethodInfluential
Cited by
ViSiL: Fine-Grained Spatio-Temporal Video Similarity Learning
Kordopatis-Zilos, G., Papadopoulos, S., Patras, I., & Kompatsiaris, I. Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00645 · Cited by 109
“The proposed approach is compared with the publicly available implementation of Deep Metric Learning (DML) [21], the Circulant Temporal Encoding (CTE) [28] (we report the results of the original paper) and our two reimplementations based on Dynamic Programming (DP) [7] and Temporal Networks (TN) [32].”
BackgroundMethodInfluentialFIVR: Fine-Grained Incident Video Retrieval
Kordopatis-Zilos, G., Papadopoulos, S., Patras, I., & Kompatsiaris, I. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2905741 · Cited by 79
“techniques include the extraction from the activations of one of the fully connected layers of CNN architectures [23], [48], the application of the Maximum Activation of Convolution (MAC)”
Feng, Y., Ma, L., Liu, W., Zhang, T., & Luo, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_4 · Cited by 71
Backdoor attacks against learning systems
Ji, Y., Zhang, X., & Wang, T. (2017) · IEEE Conference on Communications and Network Security · doi:10.1109/cns.2017.8228656 · Cited by 85
Learning spatial-temporal features for video copy detection by the combination of CNN and RNN
Hu, Y., & Lu, X. (2018) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2018.05.013 · Cited by 57
Metadata
- Authors
- Jiang, Y.-G., & Wang, J.
- Affiliations
- Fudan University (CN)
- Venue
- IEEE Transactions on Big Data
- Published
- 2016-03-01
- Volume
- 2
- Issue
- 1
- Pages
- 32-42
- ISSN
- 2332-7790 (ISSN-L)
- OpenAlex
- W2344188636
- Cited by
- 72
- References
- 33
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 8 · 2023: 10 · 2022: 9 · 2021: 9 · 2020: 3 · 2019: 7 · 2018: 13 · 2017: 3 · 2016: 4
- FWCI
- 6.11
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Big Data (publishedVersion)
- Funders
- National 863 Program (#2014AA015101)
- NSF (#61572134)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- benchmark, video similarity, deep features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.