The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Highlight Detection with Pairwise Deep Ranking for First-Person Video Summarization
Abstract
The emergence of wearable devices such as portable cameras and smart glasses makes it possible to record life logging first-person videos. Browsing such long unstructured videos is time-consuming and tedious. This paper studies the discovery of moments of user's major or special interest (i.e., highlights) in a video, for generating the summarization of first-person videos. Specifically, we propose a novel pairwise deep ranking model that employs deep learning techniques to learn the relationship between high-light and non-highlight video segments. A two-stream network structure by representing video segments from complementary information on appearance of video frames and temporal dynamics across frames is developed for video highlight detection. Given a long personal video, equipped with the highlight detection model, a highlight score is assigned to each segment. The obtained highlight segments are applied for summarization in two ways: video time-lapse and video skimming. The former plays the highlight (non-highlight) segments at low (high) speed rates, while the latter assembles the sequence of segments with the highest scores. On 100 hours of first-person videos for 15 unique sports categories, our highlight detection achieves the improvement over the state-of-the-art RankSVM method by 10.5% in terms of accuracy. Moreover, our approaches produce video summary with better quality by a user study from 35 human subjects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Separate visual pathways for perception and action
Goodale, M. A., & Milner, A. D. (1992) · Trends in Neurosciences · doi:10.1016/0166-2236(92)90344-8 · Cited by 6,519
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Learning Fine-Grained Image Similarity with Deep Ranking
Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., & Wu, Y. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.180 · Cited by 1,269
Large-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,266
Creating Summaries from User Videos
Gygli, M., Grabner, H., Riemenschneider, H., & Van Gool, L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_33 · Cited by 752
Discovering important people and objects for egocentric video summarization
Lee, Y. J., Ghosh, J., & Grauman, K. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247820 · Cited by 658
A generic framework of user attention model and its application in video summarization
Ma, Y., Hua, X., Lu, L., & Zhang, H.-J. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2005.854410 · Cited by 516
Story-Driven Summarization for Egocentric Video
Lu, Z., & Grauman, K. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.350 · Cited by 491
Category-Specific Video Summarization
Potapov, D., Douze, M., Harchaoui, Z., & Schmid, C. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_35 · Cited by 481
Action and Event Recognition with Fisher Vectors on a Compact Feature Set
Oneata, D., Verbeek, J., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.228 · Cited by 376
Video summarization and scene detection by graph modeling
Ngo, C.-W., Ma, Y.-F., & Zhang, H.-J. (2005) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2004.841694 · Cited by 345
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
Algorithm Design: Foundations, Analysis, and Internet Examples
Goodrich, M. T., & Tamassia, R. (2001) · Medical Entomology and Zoology · Cited by 249
Ranking Domain-Specific Highlights by Analyzing Edited Videos
Sun, M., Farhadi, A., & Seitz, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_51 · Cited by 180
Learning Query and Image Similarities with Ranking Canonical Correlation Analysis
Yao, T., Mei, T., & Ngo, C. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.12 · Cited by 84
Self-citationClick-through-based cross-view learning for image search
Pan, Y., Yao, T., Mei, T., et al. (2014) · doi:10.1145/2600428.2609568 · Cited by 74
Self-citationYao, T., Mei, T., Ngo, C., & Li, S. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502085 · Cited by 49
Self-citationScene Aligned Pooling for Complex Video Recognition
Cao, L., Mu, Y., Natsev, A., Chang, S., Hua, G., & Smith, J. R. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_49 · Cited by 48
Near-lossless semantic video summarization and its applications to video analysis
Mei, T., Tang, L.-X., Tang, J., et al. (2013) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/2487268.2487269 · Cited by 41
Self-citation
Cited by
A Survey of Deep Learning-Based Object Detection
Jiao, L., Zhang, F., Liu, F., Yang, S., Li, L., Feng, Z., & Qu, R. (2019) · IEEE Access · doi:10.1109/access.2019.2939201 · Cited by 1,323
Dense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,251
Localizing Moments in Video with Natural Language
Hendricks, L. A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., & Russell, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.618 · Cited by 958
Stacked Memory Network for Video Summarization
Wang, J., Wang, W., Wang, Z., Wang, L., Feng, D., & Tan, T. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350992 · Cited by 60
Metadata
- Authors
- Yao, T., Mei, T., & Rui, Y.
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 982-990
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2467794422
- Cited by
- 276
- References
- 28
- FWCI
- 11.98
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.