The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach
Open accessAbstract
Panoramic video provides immersive and interactive experience by enabling humans to control the field of view (FoV) through head movement (HM). Thus, HM plays a key role in modeling human attention on panoramic video. This paper establishes a database collecting subjects' HM in panoramic video sequences. From this database, we find that the HM data are highly consistent across subjects. Furthermore, we find that deep reinforcement learning (DRL) can be applied to predict HM positions, via maximizing the reward of imitating human HM scanpaths through the agent's actions. Based on our findings, we propose a DRL-based HM prediction (DHP) approach with offline and online versions, called offline-DHP and online-DHP. In offline-DHP, multiple DRL workflows are run to determine potential HM positions at each panoramic frame. Then, a heat map of the potential HM positions, named the HM map, is generated as the output of offline-DHP. In online-DHP, the next HM position of one subject is estimated given the currently observed HM position, which is achieved by developing a DRL algorithm upon the learned offline-DHP model. Finally, the experiments validate that our approach is effective in both offline and online prediction of HM positions for panoramic video, and that the learned offline-DHP model can improve the performance of online-DHP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Koch, C., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,271
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Overview of the High Efficiency Video Coding (HEVC) Standard
Sullivan, G. J., Ohm, J.-R., Han, W.-J., & Wiegand, T. (2012) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2221191 · Cited by 8,263
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Citizens as sensors: the world of volunteered geography
Goodchild, M. F. (2007) · GeoJournal · doi:10.1007/s10708-007-9111-y · Cited by 4,522
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1023/a:1022676722315 · Cited by 3,907
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Self-citationRecurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568
Learning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016
State-of-the-Art in Visual Attention Modeling
Borji, A., & Itti, L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.89 · Cited by 1,817
Bayesian surprise attracts human attention
Itti, L., & Baldi, P. (2009) · Vision Research · doi:10.1016/j.visres.2008.09.007 · Cited by 1,224
Guo, C., & Zhang, L. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2030969 · Cited by 934
Sinnott, R. W. (1984) · Sky and Telescope · Cited by 890
Saccadic suppression: A review and an analysis.
Matin, E. (1974) · Psychological Bulletin · doi:10.1037/h0037368 · Cited by 790
Automatic Foveation for Video Compression Using a Neurobiological Model of Visual Attention
Itti, L. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2004.834657 · Cited by 759
Deep Recurrent Q-Learning for Partially Observable MDPs
Hausknecht, M., & Stone, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06527 · Cited by 667
Video Salient Object Detection via Fully Convolutional Networks
Wang, W., Shen, J., & Shao, L. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2754941 · Cited by 642
SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks
Huang, X., Shen, C., Boix, X., & Zhao, Q. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.38 · Cited by 597
Saliency Detection with Recurrent Fully Convolutional Networks
Wang, L., Wang, L., Lu, H., Zhang, P., & Ruan, X. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_50 · Cited by 509
DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations
Kruthiventi, S. S. S., Ayush, K., & Babu, R. V. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2710620 · Cited by 478
Citizens as Sensors: The World of Volunteered Geography
Goodchild, M. F. (2011) · doi:10.1002/9780470979587.ch48 · Cited by 437
Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images
Vig, E., Dorr, M., & Cox, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.358 · Cited by 428
SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
Pan, J., Canton-Ferrer, C., McGuinness, K., O'Connor, N. E., Torres, J., Sayrol, E., & Giró-i-Nieto, X. (2017) · Dublin City University Open Access Institutional Repository (Dublin City University) · doi:10.48550/arxiv.1701.01081 · Cited by 315
Predicting the Driver's Focus of Attention: The DR(eye)VE Project
Palazzi, A., Abati, D., Calderara, S., Solera, F., & Cucchiara, R. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2845370 · Cited by 313
Understanding Low- and High-Level Contributions to Fixation Prediction
Kummerer, M., Wallis, T. S. A., Gatys, L. A., & Bethge, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.513 · Cited by 282
Reinforcement Learning with Unsupervised Auxiliary Tasks
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 247
Exploiting Surroundedness for Saliency Detection: A Boolean Map Approach
Zhang, J., & Sclaroff, S. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2473844 · Cited by 235
Predicting eye fixations using convolutional neural networks
Liu, N., Han, J., Zhang, D., Wen, S., & Liu, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298633 · Cited by 201
Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Videos
Hu, H.-N., Lin, Y.-C., Liu, M.-Y., Cheng, H.-T., Chang, Y.-J., & Sun, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.153 · Cited by 178
Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction
Bak, C., Kocak, A., Erdem, E., & Erdem, A. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2777665 · Cited by 176
Tiling in Interactive Panoramic Video: Approaches and Evaluation
Gaddam, V. R., Riegler, M., Eg, R., Griwodz, C., & Halvorsen, P. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2586304 · Cited by 160
Simulating human saccadic scanpaths on natural images
Wang, W., Chen, C., Wang, Y., Jiang, T., Fang, F., & Yao, Y. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995423 · Cited by 145
Recurrent Mixture Density Network for Spatiotemporal Visual AttentionPreprint
Bazzani, L., Larochelle, H., & Torresani, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08199 · Cited by 139
Lin, Y.-C., Chang, Y.-J., Hu, H.-N., Cheng, H.-T., Huang, C.-W., & Sun, M. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025757 · Cited by 139
SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes
Assens, M., Giró-i-Nieto, X., McGuinness, K., & O'Connor, N. E. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.275 · Cited by 122
How many bits does it take for a stimulus to be salient?
Khatoonabadi, S. H., Vasconcelos, N. M., Bajić, I. V., & Shan, Y. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299189 · Cited by 110
Neumann, U., Pintaric, T., & Rizzo, A. (2000) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/354384.376408 · Cited by 96
A subjective visual quality assessment method of panoramic videos
Xu, M., Li, C., Liu, Y., Deng, X., & Lu, J. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019351 · Cited by 83
Self-citationA reinforcement learning model of selective visual attention
Minut, S., & Mahadevan, S. (2001) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/375735.376414 · Cited by 77
Region of interest extraction and virtual camera control based on panoramic video capturing
Sun, X., Foote, J. T., Kimber, D., & Manjunath, B. S. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2005.854388 · Cited by 75
Learning to Predict Sequences of Human Visual Fixations
Jiang, M. X., Boix, X., Roig, G., Xu, J., Gool, L. V., & Zhao, Q. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2496306 · Cited by 61
What are we looking for: Towards statistical modeling of saccadic eye movements and visual saliency
Sun, X., Yao, H., & Ji, R. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247846 · Cited by 59
Regularized Feature Reconstruction for Spatio-Temporal Saliency Detection
Ren, Z., Gao, S., Chia, L.-T., & Rajan, D. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2259837 · Cited by 54
Semantically-Based Human Scanpath Estimation with HMMs
Liu, H., Xu, D., Huang, Q., et al. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.401 · Cited by 48
Predicting Salient Face in Multiple-Face Videos
Liu, Y., Zhang, S., Xu, M., & He, X. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.343 · Cited by 41
Self-citationLin, Y., Tang, Y. Y., Fang, B., Shang, Z., Huang, Y., & Wang, S. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.119 · Cited by 39
Two-Stream Convolutional Networks for Dynamic Saliency Prediction.Preprint
Bak, Ç., Erdem, A., & Erdem, E. (2016) · arXiv (Cornell University) · Cited by 25
Cited by
State-of-the-Art in 360° Video/Image Processing: Perception, Assessment and Compression
Xu, M., Li, C., Zhang, S., & Callet, P. L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2966864 · Cited by 281
Self-citationA Survey on Adaptive 360° Video Streaming: Solutions, Challenges and Opportunities
Yaqoob, A., Bi, T., & Muntean, G.-M. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3006999 · Cited by 254
Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks
Xu, J., Zhou, W., & Chen, Z. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3015186 · Cited by 166
LiveDeep: Online Viewport Prediction for Live Virtual Reality Streaming Using Lifelong Deep Learning
Feng, X., Liu, Y., & Wei, S. (2020) · 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr46266.2020.00104 · Cited by 45
LiveObj: Object Semantics-based Viewport Prediction for Live Mobile Virtual Reality Streaming
Feng, X., Bao, Z., & Wei, S. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3067686 · Cited by 34
Metadata
- Authors
- Xu, M., Song, Y., Wang, J., Qiao, M., Huo, L., & Wang, Z.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-11-01
- Volume
- 41
- Issue
- 11
- Pages
- 2693-2708
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2884414611
- Cited by
- 259
- References
- 48
- FWCI
- 28.29
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.