The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Gaze Prediction in Dynamic 360° Immersive Videos
Abstract
This paper explores gaze prediction in dynamic 360° immersive videos, i.e., based on the history scan path and VR contents, we predict where a viewer will look at an upcoming time. To tackle this problem, we first present the large-scale eye-tracking in dynamic VR scene dataset. Our dataset contains 208 360° videos captured in dynamic scenes, and each video is viewed by at least 31 subjects. Our analysis shows that gaze prediction depends on its history scan path and image contents. In terms of the image contents, those salient objects easily attract viewers' attention. On the one hand, the saliency is related to both appearance and motion of the objects. Considering that the saliency measured at different scales is different, we propose to compute saliency maps at different spatial scales: the sub-image patch centered at current gaze point, the sub-image corresponding to the Field of View (FoV), and the panorama image. Then we feed both the saliency maps and the corresponding images into a Convolutional Neural Network (CNN) for feature extraction. Meanwhile, we also use a Long-Short-Term-Memory (LSTM) to encode the history scan path. Then we combine the CNN features and LSTM features for gaze displacement prediction between gaze point at a current time and gaze point at an upcoming time. Extensive experiments validate the effectiveness of our method for gaze prediction in dynamic VR scenes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry
Koch, C., & Ullman, S. (1987) · Matters of Intelligence · doi:10.1007/978-94-009-3833-5_5 · Cited by 3,722
Harel, J., Koch, C., & Perona, P. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0073 · Cited by 3,439
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
Context-Aware Saliency Detection
Goferman, S., Zelnik-Manor, L., & Tal, A. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.272 · Cited by 1,567
Analyzing and Visualizing State Sequences in R with TraMineR
Gabadinho, A., Ritschard, G., Müller, N. S., & Studer, M. (2011) · Journal of Statistical Software · doi:10.18637/jss.v040.i04 · Cited by 1,357
Guo, C., & Zhang, L. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2030969 · Cited by 934
DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection
Liu, N., & Han, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.80 · Cited by 833
Human Pose Estimation with Iterative Error Feedback
Carreira, J., Agrawal, P., Fragkiadaki, K., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.512 · Cited by 805
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., & Durand, F. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2815601 · Cited by 802
1Salient Object Detection: A Survey
Borji, A., Jiang, H., & Li, J. (2016) · Cited by 761
Spatio-temporal Saliency detection using phase spectrum of quaternion fourier transform
Guo, C., Ma, Q., & Zhang, L. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587715 · Cited by 754
Static and space-time visual saliency detection by self-resemblance
Seo, H. J., & Milanfar, P. (2009) · Journal of Vision · doi:10.1167/9.12.15 · Cited by 633
SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks
Huang, X., Shen, C., Boix, X., & Zhao, Q. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.38 · Cited by 597
DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations
Kruthiventi, S. S. S., Ayush, K., & Babu, R. V. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2710620 · Cited by 478
Shallow and Deep Convolutional Networks for Saliency Prediction
Pan, J., Sayrol, E., Giro-I-Nieto, X., McGuinness, K., & OConnor, N. E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.71 · Cited by 471
Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion
Mital, P. K., Smith, T. J., Hill, R. L., & Henderson, J. M. (2011) · Cognitive Computation · doi:10.1007/s12559-010-9074-z · Cited by 468
Coarse-to-Fine Auto-Encoder Networks (CFAN) for Real-Time Face Alignment
Zhang, J., Shan, S., Kan, M., & Chen, X. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_1 · Cited by 468
Saliency in VR: How Do People Explore Virtual Environments?
Sitzmann, V., Serrano, A., Pavel, A., Agrawala, M., Gutierrez, D., Masia, B., & Wetzstein, G. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2793599 · Cited by 459
A Framework to Evaluate Omnidirectional Video Coding Schemes
Yu, M., Lakshman, H., & Girod, B. (2015) · 2015 IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2015.12 · Cited by 441
Spatiotemporal Saliency in Dynamic Scenes
Mahadevan, V., & Vasconcelos, N. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.112 · Cited by 409
A deep multi-level network for saliency prediction
Cornia, M., Baraldi, L., Serra, G., & Cucchiara, R. (2016) · 2016 23rd International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2016.7900174 · Cited by 360
Saliency-Aware Video Compression
Hadizadeh, H., & Bajic, I. V. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2282897 · Cited by 326
360-Degree Video Head Movement Dataset
Corbillon, X., De Simone, F., & Simon, G. (2017) · Proceedings of the 8th ACM on Multimedia Systems Conference · doi:10.1145/3083187.3083215 · Cited by 284
Learning to Predict Gaze in Egocentric Video
Li, Y., Fathi, A., & Rehg, J. M. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.399 · Cited by 230
360° Video Viewing Dataset in Head-Mounted Virtual Reality
Lo, W.-C., Fan, C.-L., Lee, J., Huang, C., Chen, K.-T., & Hsu, C.-H. (2017) · ACM Multimedia Systems Conference (MMSys) · doi:10.1145/3083187.3083219 · Cited by 222
Predicting eye fixations using convolutional neural networks
Liu, N., Han, J., Zhang, D., Wen, S., & Liu, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298633 · Cited by 201
A Dataset of Head and Eye Movements for 360 Degree Images
Rai, Y., Gutiérrez, J., & Le Callet, P. (2017) · Proceedings of the 8th ACM on Multimedia Systems Conference · doi:10.1145/3083187.3083218 · Cited by 197
Evaluation of the Tobii EyeX Eye tracking controller and Matlab toolkit for research
Gibaldi, A., Vanegas, M., Bex, P. J., & Maiello, G. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0762-9 · Cited by 186
Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Videos
Hu, H.-N., Lin, Y.-C., Liu, M.-Y., Cheng, H.-T., Chang, Y.-J., & Sun, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.153 · Cited by 178
Pedestrian Behavior Understanding and Prediction with Deep Neural Networks
Yi, S., Li, H., & Wang, X. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_16 · Cited by 168
What do saliency models predict?
Koehler, K., Guo, F., Zhang, S., & Eckstein, M. P. (2014) · Journal of Vision · doi:10.1167/14.3.14 · Cited by 156
Kruthiventi, S. S. S., Gudisa, V., Dholakiya, J. H., & Babu, R. V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.623 · Cited by 145
Visual perception and sequences of eye movement fixations: a stochastic modeling approach
Hacısalihzade, S. S., STARK, L. W., & Allen, J. S. (1992) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.155948 · Cited by 129
How many bits does it take for a stimulus to be salient?
Khatoonabadi, S. H., Vasconcelos, N. M., Bajić, I. V., & Shan, Y. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299189 · Cited by 110
Deep Future Gaze: Gaze Anticipation on Egocentric Videos Using Adversarial Networks
Zhang, M., Ma, K. T., Lim, J., Zhao, Q., & Feng, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.377 · Cited by 104
Temporal spectral residual: fast motion saliency detection
Cui, X., Liu, Q., & Metaxas, D. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631370 · Cited by 78
Forecast the Plausible Paths in Crowd Scenes
Su, H., Zhu, J., Dong, Y., & Zhang, B. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/386 · Cited by 72
Self-citationAttention Prediction in Egocentric Video Using Motion and Visual Saliency
Yamada, K., Sugano, Y., Okabe, T., Sato, Y., Sugimoto, A., & Hiraki, K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25367-6_25 · Cited by 54
Deep Learning for Saliency Prediction in Natural VideoPreprint
Chaabouni, S., Benois‐Pineau, J., Hadar, O., & Amar, C. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.08010 · Cited by 31
Cited by
Deep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540
State-of-the-Art in 360° Video/Image Processing: Perception, Assessment and Compression
Xu, M., Li, C., Zhang, S., & Callet, P. L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2966864 · Cited by 281
A Survey on Adaptive 360° Video Streaming: Solutions, Challenges and Opportunities
Yaqoob, A., Bi, T., & Muntean, G.-M. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3006999 · Cited by 254
LiveDeep: Online Viewport Prediction for Live Virtual Reality Streaming Using Lifelong Deep Learning
Feng, X., Liu, Y., & Wei, S. (2020) · 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr46266.2020.00104 · Cited by 45
LiveObj: Object Semantics-based Viewport Prediction for Live Mobile Virtual Reality Streaming
Feng, X., Bao, Z., & Wei, S. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3067686 · Cited by 34
Metadata
- Authors
- Xu, Y., Dong, Y., Wu, J., Sun, Z., Shi, Z., Yu, J., & Gao, S.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 5333-5342
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2799064164
- Cited by
- 273
- References
- 41
- FWCI
- 21.89
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.