$360+x$: A Panoptic Multi-modal Scene Understanding Dataset
Open accessAbstract
Human perception of the world is shaped by a multitude of viewpoints and modalities. While many existing datasets focus on scene understanding from a certain perspective (e.g. egocentric or third-person views), our dataset offers a panoptic perspective (i.e. multiple viewpoints with multiple data modalities). Specifically, we encapsulate third-person panoramic and front views, as well as egocentric monocular/binocular views with rich modalities including video, multi-channel audio, directional binaural delay, location data and textual scene descriptions within each scene captured, presenting comprehensive observation of the world. To the best of our knowledge, this is the first database that covers multiple viewpoints with multiple data modalities to mimic how daily information is accessed in the real world. Through our benchmark analysis, we presented 5 different scene understanding tasks on the proposed 360+x dataset to evaluate the impact and benefit of each data modality and perspective in panoptic scene understanding. We hope this unique dataset could broaden the scope of comprehensive scene understanding and encourage the community to approach these problems from more diverse perspectives.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1× · see citing works
“To establish comprehensive and authoritative scene categories that reflect daily life, we referred to the Places Database [34], which is derived from WordNet [18], as our primary basis.”
MethodUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · see citing works
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042 · see citing works
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965 · see citing works
The Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 3,711 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,605 · see citing works
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578 · see citing works
The kinetics human action video dataset
Kay, W. (2024) · TIB Data Manager · doi:10.57702/jtuxnt5m · Cited by 1,646 · see citing works
High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth
Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., & Westling, P. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11752-2_3 · Cited by 1,377 · see citing works
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions
Gu, C., Sun, C., Ross, D. A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan, S., Toderici, G. D., Ricco, S., Sukthankar, R., et al. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00633 · Cited by 1,029 · see citing works
Scaling Egocentric Vision: The "Equation missing" Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_44 · Cited by 875 · see citing works
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
Liao, Y., Xie, J., & Geiger, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3179507 · Cited by 770 · see citing works
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · see citing works
What Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588 · see citing works
ActionFormer: Localizing Moments of Actions with Transformers
Zhang, C.-L., Wu, J., & Li, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19772-7_29 · Cited by 471 · see citing works
Vggsound: A Large-Scale Audio-Visual Dataset
Chen, H., Xie, W., Vedaldi, A., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053174 · Cited by 452 · see citing works
Self-citationSelf-Supervised Spatiotemporal Learning via Video Clip Order Prediction
Xu, D., Xiao, J., Zhao, Z., Shao, J., Xie, D., & Zhuang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01058 · Cited by 417 · see citing works
Recognizing scene viewpoint using panoramic place representation
Xiao, J.-X., Ehinger, K. A., Oliva, A., & Torralba, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247991 · Cited by 414 · see citing works
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
Yeung, S. Y., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., & Fei-Fei, L. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1013-y · Cited by 403 · see citing works
Balanced Multimodal Learning via On-the-fly Gradient Modulation
Peng, X., Wei, Y., Deng, A., Wang, D., & Hu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00806 · Cited by 374 · see citing works
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization
Zhao, H., Torralba, A. B., Torresani, L., & Yan, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00876 · Cited by 261 · see citing works
DrivingStereo: A Large-Scale Dataset for Stereo Matching in Autonomous Driving Scenarios
Yang, G., Song, X., Huang, C., Deng, Z., Shi, J., & Zhou, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00099 · Cited by 253 · see citing works
(2023) · doi:10.1109/cvpr52729.2023.01808 · Cited by 226 · see citing works
Self-supervised Video Representation Learning by Pace Prediction
Wang, J., Jiao, J., & Liu, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_30 · Cited by 223 · see citing works
Self-citationMeagher, B. R. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868319845938 · Cited by 89 · see citing works
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
Song, S., Zeng, A., Chang, A. X., Savva, M., Savarese, S., & Funkhouser, T. A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00405 · Cited by 66 · see citing works
TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action LocalizationPreprint
Tang, T. N., Kim, K., & Sohn, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09055 · Cited by 22 · see citing works
Proposal Relation Network for Temporal Action DetectionPreprint
Wang, X., Qing, Z., Huang, Z., Feng, Y., Zhang, S., Jiang, J., Tang, M., Gao, C., & Sang, N. (2021) · arXiv (Cornell University) · doi:10.13140/rg.2.2.29021.54247 · Cited by 18 · see citing works
Proposal Relation Network for Temporal Action DetectionPreprint
Wang, X., Qing, Z., Huang, Z., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11812 · Cited by 15 · see citing works
Sound Localization by Self-supervised Time Delay Estimation
Chen, Z., Fouhey, D. F., & Owens, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_28 · Cited by 12 · see citing works
Egok360: A 360 Egocentric Kinetic Human Activity Video Dataset
Bhandari, K., DeLaGarza, M. A., Zong, Z., Latapie, H. M., & Yan, Y. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191256 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“For instance, 360+x [251] combines 360° panoramic views, first-person, and third-person perspectives while integrating modalities such as audio and textual descriptions, providing a rich experimental foundation for multi-view scene understanding and multimodal fusion research.”
Ai, H., Cao, Z., & Wang, L. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02391-w · Cited by 19 · see citing works
MMGeo: Multimodal Compositional Geo-Localization for UAVs
Ji, Y., He, B., Tan, Z., & Wu, L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02334 · Cited by 9 · see citing works
ImViD: Immersive Volumetric Videos for Enhanced VR Engagement
Yang, Z., Pan, S., Wang, S., Wang, H., Lin, L., Li, G., Wen, Z., Lin, B., Tao, J., & Yu, T. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01543 · Cited by 5 · see citing works
Sheng, Z., Huang, Z., Qu, Y., Chen, J., Luo, Y., Chen, Y., Leng, Y., & Chen, S. (2026) · CHAIN · doi:10.23919/chain.2026.000005 · Cited by 6 · see citing works
Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description
Halacheva, A.-M., Miao, Y., Zaech, J.-N., Wang, X., Gool, L. V., & Paudel, D. P. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00534 · Cited by 3 · see citing works
He, Y., Huang, Y., Guo, C., Lü, L., Pei, B., Xu, J., Lu, T., & Sato, Y. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02608-y · Cited by 3 · see citing works
OmniDrag: Enabling Motion Control for Omnidirectional Image-to-Video Generation
Li, W., Zhao, S., Mou, C., Sheng, X., Zhang, Z., Wang, Q., Li, J., Zhang, L., & Zhang, J. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02629-7
Wang, P., Mirzoyev, L., & Alzhanova, A. B. (2026) · IEEE Access · doi:10.1109/access.2026.3653001
Domain-Adaptive Panoramic Video Generation Using Image Diffusers
Prabhu, P. A., Raghavendra, S., Abhilash, S. K., Nookala, V. M., Kumar, P. V. A., Adesh, N. D., & Chandrakala, C. B. (2026) · IEEE Access · doi:10.1109/access.2026.3667172
Liu, Y., Chen, C., Yu, H., Peng, J., & Shen, Z. (2026) · Signal Image and Video Processing · doi:10.1007/s11760-026-05400-8
Xue, W., & Huo, Z. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810646
Huynh, V.-T., Nguyen, M.-K., Le-Hinh, N.-T., Ly, D.-N., Nguyen, T., & Tran, M. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-2590-3_1
Diffusion-Based Immersive Visual Reasoning
Zhang, Y., Jiang, M. X., & Zhao, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37098-3_19
OmniPoint: Universal Monocular Metric Pointcloud from Any Camera
Ye, B., Pollefeys, M., Yang, M., & Kundu, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37602-2_32
Lin, J., Zhu, C., Kneuertz, P. J., Bai, Y., & Xue, Y. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00534 · Cited by 1 · see citing works
Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?
Dongfang, Z., Xu, Z., Weng, Z., Lyu, Y., Paudel, D. P., Gool, L. V., Yang, K., & Hu, X. (2026) · doi:10.1109/cvprf71909.2026.00934
- Authors
- Chen, H., Hou, Y., Qu, C., Testini, I., Hong, X., & Jiao, J.
- Affiliations
- University of Birmingham (GB)
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2024-06-16
- Pages
- 19373-19382
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4402726979
- Cited by
- 17
- References
- 32
- Citations by year
- 2026: 12 · 2025: 5
- FWCI
- 3.52
- Type
- conference paper
- Language
- en
- Open access
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Funders
- CSC (202308060328)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- multimodal data, egocentric vision, panoramic video, binaural audio, scene description, location data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
