CoChat
Sign Up
Cited by 28Multimodal Machine Learning ApplicationsHuman Pose and Action RecognitionRobot Manipulation and Learning

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

EgoTaskQA: Understanding Human Tasks in Egocentric Videos

Open access
Jia, Baoxiong, Lei, Ting, Zhu, Song-Chun, Huang, Siyuan·2022·Advances in Neural Information Processing Systems 35
DOI 10.52202/068431-0242
Open this paper in CoChat — free accountOpen PDF

Abstract

Understanding human tasks through video observations is an essential capability of intelligent agents. The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i.e., state changes), and their causal dependencies. These challenges are further aggravated by the natural parallelism from multi-tasking and partial observations in multi-agent collaboration. Most prior works leverage action localization or future prediction as an indirect metric for evaluating such task understanding from videos. To make a direct evaluation, we introduce the EgoTaskQA benchmark that provides a single home for the crucial dimensions of task understanding through question-answering on real-world egocentric videos. We meticulously design questions that target the understanding of (1) action dependencies and effects, (2) intents and goals, and (3) agents' beliefs about others. These questions are divided into four types, including descriptive (what status?), predictive (what will?), explanatory (what caused?), and counterfactual (what if?) to provide diagnostic analyses on spatial, temporal, and causal understandings of goal-oriented tasks. We evaluate state-of-the-art video reasoning models on our benchmark and show their significant gaps between humans in understanding complex goal-oriented egocentric videos. We hope this effort will drive the vision community to move onward with goal-oriented video understanding and reasoning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/068431-0242

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent works evaluating VLMs for embodied and online scenarios primarily focus on perception tasks rather than long-horizon decision making.

    “Recent works evaluating VLMs for embodied and online scenarios [86–91] primarily focus on perception tasks rather than long-horizon decision making.”

    From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [89]

    The Introduction (p. 2) describes EgoTaskQA as an evaluation of models using questions about egocentric videos, rather than long-horizon decision making. But it specifically targets action dependencies, goals, and causal reasoning—not merely perception—and the cited passages do not establish that it evaluates VLMs in an online scenario. Thus the paper supports the contrast with decision-making evaluation, but not the full characterization. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 100

  1. [1]Dare A Baldwin, Jodie A Baird, Megan M Saylor, and M Angela Clark. Infants parse dynamic action. Child development , 72(3):708-717, 2001. 1
  2. [2]György Gergely, Harold Bekkering, and Ildikó Király. Rational imitation in preverbal infants. Nature , 415(6873):755-755, 2002.
  3. [3]Amanda L Woodward. Infants selectively encode the goal object of an actor's reach. Cognition , 69(1):1-34, 1998. 1
  4. [5]Michael Land, Neil Mennie, and Jennifer Rusted. The roles of vision and eye movements in the control of activities of daily living. Perception , 28(11):1311-1328, 1999. 2
  5. [6]Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 , 2022. 2 · arXiv
  6. [7]Lifeng Fan, Shuwen Qiu, Zilong Zheng, Tao Gao, Song-Chun Zhu, and Yixin Zhu. Learning triadic belief dynamics in nonverbal communication from videos. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 2, 3
  7. [8]Hamed Pirsiavash and Deva Ramanan. Detecting activities of daily living in first-person camera views. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2012. 2, 4
  8. [9]Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626 , 2018. 4 · arXiv
  9. [10]Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of European Conference on Computer Vision (ECCV) , 2018. 3
  10. [11]Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-chun Zhu. Lemma: A multi-view dataset for learning multi-agent multi-task activities. In Proceedings of European Conference on Computer Vision (ECCV) , 2020. 2, 4, 18, 25
  11. [12]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV) , 130(1):33-55, 2022. 4
  12. [13]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 2, 4
  13. [14]Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cognition , 113(3):329-349, 2009. 3
  14. [15]Michael Shum, Max Kleiman-Weiner, Michael L Littman, and Joshua B Tenenbaum. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI) , 2019. 3
  15. [18]Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence , 2(3-4):189-208, 1971. 3
  16. [20]Kun Duan, Devi Parikh, David Crandall, and Kristen Grauman. Discovering localized attributes for fine-grained recognition. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2012. 3
  17. [21]Phillip Isola, Joseph J Lim, and Edward H Adelson. Discovering states and transformations in image collections. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2015.
  18. [22]Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attributeobject compositions. In Proceedings of European Conference on Computer Vision (ECCV) , 2018. 3
  19. [23]Alireza Fathi and James M Rehg. Modeling actions through state changes. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2013. 3
  20. [24]Amy Fire and Song-Chun Zhu. Learning perceptual causality from video. ACM Transactions on Intelligent Systems and Technology (TIST) , 7(2):1-22, 2015.
  21. [25]Xiaolong Wang, Ali Farhadi, and Abhinav Gupta. Actions˜ transformations. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.
  22. [26]Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Simon Lacoste-Julien. Joint discovery of object states and manipulation actions. In Proceedings of International Conference on Computer Vision (ICCV) , 2017.
  23. [27]Yang Liu, Ping Wei, and Song-Chun Zhu. Jointly recognizing object fluents and tasks in egocentric videos. In Proceedings of International Conference on Computer Vision (ICCV) , pages 2924-2932, 2017. 3
  24. [28]Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Actionconditional video prediction using deep networks in atari games. Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2015. 3
  25. [29]Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems , 29, 2016.
  26. [30]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2018. 3
  27. [31]Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. Piglet: Language grounding through neuro-symbolic interaction in a 3d world. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2021. 3
  28. [32]Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali NarayanChen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI) , 2022. 3
  29. [33]Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2014. 3
  30. [34]Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4575-4583, 2016.
  31. [35]Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.
  32. [36]Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In Proceedings of International Conference on Computer Vision (ICCV) , 2019. 3
  33. [37]David Paulius, Yongqiang Huang, Roger Milton, William D Buchanan, Jeanine Sam, and Yu Sun. Functional object-oriented network for manipulation learning. In Proceedings of International Conference on Intelligent Robots and Systems (IROS) , 2016. 3, 4
  34. [38]Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305 , 2018. 3 · arXiv
  35. [39]Dave Epstein, Jiajun Wu, Cordelia Schmid, and Chen Sun. Learning temporal dynamics from cycles in narrated video. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 3
  36. [40]Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. In Proceedings of European Conference on Computer Vision (ECCV) , 2020. 3
  37. [41]Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, and Animesh Garg. Plate: Visually-grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters (RAL) , 7(2):4924-4930, 2022. 3
  38. [42]Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. Marioqa: Answering questions by watching gameplay videos. In Proceedings of International Conference on Computer Vision (ICCV) , 2017. 3, 4
  39. [43]Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. Deepstory: Video story QA by deep embedded memory networks. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI) , 2017. 3, 4
  40. [44]Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: collision events for video representation and reasoning. In International Conference on Learning Representations (ICLR) , 2020. 3, 4, 5, 20
  41. [45]Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 3, 4
  42. [46]Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. 3
  43. [47]Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 3, 4
  44. [48]Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. TVQA: localized, compositional video question answering. In Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018. 3, 4
  45. [49]Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. TVQA+: spatio-temporal grounding for video question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2020. 3, 4
  46. [50]Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of ACM International Conference on Multimedia (MM) , 2017. 3, 24
  47. [51]Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Leveraging video descriptions to learn video question answering. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI) , 2017. 3
  48. [52]Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI) , 2019. 3, 4
  49. [53]Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. TGIF-QA: toward spatio-temporal reasoning in visual question answering. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. 3, 4, 24
  50. [54]Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: hierarchical encoder for video+language omni-representation pre-training. In Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020. 3
  51. [55]Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 3, 4
  52. [56]Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. AGQA: A benchmark for compositional spatio-temporal reasoning. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 3, 4, 7, 8, 20, 22
  53. [57]Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of questionanswering to explaining temporal actions. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 3, 4
  54. [58]Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Proceedings of Advances in Neural Information Processing Systems Datasets and Benchmarks (NeurIPS Datasets and Benchmarks Track) , 2021. 3, 4, 8
  55. [59]Chenyou Fan. EgoVQA: An egocentric video question answering benchmark dataset. In Proceedings of International Conference on Computer Vision Workshops (ICCVW) , 2019. 3
  56. [60]Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. Discovering important people and objects for egocentric video summarization. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2012. 3, 4
  57. [61]Zheng Lu and Kristen Grauman. Story-driven summarization for egocentric video. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2013. 3
  58. [62]Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of International Conference on Computer Vision (ICCV) , pages 6202-6211, 2019. 3
  59. [63]Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.
  60. [64]Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Single Model for Many Visual Modalities. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3
  61. [65]Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 3
  62. [66]Antonino Furnari and Giovanni Maria Farinella. Rolling-unrolling lstms for action anticipation from first-person video. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 43(11):4021-4036, 2020.
  63. [67]Siyuan Qi, Baoxiong Jia, and Song-Chun Zhu. Generalized earley parser: Bridging symbolic grammars and sequence data for future prediction. In Proceedings of International Conference on Machine Learning (ICML) , 2018.
  64. [68]Siyuan Qi, Baoxiong Jia, Siyuan Huang, Ping Wei, and Song-Chun Zhu. A generalized earley parser for human activity parsing and prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(8):2538-2554, 2020.
  65. [69]Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 3
  66. [70]Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In Proceedings of International Conference on Computer Vision (ICCV) , 2019. 3
  67. [72]Minjie Cai, Kris M Kitani, and Yoichi Sato. Understanding hand-object manipulation with grasp types and object attributes. In Proceedings of Robotics: Science and Systems (RSS) , 2016.
  68. [73]Sven Bambach, Stefan Lee, David J Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of International Conference on Computer Vision (ICCV) , 2015.
  69. [74]Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In Proceedings of European Conference on Computer Vision (ECCV) , 2018.
  70. [75]Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  71. [76]Minghuang Ma, Haoqi Fan, and Kris M Kitani. Going deeper into first-person activity recognition. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. 3
  72. [77]Ping Wei, Yang Liu, Tianmin Shu, Nanning Zheng, and Song-Chun Zhu. Where and why are they looking? jointly inferring human attention and intentions in complex tasks. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 3
  73. [78]Alircza Fathi, Jessica K Hodgins, and James M Rehg. Social interactions: A first-person perspective. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2012. 3
  74. [79]Hyun Soo Park and Jianbo Shi. Social saliency prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4777-4785, 2015. 3
  75. [80]Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, and Yoichi Sato. Future person localization in first-person videos. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 3
  76. [81]Evonne Ng, Donglai Xiang, Hanbyul Joo, and Kristen Grauman. You2me: Inferring body pose in egocentric video via first and second person interactions. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 3
  77. [82]Curtis Northcutt, Shengxin Zha, Steven Lovegrove, and Richard Newcombe. Egocom: A multi-person multi-modal egocentric communications dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2020. 3
  78. [83]Nishant Rai, Haofeng Chen, Jingwei Ji, Rishi Desai, Kazuki Kozuka, Shun Ishizaka, Ehsan Adeli, and Juan Carlos Niebles. Home action genome: Cooperative compositional action understanding. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 4
  79. [84]Kewei Tu, Meng Meng, Mun Wai Lee, Tae Eun Choe, and Song-Chun Zhu. Joint video and text parsing for understanding events and answering queries. IEEE MultiMedia , 21(2):42-70, 2014. 4
  80. [85]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of International Conference on Computer Vision (ICCV) , 2015.
  81. [86]Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. 20
  82. [87]Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 4, 7
  83. [88]David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In Proceedings of International Conference on Machine Learning (ICML) . PMLR, 2018. 4
  84. [89]Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.
  85. [90]Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. Bongardlogo: A new benchmark for human-level concept learning and reasoning. Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2020.
  86. [91]Chi Zhang, Baoxiong Jia, Song-Chun Zhu, and Yixin Zhu. Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.
  87. [92]Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. Acre: Abstract causal reasoning beyond covariation. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.
  88. [93]Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chun Zhu, and Yixin Zhu. Learning algebraic representation for systematic generalization in abstract reasoning. In Proceedings of European Conference on Computer Vision (ECCV) , 2022. 4
  89. [94]Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visualcomet: Reasoning about the dynamic context of a still image. In European Conference on Computer Vision , pages 508-524. Springer, 2020. 4
  90. [95]Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 4
  91. [96]Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 4
  92. [97]Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019. 7, 8 · arXiv
  93. [98]Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Beyond rnns: Positional self-attention with co-attention for video question answering. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI) , 2019. 7, 8, 24
  94. [99]Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 7, 8, 24
  95. [100]Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI) , 2020. 7, 8, 24
  96. [101]Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 7, 8, 24
  97. [102]Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 7, 8, 24
  98. [103]Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 7, 24
  99. [104]Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 8
  100. [105]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. 8

Cited by

Sorted by citations

Metadata

Authors
Jia, B., Lei, T., Zhu, S.-C., & Huang, S.
ORCID
Venue
Advances in Neural Information Processing Systems 35
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2022
Pages
3343-3360
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/068431-0242
OpenAlex
W7133185840
Cited by
28
References
1
Citations by year
2026: 9 · 2025: 4 · 2024: 9 · 2023: 6
FWCI
1.68
Type
conference paper
Language
en
Open access
Licence
cc-by-nc-sa
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
egocentric video, human tasks, task understanding, question answering, video understanding, human activity recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review