CoChat
Sign Up
PreprintCited by 26Advanced Vision and ImagingAdvanced Neural Network ApplicationsAdvanced Image and Video Retrieval Techniques

The DOI matches its DataCite record.

Geometry-Aware Recurrent Neural Networks for Active Visual Recognition

Open access
Cheng, Ricson, Wang, Ziyan, Fragkiadaki, Katerina·2018·arXiv
DOI 10.48550/arxiv.1811.01292arXiv 1811.01292
Open this paper in CoChat — free accountOpen PDF

Abstract

We present recurrent geometry-aware neural networks that integrate visual information across multiple views of a scene into 3D latent feature tensors, while maintaining an one-to-one mapping between 3D physical locations in the world scene and latent feature locations. Object detection, object segmentation, and 3D reconstruction is then carried out directly using the constructed 3D feature memory, as opposed to any of the input 2D images. The proposed models are equipped with differentiable egomotion-aware feature warping and (learned) depth-aware unprojection operations to achieve geometrically consistent mapping between the features in the input frame and the constructed latent model of the scene. We empirically show the proposed model generalizes much better than geometryunaware LSTM/GRU networks, especially under the presence of multiple objects and cross-object occlusions. Combined with active view selection policies, our model learns to select informative viewpoints to integrate information from by "undoing" cross-object occlusions, seamlessly combining geometry with learning from experience.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1811.01292

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Active recognition methods predominantly operate within a closed-vocabulary framework, restricting agents to pre-defined classes.

    “Although numerous studies have shown promising results in active recognition, these methods predominantly operate within a closed-vocabulary framework.”

    From Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations (2024) · cites this paper as [8]

    The Experiments section (p. 7) describes this paper’s active-recognition task using four predefined object categories, which supports the closed-vocabulary characterization of this method. The Related Work section (p. 2) surveys other active-recognition approaches but does not establish that such methods predominantly use closed vocabularies. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 44

  1. [1][1] J. Aloimonos, I. Weiss, and A. Bandyopadhyay. Active vision. In IJCV , 1988. -
  2. [2][2] P. Ammirato, P. Poirson, E. Park, J. Kosecka, and A. C. Berg. A dataset for developing and benchmarking active vision. In ICRA , 2017. -
  3. [3][3] R. Bajcsy. Active perception. In in Proceedings of the IEEE , 1988. -
  4. [4][4] J. Caicedo and S. Lazebnik. Active object localization with deep reinforcement learning. In ICCV , 2015. -
  5. [5][5] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University - Princeton University Toyota Technological Institute at Chicago, 2015. - · arXiv
  6. [6][6] K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR , abs/1406.1078, 2014. - · arXiv
  7. [7][7] J. Denzler, M. Zobel, and H. Niemann. Information theoretic focal length selection for real-time active 3d object tracking. In Proceedings Ninth IEEE International Conference on Computer Vision , pages 400-407 vol.1, Oct 2003. -
  8. [8][8] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with left-right consistency. CoRR , abs/1609.03677, 2016. - · arXiv
  9. [9][9] A. Gonzalez-Garcia, A. Vezhnevets, and V. Ferrari. An active search strategy for efficient object class detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3022-3031, 2015. -
  10. [10][10] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. IQA: visual question answering in interactive environments. CoRR , abs/1712.03316, 2017. - · arXiv
  11. [11][11] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2616-2625, 2017. -
  12. [12][12] R. Hadsell, S. Chopra, and Y. Lecun. Dimensionality reduction by learning an invariant mapping. In CVPR , 2006. -
  13. [13][13] A. W. Harley, K. G. Derpanis, and I. Kokkinos. Learning dense convolutional embeddings for semantic segmentation. CoRR , abs/1511.04377, 2015. - · arXiv
  14. [14][14] R. Held and A. Hein. Movement-produced stimulation in the development of visually guided behavior. In Journal of Comparative and Physiological Psychology , 1963. -
  15. [15][15] J. F. Henriques and A. Vedaldi. Mapnet: An allocentric spatial memory for mapping environments. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. -
  16. [16][16] J. Hoffman, E. Tzeng, T. Park, J. Zhu, P. Isola, K. Saenko, A. A. Efros, and T. Darrell. Cycada: Cycle-consistent adversarial domain adaptation. CoRR , abs/1711.03213, 2017. - · arXiv
  17. [17][17] D. Jayaraman and K. Grauman. Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion. In European Conference on Computer Vision , pages 489-505. Springer, 2016. -
  18. [18][18] D. Jayaraman and K. Grauman. Learning to look around. CoRR , abs/1709.00507, 2017. - · arXiv
  19. [19][19] E. Johns, S. Leutenegger, and A. J. Davison. Pairwise decomposition of image sequences for active multi-view recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3813-3822, 2016. -
  20. [20][20] A. Kar, C. Häne, and J. Malik. Learning a multi-view stereo machine. CoRR , abs/1708.05375, 2017. - · arXiv
  21. [21][21] C. Kerl, J. Sturm, and D. Cremers. Dense visual SLAM for RGB-D cameras. In IROS , 2013. -
  22. [22][22] M. Malmir, K. Sikka, D. Forster, J. R. Movellan, and G. Cottrell. Deep q-learning for active recognition of germs. In BMVC , 2015. -
  23. [23][23] S. Mathe, A. Pirinen, and C. Sminchisescu. Reinforcement learning for visual object detection. In CVPR , 2016. -
  24. [24][24] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673 , 2016. - · arXiv
  25. [25][25] V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu. Recurrent models of visual attention. CoRR , abs/1406.6247, 2014. - · arXiv
  26. [26][26] E. Parisotto and R. Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. CoRR , abs/1702.08360, 2017. - · arXiv
  27. [27][27] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. CoRR , abs/1612.00593, 2016. - · arXiv
  28. [28][28] M. Ranzato. On learning where to look. In arXiv preprint arXiv:1405.5488 , 2014. - · arXiv
  29. [29][29] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR , abs/1506.01497, 2015. - · arXiv
  30. [30][30] E. Rivlin and H. Rotstein. Control of a camera for active vision: Foveal vision, smooth tracking and saccade. In IJCV , 2000. -
  31. [32][32] S. Soatto. Actionable information in vision. In ICCV , 2009. -
  32. [33][33] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion from a single depth image. IEEE Conference on Computer Vision and Pattern Recognition , 2017. -
  33. [34][34] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems , pages 1057-1063, 2000. -
  34. [35][35] B. W. Tatler, M. M. Hayhoe, M. F. Land, and D. H. Ballard. Eye guidance in natural vision: Reinterpreting salience. Journal of Vision , 11(5):5, 2011. -
  35. [36][36] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. CoRR , abs/1704.06254, 2017. - · arXiv
  36. [37][37] H. F. Tung, A. Harley, W. Seto, and K. Fragkiadaki. Adversarial inverse graphics networks: Learning 2d-to-3d lifting and image-to-image translation with unpaired supervision. ICCV , 2017. -
  37. [38][38] S. Vijayanarasimhan, S. Ricco, C. Schmid, R. Sukthankar, and K. Fragkiadaki. Sfm-net: Learning of structure and motion from video. arXiv preprint arXiv:1704.07804 , 2017. - · arXiv
  38. [39][39] D. Wilkes and J. Tsotsos. Active object recognition. In CVPR , 1992. -
  39. [40][40] L. L. Wong, L. P. Kaelbling, and T. Lozano-Perez. Constructing semantic world models from partial views. In Robotics: Science and Systems (RSS) Workshop on Robots in Clutter , 2013. -
  40. [41][41] J. Wu, Y. Wang, T. Xue, X. Sun, W. T. Freeman, and J. B. Tenenbaum. Marrnet: 3d shape reconstruction via 2.5d sketches. CoRR , abs/1711.03129, 2017. - · arXiv
  41. [42][42] J. Wu, T. Xue, J. J. Lim, Y . Tian, J. B. Tenenbaum, A. Torralba, and W. T. Freeman. 3d interpreter networks for viewer-centered wireframe modeling. International Journal of Computer Vision (IJCV) , 2018. -
  42. [43][43] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR , pages 1912-1920. IEEE Computer Society, 2015. -
  43. [44][44] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and egomotion from video. In arxiv , 2017. -
  44. [45][45] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA , 2017.

Cited by

Sorted by citations

Metadata

Authors
Cheng, R., Wang, Z., & Fragkiadaki, K.
Affiliations
Carnegie Mellon University (US)
ORCID
Wang, Ziyan
Venue
arXiv
Publisher
arXiv
Published
2018
Volume
31
Pages
5081-5091
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1811.01292
arXiv
1811.01292
OpenAlex
W2898979134
Cited by
26
References
0
Citations by year
2025: 1 · 2024: 2 · 2023: 2 · 2022: 4 · 2021: 7 · 2020: 7 · 2019: 1 · 2018: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
object detection, object segmentation, 3D reconstruction, viewpoint selection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review