Semantic structure from motion with points, regions, and objects
The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index.
Abstract
Structure from motion (SFM) aims at jointly recovering the structure of a scene as a collection of 3D points and estimating the camera poses from a number of input images. In this paper we generalize this concept: not only do we want to recover 3D points, but also recognize and estimate the location of high level semantic scene components such as regions and objects in 3D. As a key ingredient for this joint inference problem, we seek to model various types of interactions between scene components. Such interactions help regularize our solution and obtain more accurate results than solving these problems in isolation. Experiments on public datasets demonstrate that: 1) our framework estimates camera poses more robustly than SFM algorithms that use points only; 2) our framework is capable of accurately estimating pose and location of objects, regions, and points in the 3D scene; 3) our framework recognizes objects and regions more accurately than state-of-the-art single image recognition methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recently, the limitations of purely geometric maps have been recognized and this has spawned a significant and ongoing body of work in semantic mapping of environments”
From Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016) · cites this paper as [9]
In the citing paper
“in order to enhance robot's autonomy and robustness, facilitate more complex tasks (e.g. avoid muddy-road while driving), move from path-planning to task-planning, and enable advanced human-robot interaction [9, 26, 217]”
From Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016) · cites this paper as [9]
In the citing paper
“In their work, the authors demonstrate the improved object recognition performance and robustness”
From Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016) · cites this paper as [9]
In the citing paper
“at the cost of a run-time of 20 minutes per image-pair”
From Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016) · cites this paper as [9]
In the citing paper
“Bao et al. [9] propose one of the first approaches to jointly estimate camera parameters, scene points and object labels using both geometric and semantic attributes in the scene.”
From Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016) · cites this paper as [9]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 2×
“For instance, points do not carry semantic information, but can be robustly matched across views (e.g. by [22]).”
BackgroundOptimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134 · cited 1×
“We use simulated annealing [16] to search for the solution for Eq.”
MethodObject Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853 · cited 2×
“However, [3] has a few limitations: 1) it focuses on modeling " structured " objects (e.g. cars, cups, and bottles), and it ignores " amorphous " scene components (e.g. road, sky, and table surfaces which can be hardly identified by standard object detectors such as [8] [21]).”
BackgroundBundle Adjustment — A Modern Synthesis
Triggs, B., McLauchlan, P. F., Hartley, R. I., & Fitzgibbon, A. W. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-44480-7_21 · Cited by 3,988 · cited 1×
“Most SFM algorithms [29] [28] represent the structure of a scene as a set of 3D points.”
BackgroundScale & Affine Invariant Interest Point Detectors
Mikolajczyk, K., & Schmid, C. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000027790.02288.f2 · Cited by 3,436
Modeling the World from Internet Photo Collections
Snavely, N., Seitz, S. M., & Szeliski, R. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0107-3 · Cited by 2,155 · cited 2×
“[28] [3] Table 3: The error for camera pose estimation.”
BackgroundAn efficient solution to the five-point relative pose problem
Nister, D. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.17 · Cited by 2,009 · cited 1×
“We follow the evaluation criteria in [24].”
Learning a classification model for segmentation
Ren, & Malik (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238308 · Cited by 1,765 · cited 1×
“The region measurements are segmented regions (e.g. by segmentation algorithms such as [26] [33]).”
Segmentation and Recognition Using Structure from Motion Point Clouds
Brostow, G. J., Shotton, J., Fauqueur, J., & Cipolla, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_5 · Cited by 1,050 · cited 1×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundCombined Object Categorization and Segmentation With An Implicit Shape Model
Leibe, B., Leonardis, A., & Schiele, B. (2004) · Max Planck Digital Library · Cited by 890
Putting Objects in Perspective
Hoiem, D., Efros, A. A., & Hebert, M. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0137-5 · Cited by 805 · cited 2×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundRecovering Surface Layout from an Image
Hoiem, D., Efros, A. A., & Hebert, M. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-006-0031-y · Cited by 674 · cited 4×
“where sklk (s k r ) are image area of b k lk (b k r ), n k lk is the region normal vector of bklk (estimated by methods such as [32, 14, 20]), and nr is the normal vector of Br w.”
BackgroundMethodInfluentialDecomposing a scene into geometric and semantically consistent regions
Gould, S., Fulton, R., & Koller, D. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459211 · Cited by 672 · cited 1×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundWide Baseline Stereo Matching based on Local, Affinely Invariant Regions
Tuytelaars, T., & Gool, L. V. (2000) · Procedings of the British Machine Vision Conference 2000 · doi:10.5244/c.14.38 · Cited by 450 · cited 1×
“The point measurements are detected interest points (e.g. by detectors such as [22] [30]).”
BackgroundGeometric reasoning for single image structure recovery
Lee, D. C., Hebert, M., & Kanade, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206872 · Cited by 407 · cited 1×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundFord Campus vision and lidar data set
Pandey, G., McBride, J. R., & Eustice, R. M. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911400640 · Cited by 360
Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics
Gupta, A., Efros, A. A., & Hebert, M. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_35 · Cited by 308 · cited 1×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundGraph Cut Based Inference with Co-occurrence Statistics
Ladicky, L., Russell, C., Kohli, P., & Torr, P. H. S. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15555-0_18 · Cited by 308 · cited 2×
“Ours w/ [17]. sky road building tree others s k y r o a d b u il d in g t r e e o t h e r s 1.0 .92 .89 .87 .56 (b) Person.”
BackgroundInfluentialTILT: Transform Invariant Low-Rank Textures
Zhang, Z., Ganesh, A., Liang, X., & Ma, Y. (2012) · International Journal of Computer Vision · doi:10.1007/s11263-012-0515-x · Cited by 277 · cited 5×
“[32] estimates the region orientation by texture.”
BackgroundMethodInfluentialThinking Inside the Box: Using Appearance Models and Context Based on Room Geometry
Hedau, V., Hoiem, D., & Forsyth, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15567-3_17 · Cited by 249 · cited 1×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundSemi-Local Affine Parts for Object Recognition
Lazebnik, S., Schmid, C., & Ponce, J. (2004) · Procedings of the British Machine Vision Conference 2004 · doi:10.5244/c.18.98 · Cited by 223 · cited 1×
“For example, [27] uses point-region interaction, [19] uses point-object interaction, and [12] [15] [4] use object-region interaction.”
3D Urban Scene Modeling Integrating Recognition and Reconstruction
Cornelis, N., Leibe, B., Cornelis, K., & Van Gool, L. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0081-9 · Cited by 217
Viewpoint invariant texture matching and wide baseline stereo
Schaffalitzky, F., & Zisserman, A. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937686 · Cited by 192 · cited 2×
“As shown in works such as [27] [9], point-region interactions help improve the across-view matching accuracy of both points and regions, and help the estimation of 3D locations and poses of both points and regions.”
BackgroundJoint Optimization for Object Class Segmentation and Dense Stereo Reconstruction
Ladický, Ľ., Sturgess, P., Russell, C., Sengupta, S., Baştanlar, Y., Clocksin, W. F., & Torr, P. H. S. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0489-0 · Cited by 159 · cited 1×
“Ladicky et al. [18] show promising results on joint reconstruction and segmentation from stereo pairs, but it works under the assumption of calibrated cameras with small baselines .”
BackgroundSemantic structure from motion
Bao, S. Y., & Savarese, S. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995462 · Cited by 150 · cited 5×
“Bao and Savarese [3] have recently introduced a formulation to integrate object recognition and SFM.”
BackgroundMethodInfluentialSelf-citationLocally Invariant Fractal Features for Statistical Texture Classification
Varma, M., & Garg, R. D. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408876 · Cited by 150 · cited 1×
“Ours w/ [31].”
Wide-baseline multiple-view correspondences
Ferrari, V., Tuytelaars, T., & Gool, L. V. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2003.1211424 · Cited by 119
Consistent segmentation for optical flow estimation
Zitnick, C. W., Jojic, N., & Kang, S. B. (2005) · Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 · doi:10.1109/iccv.2005.61 · Cited by 117 · cited 1×
“The region measurements are segmented regions (e.g. by segmentation algorithms such as [26] [33]).”
TILT: Transform Invariant Low-Rank Textures
Zhang, Z., Liang, X., Ganesh, A., & Ma, Y. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19318-7_25 · Cited by 100
Parsing Images of Architectural Scenes
Berg, A. C., Grabler, F., & Malik, J. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4409091 · Cited by 52 · cited 1×
“Given aklk (the appearance of b k lk ), we use methods such as [17, 31, 5] to estimate the confidence fapp(cr = c; a k lk) that Br is of class c.”
MethodToward coherent object detection and scene layout understanding
Bao, S. Y., Sun, M., & Savarese, S. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.08.001 · Cited by 38 · cited 3×
“For example, [27] uses point-region interaction, [19] uses point-object interaction, and [12] [15] [4] use object-region interaction.”
BackgroundSelf-citationWhere's Waldo: Matching people in images of crowds
Garg, R. D., Seitz, S. M., Ramanan, D., & Snavely, N. (2011) · doi:10.1109/cvpr.2011.5995546 · Cited by 24 · cited 1×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundScene shape from texture of objects
Payet, N., & Todorović, S. (2011) · doi:10.1109/cvpr.2011.5995326 · Cited by 17 · cited 1×
“Among these works (e.g. [12] [15] [11] [25] [10] [20] [6] [13]), most assume that only a single image is available, whereas a handful of them model this problem with multiple images.”
BackgroundSemantic structure from motion with object and point interactions
Bao, S. Y., Bagra, M., & Savarese, S. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130358 · Cited by 7 · cited 1×
“A recent extension of [3] is presented in [2] wherein the idea of capturing the relationship between points and objects is explored.”
BackgroundSelf-citation
Cited by
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., & Leonard, J. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2624754 · Cited by 3,596
“muddy-road while driving), move from path-planning to taskplanning, and enable advanced human-robot interaction [10],”
Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I. D., & Leonard, J. J. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-138892 · Cited by 3,048
DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments
Yu, C., Liu, Z., Liu, X.-J., Xie, F., Yang, Y., Wei, Q., & Fei, Q. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593691 · Cited by 1,154
“[14] first try to estimate camera pose, points, and objects jointly utilizing both geometry and semantic attributes in the scene, which significantly improve the object recognition accuracy.”
BackgroundSLAM++: Simultaneous Localisation and Mapping at the Level of Objects
Salas-Moreno, R. F., Newcombe, R. A., Strasdat, H., Kelly, P. H. J., & Davison, A. J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.178 · Cited by 945
“[3] represented a scene as a set of points, objects and regions in two-view SfM, solving ex-pensively and jointly in a graph optimisation for a labelling and reconstruction solution taking account of interactions between all scene entities.”
BackgroundPL-SLAM: Real-time monocular visual SLAM with points and lines
Pumarola, A., Vakhitov, A., Agudo, A., Sanfeliu, A., & Moreno-Noguer, F. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989522 · Cited by 482
“Scene prior information have been also exploited to provide a significant boost to SLAM systems [3], [4].”
Method
Metadata
- Authors
- Bao, S. Y., Bagra, M., Chao, Y.-W., & Savarese, S.
- Affiliations
- University of Michigan (US)
- ORCID
- Yu-Wei Chao
- Venue
- 2012 IEEE Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2012-06
- Pages
- 2703-2710
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1997452076
- Cited by
- 107
- References
- 35
- Citations by year
- 2025: 3 · 2024: 3 · 2023: 3 · 2022: 9 · 2021: 7 · 2020: 9 · 2019: 17 · 2018: 14 · 2017: 10 · 2016: 9 · 2015: 10 · 2014: 5 · 2013: 7 · 2012: 1
- FWCI
- 5.22
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Earth and Planetary Sciences › Geology › 3D Surveying and Cultural Heritage
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Keywords
- structure from motion, 3D object recognition, camera pose estimation, multi-view images
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.