Cutting Edge: Soft Correspondences in Multimodal Scene Parsing
Abstract
Exploiting multiple modalities for semantic scene parsing has been shown to improve accuracy over the single modality scenario. Existing methods, however, assume that corresponding regions in two modalities have the same label. In this paper, we address the problem of data misalignment and label inconsistencies, e.g., due to moving objects, in semantic labeling, which violate the assumption of existing techniques. To this end, we formulate multimodal semantic labeling as inference in a CRF, and introduce latent nodes to explicitly model inconsistencies between two domains. These latent nodes allow us not only to leverage information from both domains to improve their labeling, but also to cut the edges between inconsistent regions. To eliminate the need for hand tuning the parameters of our model, we propose to learn intra-domain and inter-domain potential functions from training data. We demonstrate the benefits of our approach on two publicly available datasets containing 2D imagery and 3D point clouds. Thanks to our latent nodes and our learning strategy, our method outperforms the state-of-the-art in both cases.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many different methods are available, such as soft correspondences and 3D scene analysis from RGB-D data, to ensure dense and accurate scene parsing of indoor and outdoor environments.
“Many different methods are available, such as soft correspondences [94], 3D scene analysis from RGB-D data [95], to ensure dense and accurate scene parsing of indoor and outdoor environments.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [94]
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Mean shift: a robust approach toward feature space analysis
Comaniciu, D., & Meer, P. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.1000236 · Cited by 11,417 · see citing works
Graphical Models, Exponential Families, and Variational Inference
Wainwright, M. J., & Jordan, M. I. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601981851 · Cited by 3,063 · see citing works
Graphical Models, Exponential Families, and Variational Inference
Wainwright, M. J., & Jordan, M. I. (2008) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000001 · Cited by 1,842 · see citing works
Robust Higher Order Potentials for Enforcing Label Consistency
Kohli, P., Ladický, Ľ., & Torr, P. H. S. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-008-0202-0 · Cited by 858 · see citing works
Semantic Labeling of 3D Point Clouds for Indoor Scenes
Koppula, H. S., Anand, A., Joachims, T., & Saxena, A. (2011) · neural information processing systems · Cited by 322 · see citing works
What, Where and How Many? Combining Object Detectors and CRFs
Ladický, Ľ., Sturgess, P., Alahari, K., Russell, C., & Torr, P. H. S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_31 · Cited by 312 · see citing works
Holistic Scene Understanding for 3D Object Detection with RGBD Cameras
Lin, D., Fidler, S., & Urtasun, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.179 · Cited by 253 · see citing works
Parameter Learning and Convergent Inference for Dense Random Fields
Kraehenbuehl, P., & Koltun, V. (2013) · International Conference on Machine Learning · Cited by 170 · see citing works
Multiple view semantic segmentation for street view images
Xiao, J., & Quan, L. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459249 · Cited by 150 · see citing works
Context Driven Scene Parsing with Attention to Rare Classes
Yang, J., Price, B. L., Cohen, S. D., & Yang, M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.415 · Cited by 148 · see citing works
Learning Graphical Model Parameters with Approximate Marginal Inference
Domke, J. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.31 · Cited by 136 · see citing works
Learning Probabilistic Models for Contour Completion in Natural Images
Ren, X., Fowlkes, C. C., & Malik, J. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0092-6 · Cited by 98 · see citing works
Learning to Combine Bottom-Up and Top-Down Segmentation
Levin, A., & Weiss, Y. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0166-0 · Cited by 96 · see citing works
NON-ASSOCIATIVE MARKOV NETWORKS FOR 3D POINT CLOUD CLASSIFICATION
Шаповалов, Р., Velizhev, A., Barinova, O., & Gory, L. (2010) · Cited by 95 · see citing works
Nonparametric Scene Parsing with Adaptive Feature Relevance and Semantic Context
Singh, G., & Košecká, J. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.405 · Cited by 87 · see citing works
Contextually Guided Semantic Labeling and Search for 3D Point CloudsPreprint
Anand, A., Koppula, H. S., Joachims, T., & Saxena, A. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.5358 · Cited by 61 · see citing works
Non-associative Higher-Order Markov Networks for Point Cloud Classification
Najafi, M., Namin, S. T., Salzmann, M., & Petersson, L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_33 · Cited by 55 · see citing works
Self-citationCo-inference for Multi-modal Scene Analysis
Munoz, D., Bagnell, J. A., & Hebert, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33783-3_48 · Cited by 49 · see citing works
Semantic segmentation with heterogeneous sensor coverages
Cadena, C., & Košecká, J. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907237 · Cited by 44 · see citing works
Fast Probabilistic Labeling of City Maps
Posner, I., Cummins, M., & Newman, P. M. (2008) · doi:10.15607/rss.2008.iv.003 · Cited by 33 · see citing works
A 3D laser and vision based classifier
Douillard, B., Brooks, A., & Ramos, F. T. (2009) · International Conference on Intelligent Sensors, Sensor Networks and Information Processing · doi:10.1109/issnip.2009.5416828 · Cited by 29 · see citing works
A Multi-modal Graphical Model for Scene Analysis
Namin, S. T., Najafi, M., Salzmann, M., & Petersson, L. (2015) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.139 · Cited by 20 · see citing works
Self-citationZhang, H., Wang, J., Fang, T., & Quan, L. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2348795 · Cited by 16 · see citing works
Efficiently Learning Random Fields for Stereo Vision with Sparse Message Passing
Weinman, J. J., Tran, L., & Pal, C. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88682-2_47 · Cited by 14 · see citing works
Flagged first, then by citations
3 of the 29 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
What am I touching? Learning to classify terrain via haptic sensing
Bednarek, J., Bednarek, M., Wellhausen, L., Hutter, M., & Walas, K. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794478 · Cited by 40 · see citing works
Deep neural networks for terrain recognition task
Kozlowski, P., & Walas, K. (2018) · Baltic URSI Symposium (URSI) · doi:10.23919/ursi.2018.8406736 · Cited by 15 · see citing works
Multimodal Visual Data Registration for Web-Based Visualization in Media Production
Kim, H., Evans, A., Blat, J., & Hilton, A. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2642825 · Cited by 9 · see citing works
Depth data fusion for simultaneous localization and mapping — RGB-DD SLAM
Walas, K., Nowicki, M., Ferstl, D., & Skrzypczyński, P. (2016) · IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) · doi:10.1109/mfi.2016.7849459 · Cited by 8 · see citing works
2D-3D semantic segmentation using cardinality as higher-order loss
Namin, S. R., Álvarez, J. M., & Petersson, L. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7900222 · Cited by 1 · see citing works
Self-citationMulti-Object Recognition Method Inspired by Multimodal Information Processing in the Human Brain
Seki, R., Kominami, D., Shimonishi, H., Murata, M., & Fujiwaka, M. (2022) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps56602.2022.10008732 · Cited by 1 · see citing works
Big Multimodal Visual Data Registration for Digital Media Production
Kim, H., & Hilton, A. (2019) · Springer eBooks · doi:10.1007/978-3-319-97598-6_11
Seki, R., Kominami, D., Shimonishi, H., Murata, M., & Fujiwaka, M. (2022) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc55113.2022.9825068
- Authors
- Namin, S. T., Najafi, M., Salzmann, M., & Petersson, L.
- Affiliations
- Australian National University (AU)
- Data61 (AU)
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 1188-1196
- OpenAlex
- W2204747942
- Cited by
- 9
- References
- 29
- Citations by year
- 2022: 2 · 2021: 1 · 2019: 2 · 2018: 1 · 2016: 3
- FWCI
- 0.55
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- semantic annotation, label inconsistency, conditional random fields, 3D point clouds, moving objects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
