Large-scale semantic mapping and reasoning with heterogeneous modalities
The DOI matches its Crossref record; 24 of the 27 references it cites were found in a registry or an index.
Abstract
This paper presents a probabilistic framework combining heterogeneous, uncertain, information such as object observations, shape, size, appearance of rooms and human input for semantic mapping. It abstracts multi-modal sensory information and integrates it with conceptual common-sense knowledge in a fully probabilistic fashion. It relies on the concept of spatial properties which make the semantic map more descriptive, and the system more scalable and better adapted for human interaction. A probabilistic graphical model, a chaingraph, is used to represent the conceptual information and perform spatial reasoning. Experimental results from online system tests in a large unstructured office environment highlight the system's ability to infer semantic room categories, predict existence of objects and values of other spatial properties as well as reason about unexplored space.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Speeded-Up Robust Features (SURF)
Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. (2008) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2007.09.014 · Cited by 13,557
Context-based vision system for place and object recognition
Torralba, Murphy, Freeman, & Rubin (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238354 · Cited by 867
The Spatial Semantic Hierarchy
Kuipers, B. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00017-5 · Cited by 726
Towards semantic maps for mobile robots
Nüchter, A., & Hertzberg, J. (2008) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.08.001 · Cited by 406
Multi-hierarchical semantic maps for mobile robotics
Galindo, C., Saffiotti, A., Coradeschi, S., Buschka, P., Fernandez-Madrigal, J. A., & Gonzalez, J. (2005) · 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2005.1545511 · Cited by 308
Conceptual spatial representations for indoor mobile robots
Zender, H., Martínez Mozos, O., Jensfelt, P., Kruijff, G.-J. M., & Burgard, W. (2008) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.03.007 · Cited by 300
Self-citationHighly scalable appearance-only SLAM - FAB-MAP 2.0
Cummins, M., & Newman, P. (2009) · Robotics: Science and Systems V · doi:10.15607/rss.2009.v.039 · Cited by 282
Chain Graph Models and their Causal Interpretations
Lauritzen, S. L., & Richardson, T. S. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00340 · Cited by 265
libDAI: A Free and Open Source C++ Library for Discrete Approximate Inference in Graphical Models
Mooij, J. M. (2010) · Max Planck Institute for Plasma Physics · doi:10.5555/1756006.1859925 · Cited by 227
Curious George: An attentive semantic robot
Meger, D., Forssén, P.-E., Lai, K., Helmer, S., McCann, S., Southey, T., Baumann, M., Little, J. J., & Lowe, D. G. (2008) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.03.008 · Cited by 167
Research works printed but not linked: 1
- mo?rwald, 0, BLORT - The blocks world robotic vision toolbox, ICRA Workshop Best Practice in 3D Perc and Model for Mobile Manipul 2010
Cited by
Items 201–247 of 247
Kaszuba, S., Sabbella, S. R., Suriani, V., Riccio, F., & Nardi, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.07938 · Cited by 3
“Finally, Pronobis et al. [17], represent a conceptual map as a probabilistic chain graph model and evaluate their method by comparing the robot belief to be in a certain location against the ground truth.”
Information fusion for decision making
Saha, H. (2021) · doi:10.31274/td-20240329-429 · Cited by 2
Semantic Localization through Propagation of Scene Information in a Hierarchical Model
Gómez, C., Hernandez, A. C., Derner, E., & Barber, R. (2019) · European Conference on Mobile Robots · doi:10.1109/ecmr.2019.8870972 · Cited by 1
“More recently, a mapping system to enable spatial understanding was proposed in [12].”
BackgroundA Semantic Mapping Framework for Service Robots
Frota, L., & Rocha, R. P. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707779 · Cited by 1
“A common downside of these approaches is the more extensive memory requirement, if compared with works based on topological maps for semantic representation [15] , thus having a more compact and efficient representation.”
Sampling-Based Path Planning and Semantic Navigation for Complex Large-Scale Environments
Ahmad, S., & Humbert, J. S. (2025) · Robotics · doi:10.3390/robotics14110149 · Cited by 2
“A large amount of semantic mapping literature uses metric maps to add semantic information on top of them [59,61–63].”
Knowledge-Based Reasoning on Semantic Maps
Capobianco, R., Gemignani, G., Nardi, D., Bloisi, D. D., & Iocchi, L. (2014) · Cited by 1
Guess from Far, Recognize when Near
Karthik, M. S., Mittal, S., & Krishna, K. M. (2014) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/2683483.2683544 · Cited by 1
“Present object search paradigms [3] cater to the cases where the objects are close to the camera, large in size and are generally lying on tables in an environment, which is small.”
BackgroundContexts for Symbiotic Autonomy: Semantic Mapping, Task Teaching and Social Robotics.
Capobianco, R., Gemignani, G., Iocchi, L., Nardi, D., Riccio, F., & Vanzo, A. (2016) · IRIS Research product catalog (Sapienza University of Rome) · Cited by 1
Indoor Environment Semantic Topological Mapping Based on Deep Learning
Zhao, L., Luo, P., Zhao, Z., & Dong, L. F. (2018) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar.2018.8621692 · Cited by 1
Capturing Frame-Like Object Descriptors in Human Augmented Mapping
Faridghasemnia, M., Vanzo, A., & Nardi, D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-35166-3_28 · Cited by 1
Towards Abstract Relational Learning in Human Robot InteractionPreprint
Faridghasemnia, M., Nardi, D., & Saffiotti, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10364 · Cited by 2
Semantic Mapping for Robotics: Survey, Trends and ChallengesPreprint
Ferreira, B. G., Sousa, A. J., & Reis, L. P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5085045 · Cited by 3
Nashed, S. B., & Biswas, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11495
Exploiting structure in man-made environments
Aydemir, A. (2012) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Two-dimensional local ternary patterns using synchronized images for outdoor place categorization
Jung, H., Kurazume, R., Iwashita, Y., & Mozos, Ó. M. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7026158
“This type of information can greatly improve communication between robots and humans[1, 2] and allows robots to make decisions with context-based understanding when completing high-level tasks[3].”
BackgroundObject guided autonomous exploration for mobile robots in indoor environments
Nieto-Granda, C., Choudhary, S., Rogers, J. G., Twigg, J., Murali, V., & Christensen, H. I. (2014) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2050818
Learning Semantic Maps Through Dialog for A Voice-Commandable Wheelchair
Hemachandra, S. M., & Walter, M. R. (2014)
Learning the state of the world: object-based world modeling for mobile manipulation robots
Wong, L. L. S. (2017) · AI Matters · doi:10.1145/3054837.3054845 · Cited by 1
Speech recognition based building semantic map method on Aldebaran Nao
Liu, S., Xing, Q., Han, Y., Zhao, Y., & Li, R. (2017) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc.2017.7979119
Recognizing outdoor scenes by convolutional features of omni-directional LiDAR scans
Nakashima, K., Nham, S., Jung, H., Iwashita, Y., Kurazume, R., & Mozos, Ó. M. (2017) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii.2017.8279243
“Information about the place greatly improves communication between robots and humans [1], [2].”
BackgroundTaniguchi, A., Hagiwara, Y., Taniguchi, T., & Inamura, T. (2018) · arXiv (Cornell University)
A Sensor Group Selection Mechanism Based on RGB Channels Scenario Sensing Method
Song, B., Shi, D., & Wang, H. (2018) · doi:10.1145/3297156.3297164
Similarity Reasoning over Semantic Context-Graphs
Boteanu, A. (2015) · Digital WPI
Leveraging Semantics for Incremental Learning in Multi-Relational EmbeddingsPreprint
Daruna, A., Liu, W., Kira, Z., & Chernova, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12181
“1 Introduction Robots operating in human environments benefit from using knowledge representations that encode information in semantically meaningful ways to facilitate generalization and adaptability, leading to more robust task execution [1, 2, 3].”
BackgroundA Target Position Decision System based on Eyewitness Account for Exploring Disaster Environment
Yang, K.-M., Han, J.-B., & Seo, K.-H. (2019) · IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO) · doi:10.1109/arso46408.2019.8948787
“It made object observations, shape, size, and appearance of rooms for semantic mapping [3].”
MethodLiu, Z. (n.d.) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich)
Object Classification and Semantic Mapping
Atman, J., & Trommer, G. F. (2020) · IEEE/ION Position Location and Navigation Symposium · doi:10.1109/plans46316.2020.9110198 · Cited by 1
“In [3] the ability to associate, among other things, object properties with a certain place using laser and camera sensors is presented.”
BackgroundArtificial Neural Networks Based Place Categorization
Kriegler, A., Wöber, W., & Aburaia, M. (2020) · Lecture notes in mechanical engineering · doi:10.1007/978-3-030-62784-3_17
An Image-based Approach of Task-driven Driving Scene CategorizationPreprint
Hu, S., Fan, H., Gao, B., Zhao, X., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05920
“[6] combines object observations, the shape, size, and appearance of rooms into a chain-graph to represent the semantic information and perform inference.”
Research on Semantic Vision SLAM Towards Dynamic Environment
Bai, N., Ma, T., Shi, W., & Wang, L. (2021) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-77569-8_7 · Cited by 1
Efficient Placard Discovery for Semantic Mapping During Frontier ExplorationPreprint
Balaban, D., Jagannathan, H., Liu, H. X., & Hart, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14742
Fast and robust visual SLAM for dynamic environments
Gaurav, S. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/155251 · Cited by 1
A Contrastive-Learning-Based Method for Alert-Scene Categorization
Hu, S., Fan, H., Gao, B., & Zhao, H. (2022) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv51971.2022.9827387
Leveraging deep-learned object detectors for real-world semantic localization and mapping
Nicholson, L. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.242733 · Cited by 2
Map representations for mobile robot localization
Zhang, M., & Su, Z. (2024) · International Conference on Mechanical Engineering, Intelligent Manufacturing, and Automation Technology (MEMAT) · doi:10.1117/12.3026290
“Pronobis et al. [45] put forward the hierarchical structure of spatial knowledge, as shown in Figure 13, which contains four layers: sensory layer, position layer, classification layer, and concept layer.”
BackgroundRF-SLAM: Fusing RFID and LiDAR SLAM for Item-Level Semantic Mapping
Chen, X., Liu, J., Hu, H., Li, S. Z., Li, X., & Lin, Z. (2025) · International Workshop on Quality of Service · doi:10.1109/iwqos65803.2025.11143373
“Pronobis et al. [24] propose an online system that builds a semantic map of the environment using laser as well as cameras.”
Mobile Robot Place Segmentation and Categorization Using Object Semantics
Moncada-Ramirez, J., Ruiz-Sarmiento, J.-R., Galindo, C., & González-Jiménez, J. (2025) · European Conference on Mobile Robots · doi:10.1109/ecmr65884.2025.11163204
“Some methods rely on geometric information to exploit the structural layout of the environment [8], [9].”
Long-Term Dynamic Object Relocalization for Mobile Robots in Human–Robot Coexisting Environments
Zhang, Y., Bi, W., Yin, M., Qu, H., Zhang, C., Hua, C., & Wen, G. (2026) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2025.3648568 · Cited by 4
“In another direction, Pronbis et al. [18] guided search by integrating conceptual common sense knowledge, which aligns the search process with human semantic expectations.”
Niu, Q. (2025) · International Conference on Mechanical Automation and Electronic Information Engineering (MAEIE) · doi:10.1109/maeie68099.2025.11406022
“In terms of interaction-driven requirements, Pronobis et al [5] . developed a heterogeneous modal semantic reasoning framework that uses natural language instructions to enhance the environmental semantic annotation capabilities of SLAM systems, improving robot task planning efficiency, as…”
Hybrid mapping for large urban environments
Üzer, F. (2016) · theses.fr (ABES) · doi:10.70675/8f51d6f5z6aecz435ez93fcz9b89660ff07b · Cited by 1
Breux, Y. (2018) · theses.fr (ABES) · doi:10.70675/c9481f77zf7ffz4591z9266zb6864ef0c150 · Cited by 1
Smart Automation and Robotics in Healthcare: Transforming Hospital Material Handling Systems
Alomran, O., & Yang, H. (2025) · Internet of things · doi:10.1007/978-3-032-02709-2_13
Achour, A. (2024) · theses.fr (ABES) · doi:10.70675/e91c896bzcd76z4018z951bz0f1c8f9c5dba
Chat2Route: A Retrieval-Augmented System for Personalized and Context-Aware Travel Planning
Li, C., Chen, Y., Liu, H., Guo, Z., Wang, X., Zeng, W., Zhang, M., & Li, L. (2026) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2026.3705615
“Early semantic frameworks include semantic mapping [48], [49] and ontological reasoning [50], [51]: embedded contextual knowledge structurally but remained limited by static features.”
Unified Semantic Reasoning and Planning for Autonomous Driving
An, Y., Choi, J.-H., Pyo, J.-W., Bae, S.-H., & Kuc, T. (2026) · Sensors · doi:10.3390/s26185973
“…semantic SLAM attach class labels and object structure to geometric maps [35], through dense per-pixel fusion [36], LiDAR semantic registration [37], and segment-level descriptors for place recognition [38], and multi-hierarchical semantic maps couple spatial layers with symbolic reasoning [39,40].”
SemGraph-Route for Safe Zero-Shot UAV Navigation in GPS-Denied Environments
Javaid, S., Saeed, N., & He, B. (2026) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2026.3739067
“Semantic room-type classification has also been explored for indoor navigation [25], establishing that spatial categories such as corridors and open spaces remain structurally stable across building types and sensor modalities.”
Metadata
- Authors
- Pronobis, A., & Jensfelt, P.
- Affiliations
- KTH Royal Institute of Technology (SE)
- ORCID
- Jensfelt, Patric
- Venue
- 2012 IEEE International Conference on Robotics and Automation
- Publisher
- IEEE
- Published
- 2012-05
- Pages
- 3515-3522
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2000321478
- Cited by
- 248
- References
- 27
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 6 · 2023: 6 · 2022: 20 · 2021: 17 · 2020: 23 · 2019: 16 · 2018: 34 · 2017: 27 · 2016: 22 · 2015: 23 · 2014: 25 · 2013: 14 · 2012: 1
- FWCI
- 37.56
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- semantic mapping, heterogeneous modalities, multimodal sensor fusion, probabilistic graphical models, chain graphs, spatial reasoning, commonsense knowledge, object prediction, spatial characteristics, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.