Motion and Depth Augmented Semantic Segmentation for Autonomous Navigation
Abstract
Motion and depth provide critical information in autonomous driving and they are commonly used for generic object detection. In this paper, we leverage them for improving semantic segmentation. Depth cues can be useful for detecting road as it lies below the horizon line. There is also a strong structural similarity for different instances of different objects including buildings and trees. Motion cues are useful as the scene is highly dynamic with moving objects including vehicles and pedestrians. This work utilizes geometric information modelled by depth maps and motion cues represented by optical flow vectors to improve the pixel-wise segmentation task. A CNN architecture is proposed and the variations regarding the stage at which color, depth, and motion information are fused, e.g. early-fusion or mid-fusion, are systematically investigated. Additionally, we implement a multimodal fusion algorithm to maximize the benefit from all the information. The proposed algorithms are evaluated on Virtual-KITTI and Cityscapes datasets where results demonstrate enhanced performance with depth and flow augmentation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MDASS, using a VGG16 backbone, achieves an mIoU of 63.13 on the Cityscapes dataset for 2.5D semantic segmentation.
“| MDASS [108] | VGG16 [15] | 63.13 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [108]
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · see citing works
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.179 · Cited by 3,177 · see citing works
Two-Frame Motion Estimation Based on Polynomial Expansion
Farnebäck, G. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45103-x_50 · Cited by 3,010 · see citing works
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620 · see citing works
Deeper Depth Prediction with Fully Convolutional Residual Networks
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., & Navab, N. (2016) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.32 · Cited by 2,061 · see citing works
Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information
Hirschmüller, H. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.56 · Cited by 1,731 · see citing works
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Garg, R., B.G., V. K., Carneiro, G., & Reid, I. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_45 · Cited by 1,632 · see citing works
DeMoN: Depth and Motion Network for Learning Monocular Stereo
Ummenhofer, B., Zhou, H., Uhrig, J., Mayer, N., Ilg, E., Dosovitskiy, A., & Brox, T. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.596 · Cited by 777 · see citing works
UnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss
Meister, S., Hur, J., & Roth, S. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.12276 · Cited by 535 · see citing works
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
Zhu, A. Z., Yuan, L., Chaney, K. P., & Daniilidis, K. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.062 · Cited by 478 · see citing works
Jain, S. D., Xiong, B., & Grauman, K. L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.228 · Cited by 371 · see citing works
Scene Parsing with Multiscale Feature Learning, Purity Trees, and Optimal CoversPreprint
Farabet, C., Couprie, C., Najman, L. A., & LeCun, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.2160 · Cited by 136 · see citing works
MODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving
Siam, M., Mahgoub, H., Zahran, M., Yogamani, S., Jagersand, M., & El-Sallab, A. (2018) · 2018 21st International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569744 · Cited by 95 · see citing works
Self-citationOptical Flow Augmented Semantic Segmentation Networks for Automated Driving
Rashed, H., Yogamani, S. K., El-Sallab, A., Křížek, P., & Elhelw, M. (2019) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0007248301650172 · Cited by 21 · see citing works
Self-citationReal-Time Segmentation with Appearance, Motion and Geometry
Siam, M., Eikerdawy, S., Gamal, M., Abdel-Razek, M., Jägersand, M., & Zhang, H. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594088 · Cited by 15 · see citing works
Enhanced Target Tracking in UAV Imagery with P-N Learning and Structural Constraints
Siam, M., & ElHelw, M. (2013) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.81 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
1 of the 20 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Deep Polarization Cues for Transparent Object Segmentation
Kalra, A., Taamazyan, V., Rao, S. K., Venkataraman, K., Raskar, R., & Kadambi, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00863 · Cited by 148 · see citing works
RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving
El Madawi, K., Rashed, H., El Sallab, A., Nasr, O., Kamel, H., & Yogamani, S. (2019) · 2019 IEEE Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc.2019.8917447 · Cited by 116 · see citing works
Self-citationRashed, H., Ramzy, M., Vaquero, V., Sallab, A. E., Sistu, G., & Yogamani, S. K. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00293 · Cited by 116 · see citing works
Self-citationSurround-View Fisheye Camera Perception for Automated Driving: Overview, Survey & Challenges
Kumar, V. R., Eising, C., Witt, C., & Yogamani, S. K. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3235057 · Cited by 121 · see citing works
Self-citationRizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
FisheyeMODNet: Moving Object Detection on Surround-view Cameras for Autonomous Driving
Yahiaoui, M. (2019) · TU Dublin · doi:10.21427/v1ar-t994 · Cited by 41 · see citing works
Self-citationFull-Time Monocular Road Detection Using Zero-Distribution Prior of Angle of Polarization
Li, N., Zhao, Y., Pan, Q., Kong, S. G., & Chan, J. C.-W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_28 · Cited by 31 · see citing works
Mohamed, E., Ewaisha, M., Siam, M., Rashed, H., Yogamani, S. K., Hamdy, W., El-Dakdouky, M., & El-Sallab, A. (2021) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv48863.2021.9575445 · Cited by 31 · see citing works
Self-citationAnalyzing Factors Influencing Situation Awareness in Autonomous Vehicles—A Survey
Ignatious, H. A., El‐Sayed, H., Khan, M. A., & Mokhtar, B. M. (2023) · Sensors · doi:10.3390/s23084075 · Cited by 30 · see citing works
DeepFoveaNet: Deep Fovea Eagle-Eye Bioinspired Model to Detect Moving Objects
Pando, A. G., & Chacon-Murguia, M. I. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3101398 · Cited by 28 · see citing works
Das, A., Krizek, P., Sistu, G., Burger, F., Madasamy, S., Uricar, M., Kumar, V. R., & Yogamani, S. (2020) · 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294677 · Cited by 25 · see citing works
Self-citationRGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous DrivingPreprint
Madawy, K. E., Rashed, H., Sallab, A. E., Nasr, O. A., Kamel, H., & Yogamani, S. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00208 · Cited by 23 · see citing works
Self-citationRST-MODNet: Real-time Spatio-temporal Moving Object Detection for Autonomous DrivingPreprint
Ramzy, M., Rashed, H., Sallab, A. E., & Yogamani, S. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.00438 · Cited by 16 · see citing works
Self-citationBEV-MODNet: Monocular Camera based Bird's Eye View Moving Object Detection for Autonomous Driving
Rashed, H., Essam, M., Mohamed, M., Sallab, A. E., & Yogamani, S. K. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9564667 · Cited by 15 · see citing works
Self-citationMambaST: A Plug-and-Play Cross-Spectral Spatial-Temporal Fuser for Efficient Pedestrian Detection
Gao, X., Kanu-Asiegbu, A. M., & Du, X. (2024) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc58415.2024.10920115 · Cited by 9 · see citing works
Similarity-Aware Fusion Network for 3D Semantic Segmentation
Zhao, L., Lu, J., & Zhou, J. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636494 · Cited by 9 · see citing works
A Review of Using Visual Odometery Methods in Autonomous UAV Navigation in GPS-Denied Environment
Rostum, H. M., & Vásárhelyi, J. (2023) · Acta Universitatis Sapientiae Electrical and Mechanical Engineering · doi:10.2478/auseme-2023-0002 · Cited by 8 · see citing works
Barbato, F., Rizzoli, G., & Zanuttigh, P. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096314 · Cited by 7 · see citing works
Combining Motion and Appearance for Robust Probabilistic Object Segmentation in Real Time
Mengers, V., Battaje, A., Baum, M., & Brock, O. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160908 · Cited by 6 · see citing works
Hegde, N., Muralidhara, S., Schuster, R., & Stricker, D. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00541 · Cited by 5 · see citing works
LiDAR-BEVMTN: Real-Time LiDAR Bird’s-Eye View Multi-Task Perception Network for Autonomous Driving
Mohapatra, S., Yogamani, S. K., Kumar, V. R., Milz, S., Gotzig, H., & Mäder, P. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3510642 · Cited by 5 · see citing works
Self-citationVideoSAM: Open-World Video Segmentation
Guo, P., Zhao, Z., Gao, J., Wu, C., He, T., Zhang, Z., Xiao, T., & Zhang, W. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128804 · Cited by 4 · see citing works
SpikiLi: A Spiking Simulation of LiDAR based Real-time Object Detection for Autonomous Driving
Mohapatra, S., Mesquida, T., Hodaei, M., Yogamani, S. K., Gotzig, H., & Maeder, P. (2022) · International Conference on Event-based Control, Communication, and Signal Processing · doi:10.1109/ebccsp56922.2022.9845647 · Cited by 4 · see citing works
Self-citationEV-FuseMODNet: Moving Object Detection Using the Fusion of Event Camera and Frame Camera
Sun, H., Chen, S., Dao, M.-Q., & Frémont, V. (2023) · IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI) · doi:10.1109/soli60636.2023.10425107 · Cited by 3 · see citing works
Kerfa, D. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15849-1 · Cited by 3 · see citing works
Cambuim, L. F. S., Severino, J. B. J., & Barros, E. N. S. (2020) · IEEE International New Circuits and Systems Conference (NEWCAS) · doi:10.1109/newcas49341.2020.9159771 · Cited by 2 · see citing works
Neural Rendering Based Urban Scene Reconstruction For Autonomous Driving
Shen, S., Kerofsky, L. J., Kumar, V. R., & Yogamani, S. K. (2024) · Electronic Imaging · doi:10.2352/ei.2024.36.17.avm-365 · Cited by 2 · see citing works
Self-citationMoving Objects Segmentation in Infrared Scene Videos
Rai, M. C. E., Al-Saad, M., Darweesh, M., Mansoori, S. A., Al-Ahmad, H., & Mansoor, W. (2021) · International Conference on Signal Processing and Information Security (ICSPIS) · doi:10.1109/icspis53734.2021.9652436 · Cited by 2 · see citing works
Analysing the Impact of Vibrations on Smart Wheelchair Systems and Users
Mohamed, E., Sirlantzis, K., & Howells, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-09037-0_3 · Cited by 2 · see citing works
Thayalan, S., Srıdevı, M., Santhakumar, K., Kiran, B. R., Liu, H., Gauthier, T., & Yogamani, S. K. (2022) · Journal of Electronic Imaging · doi:10.1117/1.jei.32.1.011004 · Cited by 1 · see citing works
Self-citationDeep learning approach to vehicle pose estimation from polarimetric image data
Siefring, M. D., Ratliff, B. M., & Ballard, B. S. (2023) · Polarization Science and Remote Sensing · doi:10.1117/12.2677123 · Cited by 1 · see citing works
Learning Instance Motion Segmentation With Geometric Embedding
Leng, Z., Chen, J., & Lin, S. (2021) · IEEE Access · doi:10.1109/access.2021.3062673 · Cited by 1 · see citing works
Polarization based urban scenes understanding
Blanchon, M. (2021) · doi:10.70675/9eb0b15bzd1bcz473bza318zfc8c58cc92f1 · Cited by 1 · see citing works
Multi-task Learning for Visual Perception in Automated Driving
Chennupati, S. (2021) · Deep Blue (University of Michigan) · doi:10.7302/1030
Folorunsho, S. O., & Norris, W. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4756096
Lin, G., Chang, S., Liu, H., & Wang, N. (2024) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case59546.2024.10711685
- Authors
- Rashed, H., El Sallab, A., Yogamani, S., & ElHelw, M.
- Affiliations
- Valeo (France) (FR)
- Nile University (EG)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 364-370
- OpenAlex
- W2966669119
- Cited by
- 38
- References
- 20
- Citations by year
- 2025: 3 · 2024: 4 · 2023: 9 · 2022: 5 · 2021: 8 · 2020: 4 · 2019: 5
- FWCI
- 1.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- semantic segmentation, depth images, optical flow, motion cues, multimodal fusion, autonomous driving, autonomous navigation, road detection, feature-level fusion, cityscape
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
