SMSnet: Semantic motion segmentation using deep convolutional neural networks
Abstract
Interpreting the semantics and motion of objects are prerequisites for autonomous robots that enable them to reason and operate in dynamic real-world environments. Existing approaches that tackle the problem of semantic motion segmentation consist of long multistage pipelines and typically require several seconds to process each frame. In this paper, we present a novel convolutional neural network architecture that learns to predict both the object label and motion status of each pixel in an image. Given a pair of consecutive images, the network learns to fuse features from self-generated optical flow maps and semantic segmentation kernels to yield pixel-wise semantic motion labels. We also introduce the Cityscapes-Motion dataset which contains over 2,900 manually annotated semantic motion labels, which is the largest dataset of its kind so far. We demonstrate that our network outperforms existing approaches achieving state-of-the-art performance on the KITTI dataset, as well as in the more challenging Cityscapes-Motion dataset while being substantially faster than existing techniques.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multi-task learning is an efficient design pattern commonly used where most of the computation is shared across all tasks.
“Multi-task learning (MTL) is an efficient design pattern commonly used where most of the computation is shared across all the tasks [2], [3].”
From OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving (2021) · cites this paper as [2]
Cites
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,749 · see citing works
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.179 · Cited by 3,177 · see citing works
Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.438 · Cited by 2,488 · see citing works
Mayer, N., Ilg, E., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · Cited by 1,214 · see citing works
ParseNet: Looking Wider to See BetterPreprint
Liu, W., Rabinovich, A., & Berg, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04579 · Cited by 1,056 · see citing works
AdapNet: Adaptive semantic segmentation in adverse environmental conditions
Valada, A., Vertens, J., Dhall, A., & Burgard, W. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989540 · Cited by 209 · see citing works
Self-citationA General Framework for Tracking Multiple People from a Moving Camera
Choi, W., Pantofaru, C., & Savarese, S. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.248 · Cited by 207 · see citing works
Moving object segmentation by background subtraction and temporal analysis
Spagnolo, P., D’Orazio, T., Leo, M., et al. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2006.01.001 · Cited by 162 · see citing works
Moving object detection by multi-view geometric techniques from a single camera mounted robot
Kundu, A., Krishna, K. M., & Sivaswamy, J. (2009) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2009.5354227 · Cited by 156 · see citing works
Vision-Based Offline-Online Perception Paradigm for Autonomous Driving
Ros, G., Ramos, S., Granados, M., Bakhtiary, A., Vázquez, D., & López, A. M. (2015) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.38 · Cited by 123 · see citing works
Deep learning for human part discovery in images
Oliveira, G. L., Valada, A., Bollen, C., Burgard, W., & Brox, T. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487304 · Cited by 95 · see citing works
Self-citationObject-Level Motion Detection From Moving Cameras
Chen, T., & Lu, S. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2587387 · Cited by 63 · see citing works
Use of speed cues in the detection of moving objects by moving observers
Royden, C. S., & Moore, K. D. (2012) · Vision Research · doi:10.1016/j.visres.2012.02.006 · Cited by 59 · see citing works
Multimodal information fusion for urban scene understanding
Xu, P., Davoine, F., Bordes, J.-B., Zhao, H., & Denœux, T. (2014) · Machine Vision and Applications · doi:10.1007/s00138-014-0649-7 · Cited by 56 · see citing works
Moving object detection based on kirsch operator combined with Optical Flow
Gao, P., Sun, X., & Wang, W. (2010) · International Conference on Image Analysis and Signal Processing · doi:10.1109/iasp.2010.5476045 · Cited by 30 · see citing works
Object Detection, Tracking, and Motion Segmentation for Object-level Video SegmentationPreprint
Drayer, B., & Brox, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03066 · Cited by 30 · see citing works
Lin, T.-H., & Wang, C.-C. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907299 · Cited by 27 · see citing works
Moving object detection with moving background using optic flow
Patel, M. P., & Parmar, S. K. (2014) · International Conference on Recent Advances and Innovations in Engineering · doi:10.1109/icraie.2014.6909190 · Cited by 25 · see citing works
Semantic Motion Segmentation Using Dense CRF Formulation
Reddy, N. D., Singhal, P., & Krishna, K. M. (2014) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/2683483.2683539 · Cited by 22 · see citing works
Moving object segmentation using depth and optical flow in car driving sequences
Kao, J.-Y., Tian, D., Mansour, H., et al. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7532309 · Cited by 13 · see citing works
Semantic motion segmentation for urban dynamic scene understanding
Qiu, F., Yang, Y., Li, H., et al. (2016) · 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/coase.2016.7743446 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving
Yogamani, S., Hughes, C., Horgan, J., Sistu, G., Chennupati, S., Uricar, M., Milz, S., Simon, M., Amende, K., Witt, C., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00940 · Cited by 303 · see citing works
“This has been explored in [49, 57, 48] for narrow angle cameras.”
BackgroundOmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving
Ravi Kumar, V., Yogamani, S., Rashed, H., Sitsu, G., Witt, C., Leang, I., Milz, S., & Mader, P. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3062324 · Cited by 117 · see citing works
“For the motion segmentation, we use the annotations provided by DeepMotion [38] for Cityscapes and KITTI MoSeg [8].”
Method
- Authors
- Vertens, J., Valada, A., & Burgard, W.
- Affiliations
- University of Freiburg (DE)
- Venue
- 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2017-09
- Pages
- 582-589
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2774058387
- Cited by
- 69
- References
- 25
- Citations by year
- 2025: 4 · 2024: 4 · 2023: 4 · 2022: 9 · 2021: 20 · 2020: 6 · 2019: 15 · 2018: 7
- FWCI
- 2.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- object motion prediction, optical flow, semantic segmentation, autonomous robots, real-time motion estimation, convolutional neural network architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
