Object scene flow for autonomous vehicles
Abstract
This paper proposes a novel model and dataset for 3D scene flow estimation with an application to autonomous driving. Taking advantage of the fact that outdoor scenes often decompose into a small number of independently moving objects, we represent each element in the scene by its rigid motion parameters and each superpixel by a 3D plane as well as an index to the corresponding object. This minimal representation increases robustness and leads to a discrete-continuous CRF where the data term decomposes into pairwise potentials between superpixels and objects. Moreover, our model intrinsically segments the scene into its constituting dynamic components. We demonstrate the performance of our model on existing benchmarks as well as a novel realistic dataset with scene flow ground truth. We obtain this dataset by annotating 400 dynamic scenes from the KITTI raw data collection using detailed 3D CAD models for all vehicles in motion. Our experiments also reveal novel challenges which cannot be handled by existing methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
KITTI was the first large-scale dataset to tackle the important issue of multimodal data in autonomous vehicles.
“Summary KITTI [8,65-67] was the first large-scale dataset to tackle the important issue of multimodal data in autonomous vehicles.”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [67]
Cites
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
Self-citationVision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
Self-citationA Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
Scharstein, D., & Szeliski, R. (2002) · International Journal of Computer Vision · doi:10.1023/a:1014573219977 · Cited by 6,740 · see citing works
Marr, D. (1982) · Cumulative Index of Computer Aided Architectural Design · Cited by 5,784 · see citing works
Barlow, H. B. (1983) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(83)90030-5 · Cited by 5,502 · see citing works
Stereo Processing by Semiglobal Matching and Mutual Information
Hirschmüller, H. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1166 · Cited by 4,199 · see citing works
Non-parametric local transforms for computing visual correspondence
Zabih, R., & Woodfill, J. (1994) · Lecture notes in computer science · doi:10.1007/bfb0028345 · Cited by 1,797 · see citing works
Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
Brox, T., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.143 · Cited by 1,295 · see citing works
Convergent Tree-Reweighted Message Passing for Energy Minimization
Kolmogorov, V. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.200 · Cited by 1,100 · see citing works
StereoScan: Dense 3d reconstruction in real-time
Geiger, A., Ziegler, J., & Stiller, C. (2011) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2011.5940405 · Cited by 1,092 · see citing works
Self-citationSun, D., Roth, S., & Black, M. J. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0644-x · Cited by 698 · see citing works
Buehler, M., Iagnemma, K. D., & Singh, S. (2009) · Springer tracts in advanced robotics · doi:10.1007/978-3-642-03991-1 · Cited by 566 · see citing works
Vedula, S., Baker, S. J., Rander, P. W., et al. (1999) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1999.790293 · Cited by 443 · see citing works
Optimizing Binary MRFs via Extended Roof Duality
Rother, C., Kolmogorov, V., Lempitsky, V., & Szummer, M. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383203 · Cited by 424 · see citing works
3D Traffic Scene Understanding From Movable Platforms
Geiger, A., Lauer, M., Wojek, C., Stiller, C., & Urtasun, R. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.185 · Cited by 397 · see citing works
Self-citationShashua, A., Gdalyahu, Y., & Hayun, G. (2004) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2004.1336346 · Cited by 353 · see citing works
Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation
Yamaguchi, K., McAllester, D., & Urtasun, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_49 · Cited by 314 · see citing works
A Variational Method for Scene Flow Estimation from Stereo Sequences
Huguet, F., & Devernay, F. (2007) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4409000 · Cited by 296 · see citing works
Vedula, S., Rander, P. W., Collins, R. T., et al. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.63 · Cited by 284 · see citing works
Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score
Pons, J.-P., Keriven, R., & Faugeras, O. D. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-006-8671-5 · Cited by 279 · see citing works
Robust Multiperson Tracking from a Mobile Platform
Ess, A., Leibe, B., Schindler, K., & Gool, L. V. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.109 · Cited by 251 · see citing works
Vogel, C., Schindler, K., & Roth, S. (2013) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.174 · Cited by 246 · see citing works
6D-Vision: Fusion of Stereo and Motion for Robust Environment Perception
Franke, U., Rabe, C., Badino, H., & Gehrig, S. K. (2005) · Lecture notes in computer science · doi:10.1007/11550518_27 · Cited by 203 · see citing works
Stereoscopic Scene Flow Computation for 3D Motion Understanding
Wedel, A., Brox, T., Vaudrey, T., Rabe, C., Franke, U., & Cremers, D. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0404-0 · Cited by 197 · see citing works
Efficient Dense Scene Flow from Sparse or Dense Stereo Data
Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., & Cremers, D. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88682-2_56 · Cited by 176 · see citing works
RGB-D flow: Dense 3-D motion estimation using color and depth
Herbst, E. V., Ren, X., & Fox, D. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630885 · Cited by 153 · see citing works
Object stereo — Joint stereo matching and object segmentation
Bleyer, M., Rother, C., Kohli, P., et al. (2011) · doi:10.1109/cvpr.2011.5995581 · Cited by 143 · see citing works
Robust Monocular Epipolar Flow Estimation
Yamaguchi, K., McAllester, D. A., & Urtasun, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.243 · Cited by 136 · see citing works
Multi-view Scene Flow Estimation: A View Centered Variational Approach
Basha, T., Moses, Y., & Kiryati, N. (2012) · International Journal of Computer Vision · doi:10.1007/s11263-012-0542-7 · Cited by 108 · see citing works
Modeling Blurred Video with Layers
Wulff, J., & Black, M. J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_16 · Cited by 106 · see citing works
Over-Parameterized Variational Optical Flow
Nir, T., Bruckstein, A. M., & Kimmel, R. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0051-2 · Cited by 105 · see citing works
Scene flow estimation by growing correspondence seeds
Čech, J., Sànchez-Riera, J., & Horaud, R. (2011) · doi:10.1109/cvpr.2011.5995442 · Cited by 104 · see citing works
Dense, Robust, and Accurate Motion Field Estimation from Stereo Image Sequences in Real-Time
Rabe, C., Müller, T., Wedel, A., & Franke, U. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_42 · Cited by 103 · see citing works
Dense Semi-rigid Scene Flow Estimation from RGBD Images
Quiroga, J., Brox, T., Frédéric Devernay, A., & Crowley, J. L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10584-0_37 · Cited by 101 · see citing works
3D scene flow estimation with a rigid motion prior
Vogel, C., Schindler, K., & Roth, S. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126381 · Cited by 100 · see citing works
Joint Estimation of Motion, Structure and Geometry from Stereo Sequences
Valgaerts, L., Bruhn, A., Zimmer, H., Weickert, J., Stoll, C., & Theobalt, C. (2010) · European Conference on Computer Vision · doi:10.1007/978-3-642-15561-1_41 · Cited by 95 · see citing works
A Fully-Connected Layered Model of Foreground and Background Flow
Sun, D., Wulff, J., Sudderth, E. B., Pfister, H., & Black, M. J. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.317 · Cited by 86 · see citing works
SphereFlow: 6 DoF Scene Flow from RGB-D Pairs
Hornáček, M., Fitzgibbon, A. W., & Rother, C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.451 · Cited by 73 · see citing works
View-Consistent 3D Scene Flow Estimation over Multiple Frames
Vogel, C. A., Roth, S., & Schindler, K. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_18 · Cited by 49 · see citing works
Extracting 3D Scene-Consistent Object Proposals and Depth from Stereo Images
Bleyer, M., Rhemann, C., & Rother, C. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_34 · Cited by 42 · see citing works
Unsupervised Learning of Stereo Vision with Monocular Cues
Trinh, H. D., & McAllester, D. (2009) · British Machine Vision Conference (BMVC) · doi:10.5244/c.23.72 · Cited by 31 · see citing works
Preserving Modes and Messages via Diverse Particle Selection
Pacheco, J. L., Zuffi, S., Black, M. J., & Sudderth, E. B. (2014) · Cited by 17 · see citing works
Weale, R. A. (1983) · The Quarterly Review of Biology · doi:10.1086/413352 · Cited by 16 · see citing works
CP-Census: A Novel Model for Dense Variational Scene Flow from RGB-D Data
Ferstl, D., Rüther, M., & Bischof, H. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.18 · Cited by 10 · see citing works
Scene Flow Estimation using Intelligent Cost Functions
Hadfield, S., & Bowden, R. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.108 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.438 · Cited by 2,488 · see citing works
Wang, Y., Chao, W.-L., Garg, D., Hariharan, B., Campbell, M., & Weinberger, K. Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00864 · Cited by 1,154 · see citing works
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation
Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., & Hajishirzi, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_34 · Cited by 1,029 · see citing works
Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924 · see citing works
Self-citationUnsupervised deep learning for optical flow estimation
Ren, Z., Yan, J., Ni, B., Liu, B., Yang, X., & Zha, H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10723 · Cited by 296 · see citing works
Luo, Y., Ren, J., Lin, M., Pang, J., Sun, W., Li, H., & Lin, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00024 · Cited by 205 · see citing works
What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
Mayer, N., Ilg, E., Fischer, P., Hazirbas, C., Cremers, D., Dosovitskiy, A., & Brox, T. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1082-6 · Cited by 191 · see citing works
A survey on face data augmentation for the training of deep neural networks
Wang, X., Wang, K., & Lian, S. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04748-3 · Cited by 141 · see citing works
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning
Milz, S., Arbeiter, G., Witt, C., Abdallah, B., & Yogamani, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00062 · Cited by 107 · see citing works
CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow
Weinzaepfel, P., Lucas, T., Leroy, V., Cabon, Y., Arora, V., Brégier, R., Csurka, G., Antsfeld, L., Chidlovskii, B., & Revaud, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01647 · Cited by 110 · see citing works
MODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving
Siam, M., Mahgoub, H., Zahran, M., Yogamani, S., Jagersand, M., & El-Sallab, A. (2018) · 2018 21st International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569744 · Cited by 95 · see citing works
Mapping Two Decades of Autonomous Vehicle Research: A Systematic Scientometric Analysis
Faisal, A., Yigitcanlar, T., Kamruzzaman, M., & Paz, A. (2021) · Journal of Urban Technology · doi:10.1080/10630732.2020.1780868 · Cited by 85 · see citing works
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
SELMA: SEmantic Large-Scale Multimodal Acquisitions in Variable Weather, Daytime and Viewpoints
Testolina, P., Barbato, F., Michieli, U., Giordani, M., Zanuttigh, P., & Zorzi, M. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3257086 · Cited by 41 · see citing works
On failures of RGB cameras and their effects in autonomous driving applications
Secci, F., & Ceccarelli, A. (2020) · 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre5003.2020.00011 · Cited by 42 · see citing works
Optical Flow Estimation with Deep Learning, a Survey on Recent Advances
Savian, S., Elahi, M., & Tillo, T. (2020) · Unsupervised and Semi-Supervised Learning · doi:10.1007/978-3-030-32583-1_12 · Cited by 29 · see citing works
- Authors
- Menze, M., & Geiger, A.
- Affiliations
- Leibniz University Hannover (DE)
- ORCID
- Geiger, Andreas
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3061-3070
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1921093919
- Cited by
- 2,270
- References
- 45
- Citations by year
- 2026: 110 · 2025: 217 · 2024: 275 · 2023: 263 · 2022: 248 · 2021: 329 · 2020: 260 · 2019: 219 · 2018: 151 · 2017: 120 · 2016: 70 · 2015: 8
- FWCI
- 47.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Keywords
- 3D scene flow, autonomous driving, superpixels, scene segmentation, KITTI dataset, outdoor scenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
