A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
Open accessAbstract
Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluation of scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Adding CityScapes unlabeled videos to KITTI training further improves monocular depth estimation performance.
“Furthermore, we show that by simply introducing an additional source of unlabeled videos, such as the publicly available CityScapes dataset (CS+K) [9], we are able to further improve monocular depth estimation performance.”
From 3D Packing for Self-Supervised Monocular Depth Estimation (2020) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · see citing works
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
Scharstein, D., & Szeliski, R. (2002) · International Journal of Computer Vision · doi:10.1023/a:1014573219977 · Cited by 6,740 · see citing works
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,748 · see citing works
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · see citing works
Self-citationStereo Processing by Semiglobal Matching and Mutual Information
Hirschmüller, H. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1166 · Cited by 4,199 · see citing works
Object scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270 · see citing works
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842 · see citing works
High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth
Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., & Westling, P. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11752-2_3 · Cited by 1,377 · see citing works
Stereo Matching by Training a Convolutional Neural Network to Compare Image PatchesPreprint
ŽbontarJure, & LeCunYann (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.05970 · Cited by 1,009 · see citing works
IM2GPS: estimating geographic information from a single image
Hays, J. H., & Efros, A. A. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587784 · Cited by 902 · see citing works
SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels
Xiao, J., Owens, A., & Torralba, A. B. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.458 · Cited by 805 · see citing works
Learning to generate chairs with convolutional neural networks
Dosovitskiy, A., Springenberg, J. T., & Brox, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298761 · Cited by 640 · see citing works
Self-citationCross-Based Local Stereo Matching Using Orthogonal Integral Images
Zhang, K., Lu, J., & Lafruit, G. (2009) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2009.2020478 · Cited by 504 · see citing works
Vedula, S., Baker, S. J., Rander, P. W., et al. (1999) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1999.790293 · Cited by 443 · see citing works
A Variational Method for Scene Flow Estimation from Stereo Sequences
Huguet, F., & Devernay, F. (2007) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4409000 · Cited by 296 · see citing works
Vedula, S., Rander, P. W., Collins, R. T., et al. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.63 · Cited by 284 · see citing works
3D Scene Flow Estimation with a Piecewise Rigid Scene Model
Vogel, C., Schindler, K., & Roth, S. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0806-0 · Cited by 246 · see citing works
Stereoscopic Scene Flow Computation for 3D Motion Understanding
Wedel, A., Brox, T., Vaudrey, T., Rabe, C., Franke, U., & Cremers, D. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0404-0 · Cited by 197 · see citing works
Self-citationSemantically-enriched 3D models for common-sense knowledge
Savva, M., Chang, A. X., & Hanrahan, P. (2015) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301289 · Cited by 104 · see citing works
Scene flow estimation by growing correspondence seeds
Čech, J., Sànchez-Riera, J., & Horaud, R. (2011) · doi:10.1109/cvpr.2011.5995442 · Cited by 104 · see citing works
A multi-block-matching approach for stereo
Einecke, N., & Eggert, J. P. (2015) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2015.7225748 · Cited by 42 · see citing works
Scene flow by tracking in intensity and depth data
Quiroga, J., Frédéric Devernay, A., & Crowley, J. L. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6239237 · Cited by 24 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- [6][6] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. NIPS , 2014. 3 -
- [9][9] R. Hartley and A. Zisserman. Multiple view geometry in computer vision . Cambridge university press, 2003. 12 -
- [14][14] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR , 2015. 5 -
- [15][15] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , pages 1106-1114, 2012. 2 -
- [20][20] M. Savva, A. X. Chang, and P. Hanrahan. SemanticallyEnriched 3D Models for Common-sense Knowledge. CVPR 2015 Workshop on Functionality, Physics, Intentionality and Causality , 2015. 4 -
- [25][25] C. Vogel, K. Schindler, and S. Roth. Piecewise rigid scene flow. In ICCV , 2013. 3 -
- [26][26] A. Wedel, C. Rabe, T. Vaudrey, T. Brox, U. Franke, and D. Cremers. Efficient dense scene flow from sparse or dense stereo data . Springer, 2008. 3 -
Cited by
Sorted by citations
Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources
Zhu, X. X., Tuia, D., Mou, L., Xia, G.-S., Zhang, L., Xu, F., & Fraundorfer, F. (2017) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2017.2762307 · Cited by 3,400 · see citing works
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · see citing works
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · see citing works
Digging Into Self-Supervised Monocular Depth Estimation
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00393 · Cited by 2,561 · see citing works
Wang, Y., Chao, W.-L., Garg, D., Hariharan, B., Campbell, M., & Weinberger, K. Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00864 · Cited by 1,154 · see citing works
Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924 · see citing works
Self-citationUnsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints
Mahjourian, R., Wicke, M., & Angelova, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00594 · Cited by 775 · see citing works
3D Packing for Self-Supervised Monocular Depth Estimation
Guizilini, V., Ambrus, R., Pillai, S., Raventos, A., & Gaidon, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00256 · Cited by 741 · see citing works
BMN: Boundary-Matching Network for Temporal Action Proposal Generation
Lin, T., Liu, X., Li, X., Ding, E., & Wen, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00399 · Cited by 694 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Learning Depth with Convolutional Spatial Propagation Network
Cheng, X., Wang, P., & Yang, R. (2018) · arXiv · doi:10.48550/arxiv.1810.02695 · Cited by 345 · see citing works
Consistent video depth estimation
Luo, X., Huang, J.-B., Szeliski, R., Matzen, K., & Kopf, J. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392377 · Cited by 322 · see citing works
Self-Supervised Monocular Depth Hints
Watson, J., Firman, M., Brostow, G., & Turmukhambetov, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00225 · Cited by 275 · see citing works
Luo, Y., Ren, J., Lin, M., Pang, J., Sun, W., Li, H., & Lin, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00024 · Cited by 205 · see citing works
What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
Mayer, N., Ilg, E., Fischer, P., Hazirbas, C., Cremers, D., Dosovitskiy, A., & Brox, T. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1082-6 · Cited by 191 · see citing works
Self-citationUnsupervised Adversarial Depth Estimation Using Cycled Generative Networks
Pilzer, A., Xu, D., Puscas, M., Ricci, E., & Sebe, N. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00073 · Cited by 192 · see citing works
TorontoCity: Seeing the World with a Million Eyes
Wang, S., Bai, M., Mattyus, G., Chu, H., Luo, W., Yang, B., Liang, J., Cheverie, J., Fidler, S., & Urtasun, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.327 · Cited by 186 · see citing works
3D Object Detection from Images for Autonomous Driving: A Survey
Ma, X., Ouyang, W., Simonelli, A., & Ricci, E. (2022) · arXiv · doi:10.48550/arxiv.2202.02980 · Cited by 147 · see citing works
CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth
Facil, J. M., Ummenhofer, B., Zhou, H., Montesano, L., Brox, T., & Civera, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01210 · Cited by 130 · see citing works
Self-citationSingle image depth estimation: An overview
Mertan, A., Duff, D. J., & Unal, G. (2022) · Digital Signal Processing · doi:10.1016/j.dsp.2022.103441 · Cited by 121 · see citing works
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning
Milz, S., Arbeiter, G., Witt, C., Abdallah, B., & Yogamani, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00062 · Cited by 107 · see citing works
CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow
Weinzaepfel, P., Lucas, T., Leroy, V., Cabon, Y., Arora, V., Brégier, R., Csurka, G., Antsfeld, L., Chidlovskii, B., & Revaud, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01647 · Cited by 110 · see citing works
Smolyanskiy, N., Kamenev, A., & Birchfield, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00147 · Cited by 104 · see citing works
MODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving
Siam, M., Mahgoub, H., Zahran, M., Yogamani, S., Jagersand, M., & El-Sallab, A. (2018) · 2018 21st International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569744 · Cited by 95 · see citing works
SMSnet: Semantic motion segmentation using deep convolutional neural networks
Vertens, J., Valada, A., & Burgard, W. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202211 · Cited by 69 · see citing works
ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond
Qiao, S., Shen, W., Qiu, W., Liu, C., & Yuille, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.199 · Cited by 41 · see citing works
Duran, O., & Turan, B. (2022) · Measurement · doi:10.1016/j.measurement.2022.110732 · Cited by 25 · see citing works
- Authors
- Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T.
- Affiliations
- University of Freiburg (DE)
- Technical University of Munich (DE)
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 4040-4048
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2259424905
- Cited by
- 2,488
- References
- 24
- Citations by year
- 2026: 157 · 2025: 279 · 2024: 360 · 2023: 375 · 2022: 317 · 2021: 327 · 2020: 232 · 2019: 238 · 2018: 123 · 2017: 73 · 2016: 6 · 2012: 1
- FWCI
- 50.90
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Keywords
- disparity estimation, optical flow, scene flow estimation, convolutional neural networks, FlowNet, large-scale training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
