Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling
Open accessAbstract
We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large data set containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The method outperforms recent methods that explicitly compute optical flow and motion masks.
“We also outperform recent methods ( [38,51]) that explicitly compute optical flow as well as motion masks.”
From Digging Into Self-Supervised Monocular Depth Estimation (2019) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Single image haze removal using dark channel prior
He, K., Sun, J., & Tang, X. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.168 · Cited by 7,601 · see citing works
Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
Oliva, A., & Torralba, A. B. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011139631724 · Cited by 6,321 · see citing works
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · see citing works
Single image haze removal using dark channel prior
He, K., Sun, J., & Tang, X. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206515 · Cited by 2,449 · see citing works
Make3D: Learning 3D Scene Structure from a Single Still Image
Saxena, A., Sun, M., & Ng, A. Y. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.132 · Cited by 1,738 · see citing works
SIFT Flow: Dense Correspondence across Scenes and Its Applications
Liu, C., Yuen, J., & Torralba, A. B. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.147 · Cited by 1,590 · see citing works
Self-citationA texture-based method for modeling the background and detecting moving objects
Heikkilä, M., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.68 · Cited by 1,249 · see citing works
(2005) · Cited by 981 · see citing works
Beyond pixels: exploring new representations and applications for motion analysis
Freeman, W. T., Adelson, E. H., & Liu, C. (2009) · Cited by 830 · see citing works
Self-citationHoiem, D., Efros, A. A., & Hebert, M. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073232 · Cited by 616 · see citing works
Single image depth estimation from predicted semantic labels
Liu, B., Gould, S. J., & Koller, D. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539823 · Cited by 484 · see citing works
Using contours to detect and localize junctions in natural images
Maire, M., Arbeláez, P., Fowlkes, C. C., & Malik, J. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587420 · Cited by 403 · see citing works
Horry, Y., Anjyo, K., & Arai, K. (1997) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/258734.258854 · Cited by 393 · see citing works
Nonparametric Scene Parsing via Label Transfer
Liu, C., Yuen, J., & Torralba, A. B. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.131 · Cited by 369 · see citing works
Self-citationUnsupervised Joint Object Discovery and Segmentation in Internet Images
Rubinstein, M., Joulin, A., Kopf, J., & Liu, C. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.253 · Cited by 359 · see citing works
Self-citationImage-based modeling and photo editing
Oh, B. M., Chen, M., Dorsey, J. O., & Durand, F. (2001) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/383259.383310 · Cited by 351 · see citing works
Nonparametric scene parsing: Label transfer via dense scene alignment
Liu, C., Yuen, J., & Torralba, A. B. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206536 · Cited by 313 · see citing works
Self-citationBackground Subtraction for Freely Moving Cameras
Sheikh, Y. A., Javed, O., & Kanade, T. (2009) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459334 · Cited by 305 · see citing works
Consistent Depth Maps Recovery from a Video Sequence
Zhang, G., Jia, J., Wong, T., & Bao, H. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.52 · Cited by 267 · see citing works
A Dynamic Bayesian Network Model for Autonomous 3D Reconstruction from a Single Indoor Image
Delage, E., Lee, H., & Ng, A. Y. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.23 · Cited by 238 · see citing works
Recovering Occlusion Boundaries from a Single Image
Hoiem, D., Stein, A. N., Efros, A. A., & Hebert, M. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408985 · Cited by 206 · see citing works
Hoiem, D., Efros, A. A., & Hebert, M. (2005) · ACM SIGGRAPH Conference Papers (SIGGRAPH) · doi:10.1145/1186822.1073232 · Cited by 180 · see citing works
3D-TV Content Creation: Automatic 2D-to-3D Video Conversion
Zhang, L., Vázquez, C., & Knorr, S. (2011) · IEEE transactions on broadcasting · doi:10.1109/tbc.2011.2122930 · Cited by 169 · see citing works
2D-to-3D image conversion by learning depth from examples
Konrad, J., Wang, M., & Ishwar, P. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6238903 · Cited by 145 · see citing works
Example Based 3D Reconstruction from Single 2D Images
Hassner, T., & Basri, R. (2006) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2006.76 · Cited by 130 · see citing works
Semi-automatic stereo extraction from video footage
Guttmann, M., Wolf, L., & Cohen‐Or, D. (2009) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459158 · Cited by 130 · see citing works
Single view modeling of free-form scenes
Zhang, L., Dugas‐Phocion, G., Samson, J., & Seitzt, S. M. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2001.990638 · Cited by 129 · see citing works
Bayesian Reconstruction of 3D Shapes and Scenes From A Single Image
(2008) · Cited by 94 · see citing works
Single‐view modelling of free‐form scenes
Zhang, L., Dugas‐Phocion, G., Samson, J., et al. (2002) · The Journal of Visualization and Computer Animation · doi:10.1002/vis.291 · Cited by 92 · see citing works
Toward Holistic Scene Understanding: Feedback Enabled Cascaded Classification Models
Li, C., Kowdle, A., Saxena, A., & Chen, T. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.232 · Cited by 90 · see citing works
Wang, O., Lang, M., Frei, M., et al. (2011) · doi:10.1145/2021164.2021173 · Cited by 78 · see citing works
Robust Bilayer Segmentation and Motion/Depth Estimation with a Handheld Camera
Zhang, G., Jia, J., Hua, W., & Bao, H. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.115 · Cited by 77 · see citing works
Repetition-based dense single-view reconstruction
Wu, C., Frahm, J., & Pollefeys, M. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995551 · Cited by 76 · see citing works
A Bayesian Approach to Alignment-Based Image Hallucination
Tappen, M. F., & Liu, C. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33786-4_18 · Cited by 74 · see citing works
Self-citationSIFT Flow: Dense Correspondence Across Scenes and Its Applications
Liu, C., Yuen, J., & Torralba, A. B. (2015) · Springer eBooks · doi:10.1007/978-3-319-23048-1_2 · Cited by 68 · see citing works
Self-citationA Viewer-Centric Editor for 3D Movies
Koppal, S. J., Zitnick, C. L., Cohen, M. M., et al. (2010) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2010.37 · Cited by 58 · see citing works
Video Stereolization: Combining Motion Analysis with User Interaction
Liao, M., Gao, J., Yang, R., et al. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.114 · Cited by 53 · see citing works
Towards Holistic Scene Understanding: Feedback Enabled Cascaded Classification ModelsPreprint
Li, C., Kowdle, A., Saxena, A., et al. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1110.5102 · Cited by 45 · see citing works
Automatic 2D-to-3D image conversion using 3D examples from the internet
Konrad, J., Brown, G., Wang, M., et al. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.910601 · Cited by 43 · see citing works
Depth Director: A System for Adding Depth to Movies
Ward, B. M., Kang, S. B., & Bennett, E. P. (2010) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2010.103 · Cited by 39 · see citing works
Self-citationRefilming with Depth-Inferred Videos
Zhang, G., Dong, Z., Jia, J., Wan, L., Wong, T., & Bao, H. (2009) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2009.47 · Cited by 36 · see citing works
Luo, K., Li, D., Feng, Y.-M., et al. (2009) · Journal of Zhejiang University. Science A · doi:10.1631/jzus.a0820806 · Cited by 26 · see citing works
Learning the right model: Efficient max-margin learning in Laplacian CRFs
Batra, D., & Saxena, A. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247920 · Cited by 25 · see citing works
Coherent spatial and temporal occlusion generation
Gunnewiek, R. K., Berretty, R.-P., Barenbrug, B., et al. (2009) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.806818 · Cited by 16 · see citing works
Continuous Parallax Adjustment for 3D-TV
Colombari, A., Fusiello, A., & Murino, V. (2005) · Cited by 8 · see citing works
Continuous parallax adjustment for 3D-TV
Colombari, Fusiello, & Murino (2006) · Conference on Visual Media Production · Cited by 7 · see citing works
Dimensionalization: converting 2D films to 3D
Pernis, A. P. V., & DeJohn, M. (2008) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.766566 · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [5][5] D. Hoiem, A. Efros, and M. Hebert, 'Automatic photo popup,' in ACM SIGGRAPH , 2005. -
- [26][26] M. Tappen and C. Liu, 'A bayesian approach to alignmentbased image hallucination,' in ECCV , 2012. -
- [27][27] M. Rubinstein, C. Liu, and W. Freeman, 'Annotation propagation: Automatic annotation of large image databases via dense image correspondence,' in ECCV , 2012. -
Cited by
Sorted by citations
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · see citing works
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · see citing works
Digging Into Self-Supervised Monocular Depth Estimation
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00393 · Cited by 2,561 · see citing works
Deep Ordinal Regression Network for Monocular Depth Estimation
Fu, H., Gong, M., Wang, C., Batmanghelich, K., & Tao, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00214 · Cited by 1,991 · see citing works
Deep convolutional neural fields for depth estimation from a single image
Liu, F., Shen, C., & Lin, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299152 · Cited by 892 · see citing works
Learning Depth from Monocular Videos Using Direct Methods
Wang, C., Buenaposada, J. M., Zhu, R., & Lucey, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00216 · Cited by 667 · see citing works
Discrete-Continuous Depth Estimation from a Single Image
Liu, M., Salzmann, M., & He, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.97 · Cited by 381 · see citing works
Consistent video depth estimation
Luo, X., Huang, J.-B., Szeliski, R., Matzen, K., & Kopf, J. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392377 · Cited by 322 · see citing works
Monocular Relative Depth Perception with Web Stereo Data Supervision
Xian, K., Shen, C., Cao, Z., Lu, H., Xiao, Y., Li, R., & Luo, Z. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00040 · Cited by 215 · see citing works
Monocular depth estimation with hierarchical fusion of dilated CNNs and soft-weighted-sum inference
Li, B., Dai, Y., & He, M. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.05.029 · Cited by 142 · see citing works
Single image depth estimation: An overview
Mertan, A., Duff, D. J., & Unal, G. (2022) · Digital Signal Processing · doi:10.1016/j.dsp.2022.103441 · Cited by 121 · see citing works
- Authors
- Karsch, K., Liu, C., & Kang, S. B.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Microsoft Research New England (United States) (US)
- Microsoft (United States) (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2014-11-01
- Volume
- 36
- Issue
- 11
- Pages
- 2144-2158
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W2074254947
- Cited by
- 453
- References
- 47
- Citations by year
- 2026: 11 · 2025: 19 · 2024: 23 · 2023: 34 · 2022: 44 · 2021: 62 · 2020: 56 · 2019: 73 · 2018: 40 · 2017: 32 · 2016: 28 · 2015: 30 · 2014: 1
- FWCI
- 25.27
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Engineering › Media Technology › Image Processing Techniques and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Keywords
- depth estimation, depth images, video, optical flow, temporal coherence, 3D video, Kinect sensor, dynamic scenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
