Light field video stabilization
The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.
Abstract
We describe a method for producing a smooth, stabilized video from the shaky input of a hand-held light field video camera—specifically, a small camera array. Traditional stabilization techniques dampen shake with 2D warps, and thus have limited ability to stabilize a significantly shaky camera motion through a 3D scene. Other recent stabilization techniques synthesize novel views as they would have been seen along a virtual, smooth 3D camera path, but are limited to static scenes. We show that video camera arrays enable much more powerful video stabilization, since they allow changes in viewpoint for a single time instant. Furthermore, we point out that the straightforward approach to light field video stabilization requires computing structure-from-motion, which can be brittle for typical consumer-level video of general dynamic scenes. We present a more robust approach that avoids input camera path reconstruction. Instead, we employ a spacetime optimization that directly computes a sequence of relative poses between the virtual camera and the camera array, while minimizing acceleration of salient visual features in the virtual image plane. We validate our novel method by comparing it to state-of-the-art stabilization software, such as Apple iMovie and 2d3 SteadyMove Pro, on a number of challenging scenes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Light-field cameras offer several advantages over standard cameras, such as depth estimation, noise reduction [57], video stabilization [227], isolation of distractors [58], and specularity removal [119].”
From Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016) · cites this paper as [227]
Citations
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 2×
“Conditions for success include the presence of enough distinct image features, minimal motion blur and noise, and scenes composed mostly of rigid background with only small dynamic scene elements whose motion can be filtered as outliers.”
BackgroundMethodA Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004 · cited 1×
“To address this issue, we do not use edge pixels that have significantly different depth values compared with their matches in adjacent frames.”
BackgroundMultiple View Geometry in Computer Vision
Hartley, R., & Zisserman, A. (2004) · doi:10.1017/cbo9780511811685 · Cited by 21,070
(2021) · A First Course in Statistical Programming with R · doi:10.1017/9781108993456.011 · Cited by 13,724
Multiple View Geometry in Computer Vision
(2001) · Kybernetes · doi:10.1108/k.2001.30.9_10.1333.2 · Cited by 3,336
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 1×
“[8] for the optical flow computation, although other state-of-the-art optical flow methods reviewed in [6] may also be used.”
BackgroundLight Field Photography with a Hand-held Plenoptic Camera
Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., & Hanrahan, P. (2005) · HAL · Cited by 1,749
High-quality video view interpolation using a layered representation
Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., & Szeliski, R. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015766 · Cited by 1,385
Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods
Bruhn, A., Weickert, J., & Schnörr, C. (2005) · International Journal of Computer Vision · doi:10.1023/b:visi.0000045324.43199.43 · Cited by 1,160
Single lens stereo with a plenoptic camera
Adelson, E. H., & Wang, J. Y. A. (1992) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.121783 · Cited by 1,101 · cited 2×
“We briefly describe our first approach as a baseline method, which motivates our second approach.”
MethodHigh performance imaging using large camera arrays
Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., & Levoy, M. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073259 · Cited by 1,042
Lucas/Kanade meets Horn/Schunck : combining local and global optic flow methods
Weickert, J., Bruhn, A., & Schnörr, C. (2003) · doi:10.22028/d291-26234 · Cited by 963
Unstructured lumigraph rendering
Buehler, C., Bosse, M., McMillan, L., Gortler, S., & Cohen, M. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383309 · Cited by 881 · cited 1×
“Pixel intensities are combined by averaging; more advanced view-dependent combinations [10] can also be used to improve results.”
A Database and Evaluation Methodology for Optical Flow
Baker, S., Roth, S., Scharstein, D., Black, M. J., Lewis, J. P., & Szeliski, R. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408903 · Cited by 618
Full-frame video stabilization with motion inpainting
Matsushita, Y., Ofek, E., Ge, W., Tang, X., & Shum, H.-Y. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.141 · Cited by 542 · cited 1×
“Recent work on motion inpainting [21] can be used instead to improve the visual quality of stabilized videos.”
BackgroundVeeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276463 · Cited by 469
Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276463 · Cited by 444
Review of image-based rendering techniques
Shum, H., & Kang, S. B. (2000) · SPIE Proceedings · doi:10.1117/12.386541 · Cited by 385 · cited 1×
“view synthesis) [19] to generate an image at any location with any orientation, as pointed out in [9]; research following this idea includes [13, 7].”
BackgroundReal-Time Visibility-Based Fusion of Depth Maps
Merrell, P., Akbarzadeh, A., Wang, L., Mordohai, P., Frahm, J.-M., Yang, R., Nister, D., & Pollefeys, M. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408984 · Cited by 358 · cited 1×
“At each frame, we first (forward) warp all input depth maps to the virtual viewpoint and fuse the warped depth maps using an approach similar to [22].”
MethodReal-time scene stabilization and mosaic construction
Hansen, M., Anandan, P., Dana, K., van der Wal, G., & Burt, P. (2002) · Proceedings of 1994 IEEE Workshop on Applications of Computer Vision · doi:10.1109/acv.1994.341288 · Cited by 203 · cited 1×
“Light field video stabilization takes as input a set of shaky1 videos from multiple viewpoints captured by a handheld camera array, such as the one shown in Figure 1, and produces a stable single-view video as output.”
BackgroundEvaluation of image stabilization algorithms
Morimoto, C., & Chellappa, R. (2002) · Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181) · doi:10.1109/icassp.1998.678102 · Cited by 185 · cited 1×
“Light field video stabilization takes as input a set of shaky1 videos from multiple viewpoints captured by a handheld camera array, such as the one shown in Figure 1, and produces a stable single-view video as output.”
BackgroundImage-based rendering using image-based priors
Fitzgibbon, Wexler, & Zisserman (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238625 · Cited by 161
Non-metric image-based rendering for video stabilization
Buehler, C., Bosse, M. C., & McMillan, L. (2001) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2001.991019 · Cited by 155 · cited 4×
“Video stabilization can be viewed as an image-based rendering problem [9]: given a sequence of input images captured along an unsteady camera path, generate a new image sequence that appears as if it were captured along a new and smooth virtual camera path, as illustrated in Figure 1(b).”
BackgroundMethodInfluentialRecovery of ego-motion using image stabilization
Irani, Rousso, & Peleg (1994) · doi:10.1109/cvpr.1994.323866 · Cited by 143 · cited 1×
“We have explored two approaches to the light field video stabilization problem.”
MethodRe-cinematography: Improving the camerawork of casual video
Gleicher, M. L., & Liu, F. (2008) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/1404880.1404882 · Cited by 127 · cited 1×
“To simplify this problem, we assume the camera array is synchronized, and the internal camera parameters and their relative poses are pre-calibrated and remain fixed for the duration of video capture.”
MethodA self-reconfigurable camera array
Zhang, C., & Chen, T. (2004) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1186223.1186412 · Cited by 125
Using Photographs to Enhance Videos of a Static Scene
Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., & Kang, S. B. (2007) · Eurographics · doi:10.5555/2383847.2383889 · Cited by 105
Self-citationImage-Based Rendering Using Image-Based Priors
Fitzgibbon, A. W., Wexler, Y., & Zisserman, A. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-6643-9 · Cited by 104
Smith, D. K. (2001) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601183 · Cited by 71
Stereo matching with nonparametric smoothness priors in feature space
Smith, B. M., Zhang, L., & Jin, H. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206793 · Cited by 50 · cited 2×
“Since edge points often cannot be matched as conveniently as SIFT features, we compute the 3D locations of these edge feature points by computing dense depth maps for each entire image using multi-view stereo [29].”
MethodSelf-citationFast 3D stabilization and mosaic construction
Morimoto, C. H., & Chellappa, R. (2002) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1997.609396 · Cited by 46 · cited 1×
“We have explored two approaches to the light field video stabilization problem.”
MethodScene Collages and Flexible Camera Arrays
Nomura, Y., Zhang, L., & Nayar, S. K. (2007) · Eurographics · doi:10.2312/egwr/egsr07/127-138 · Cited by 19
Self-citation
Research works printed but not linked: 7
- burt, 1994, Image stabilization by registration to a reference mosaic, DARPA Image Understanding Workshop
- georgiev, 0, Spatio-angular resolution tradeoffs in integral photography, Rendering Techniques 2006 17th Eurographics Workshop on Rendering
- 0, ViewPLUS Inc ProFUSION 25 5*5 camera array system
- 0, Fujifilm finepix rea13d system, 2
- 0, Apple iMovie'09
- 0, 2d3 SteadyMove Pro
- morimoto, 1996, Automatic digital image stabilization, ICPR
Cited by
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., & Leonard, J. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2624754 · Cited by 3,596
“Light-field cameras offer several advantages over standard cameras, such as depth estimation, noise reduction [58], video stabilization [227], isolation of distractors [59], and specularity removal [119].”
Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I. D., & Leonard, J. J. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-138892 · Cited by 3,048
Light Field Image Processing: An Overview
Wu, G., Masia, B., Jarabo, A., Zhang, Y., Wang, L., Dai, Q., Chai, T., & Liu, Y. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2747126 · Cited by 586
“[192] described a video stabilization approach using an integrated camera array, ProFUSION-25C [21].”
MethodBundled camera paths for video stabilization
Liu, S., Yuan, L., Tan, P., & Sun, J. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461995 · Cited by 352
“Feature track smoothing is also used in light-field camera video stabilization work [Smith et al. 2009].”
MethodAuto-directed video stabilization with robust L1 optimal camera paths
Grundmann, M., Kwatra, V., & Essa, I. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995525 · Cited by 351
“tionally, image-based rendering techniques [1] or light-field rendering (if the video was captured by a camera array [13]) can be used to recast the original video.”
Method
Metadata
- Authors
- Smith, B. M., Zhang, L., Jin, H., & Agarwala, A.
- Affiliations
- University of Wisconsin–Madison (US)
- Adobe Systems (United States) (US)
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 341-348
- OpenAlex
- W2112673789
- Cited by
- 107
- References
- 34
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 6 · 2023: 8 · 2022: 6 · 2021: 4 · 2020: 6 · 2019: 11 · 2018: 4 · 2017: 9 · 2016: 11 · 2015: 5 · 2014: 10 · 2013: 6 · 2012: 3
- FWCI
- 3.54
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Funders
- Adobe Systems
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Stabilization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Keywords
- light field video, video stabilization, camera array, relative pose estimation, structure from motion, dynamic scenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.