The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Events-To-Video: Bringing Modern Computer Vision to Event Cameras
Open accessAbstract
Event cameras are novel sensors that report brightness changes in the form of asynchronous “events” instead of intensity frames. They have significant advantages over conventional cameras: high temporal resolution, high dynamic range, and no motion blur. Since the output of event cameras is fundamentally different from conventional cameras, it is commonly accepted that they require the development of specialized algorithms to accommodate the particular nature of events. In this work, we take a different view and propose to apply existing, mature computer vision techniques to videos reconstructed from event data. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. Our experiments show that our approach surpasses state-of-the-art reconstruction methods by a large margin (> 20%) in terms of image quality. We further apply off-the-shelf computer vision algorithms to videos reconstructed from event data on tasks such as object classification and visual-inertial odometry, and show that this strategy consistently outperforms algorithms that were specifically designed for event data. We believe that our approach opens the door to bringing the outstanding properties of event cameras to an entirely new range of tasks. A video of the experiments is available at https://youtu.be/IdYrC4cUO0I.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
(2017) · Cited by 11,625
$rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation
Aharon, M., Elad, M., & Bruckstein, A. (2006) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2006.881199 · Cited by 9,451
VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator
Qin, T., Li, P., & Shen, S. (2018) · IEEE Transactions on Robotics · doi:10.1109/tro.2018.2853729 · Cited by 4,669
One-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029
A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2007.914337 · Cited by 2,485
A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation
Mourikis, A. I., & Roumeliotis, S. I. (2007) · Proceedings 2007 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2007.364024 · Cited by 1,951
Keyframe-based visual–inertial odometry using nonlinear optimization
Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., & Furgale, P. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914554813 · Cited by 1,723
On-Manifold Preintegration for Real-Time Visual--Inertial Odometry
Forster, C., Carlone, L., Dellaert, F., & Scaramuzza, D. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2597321 · Cited by 1,502
Self-citationA 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor
Brandli, C., Berner, R., Yang, M., Liu, S.-C., & Delbruck, T. (2014) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2014.2342715 · Cited by 1,144
A Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Robust visual inertial odometry using a direct EKF-based approach
Bloesch, M., Omari, S., Hutter, M., & Siegwart, R. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353389 · Cited by 917
Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades
Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00437 · Cited by 882
A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry
Zhang, Z., & Scaramuzza, D. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593941 · Cited by 658
Self-citationHOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition
Lagorce, X., Orchard, G., Galluppi, F., Shi, B. E., & Benosman, R. B. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2574707 · Cited by 612
Mueggler, E., Rebecq, H., Gallego, G., Delbruck, T., & Scaramuzza, D. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364917691115 · Cited by 606
Self-citationThe Multivehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
Zhu, A. Z., Thakur, D., Ozaslan, T., Pfrommer, B., Kumar, V., & Daniilidis, K. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2800793 · Cited by 567
Vidal, A. R., Rebecq, H., Horstschaefer, T., & Scaramuzza, D. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2793357 · Cited by 541
Self-citationEvent-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars
Maqueda, A. I., Loquercio, A., Gallego, G., Garcia, N., & Scaramuzza, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00568 · Cited by 514
Self-citationEvent-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars
Maqueda, A. I., Loquercio, A., Gallego, G., Garcia, N. N., & Scaramuzza, D. (2018) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-175994 · Cited by 493
Self-citationHATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification
Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., & Benosman, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00186 · Cited by 486
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
Zhu, A., Yuan, L., Chaney, K., & Daniilidis, K. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.062 · Cited by 470
Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera
Kim, H., Leutenegger, S., & Davison, A. J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_21 · Cited by 438
Benosman, R., Clercq, C., Lagorce, X., Ieng, S.-H., & Bartolozzi, C. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2273537 · Cited by 419
EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real Time
Rebecq, H., Horstschaefer, T., Gallego, G., & Scaramuzza, D. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2645143 · Cited by 410
Self-citationA Benchmark Comparison of Monocular Visual-Inertial Odometry Algorithms for Flying Robots
Delmerico, J., & Scaramuzza, D. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460664 · Cited by 399
Self-citationPerez-Carrasco, J. A., Zhao, B., Serrano, C., Acha, B., Serrano-Gotarredona, T., Chen, S., & Linares-Barranco, B. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.71 · Cited by 324
HFirst: A Temporal Approach to Object Recognition
Orchard, G., Meyer, C., Etienne-Cummings, R., Posch, C., Thakor, N., & Benosman, R. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2392947 · Cited by 312
Simultaneous Optical Flow and Intensity Estimation from an Event Camera
Bardow, P., Davison, A. J., & Leutenegger, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.102 · Cited by 295
Real-time Visual-Inertial Odometry for Event Cameras using Keyframe-based Nonlinear Optimization
Rebecq, H., Horstschaefer, T., & Scaramuzza, D. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.16 · Cited by 263
Self-citationEvent-based, 6-DOF pose tracking for high-speed maneuvers
Mueggler, E., Huber, B., & Scaramuzza, D. (2014) · 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2014.6942940 · Cited by 258
Self-citationEvent-Based Visual Inertial Odometry
Zhu, A. Z., Atanasov, N., & Daniilidis, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.616 · Cited by 249
Simultaneous Mosaicing and Tracking with an Event Camera
Kim, H., Handa, A., Benosman, R., Ieng, S.-H., & Davison, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.26 · Cited by 247
Continuous-Time Intensity Estimation Using Event Cameras
Scheerlinck, C., Barnes, N., & Mahony, R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20873-8_20 · Cited by 217
Event-Based, 6-DOF Camera Tracking from Photometric Depth Maps
Gallego, G., Lund, J. E. A., Mueggler, E., Rebecq, H., Delbruck, T., & Scaramuzza, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2769655 · Cited by 211
Self-citationLow-latency visual odometry using event-based feature tracks
Kueng, B., Mueggler, E., Gallego, G., & Scaramuzza, D. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7758089 · Cited by 196
Self-citationESIM: an Open Event Camera Simulator
Rebecq, H., Gehrig, D., & Scaramuzza, D. (2018) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-175993 · Cited by 194
Self-citationReal-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation
Munda, G., Reinbacher, C., & Pock, T. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1106-2 · Cited by 181
Event-based feature tracking with probabilistic data association
Zhu, A. Z., Atanasov, N., & Daniilidis, K. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989517 · Cited by 174
Keyframe-based Visual-Inertial SLAM using Nonlinear Optimization
Leutenegger, S., Furgale, P. T., Rabaud, V., Chli, M., Konolige, K., & Siegwart, R. (2013) · doi:10.3929/ethz-b-000236658 · Cited by 169
Event-Based Motion Segmentation by Motion Compensation
Stoffregen, T., Gallego, G., Drummond, T. W., Kleeman, L., & Scaramuzza, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00734 · Cited by 162
Self-citationAsynchronous, Photometric Feature Tracking Using Events and Frames
Gehrig, D., Rebecq, H., Gallego, G., & Scaramuzza, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_46 · Cited by 157
Self-citationA pencil balancing robot using a pair of AER dynamic vision sensors
Conradt, J., Cook, M., Berner, R., Lichtsteiner, P., Douglas, R. J., & Delbrück, T. (2009) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2009.5117867 · Cited by 155
Interacting maps for fast visual interpretation
Cook, M., Gugelmann, L., Jug, F., Krautz, C., & Steger, A. (2011) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2011.6033299 · Cited by 142
Direct face detection and video reconstruction from event cameras
Miyatani, Y., Barua, S., & Veeraraghavan, A. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477561 · Cited by 133 · cited 2×
“contrast to [4], we do not reconstruct individual intensity images from small windows of events, but instead reconstruct a temporally consistent video from a long stream of events (several seconds) using a recurrent network.”
BackgroundReal-time, high-speed video decompression using a frame- and event-based DAVIS sensor
Brändli, C., Müller, L. K., & Delbrück, T. (2014) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2014.6865228 · Cited by 121
Unsupervised Event-Based Optical Flow Using Motion Compensation
Zhu, A. Z., Yuan, L., Chaney, K. P., & Daniilidis, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11024-6_54 · Cited by 107
Adaptive Time-Slice Block-Matching Optical Flow Algorithm for Dynamic Vision Sensors
Liu, M., & Delbrück, T. (2018) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-168589 · Cited by 78
Semi-dense 3D Reconstruction with a Stereo Event Camera
Zhou, Y. B., Gallego, G., Rebecq, H., Kneip, L., Li, H., & Scaramuzza, D. (2018) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-176054 · Cited by 58
Self-citation
Cited by
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3008413 · Cited by 2,566
Self-citationGallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000521357 · Cited by 591
Self-citationv2e: From Video Frames to Realistic DVS Events
Hu, Y., Liu, S.-C., & Delbruck, T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00144 · Cited by 408
Chen, G., Cao, H., Conradt, J., Tang, H., Rohrbein, F., & Knoll, A. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2985815 · Cited by 304
Event-driven Re-Id: A New Benchmark and Method Towards Privacy-Preserving Person Re-Identification
Ahmad, S., Scarpellini, G., Morerio, P., & Bue, A. D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw54805.2022.00052 · Cited by 54
Metadata
- Authors
- Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 3852-3861
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2938463073
- Cited by
- 420
- References
- 57
- FWCI
- 14.42
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.