CoChat
Sign Up
Cited by 2,566Advanced Vision and ImagingAdvanced Memory and Neural ComputingCCD and CMOS Imaging Sensors

Event-based vision: a survey

Open access
Gallego, Guillermo, Delbruck, Tobi, Orchard, Garrick, Bartolozzi, Chiara, Taba, Brian, Censi, Andrea, Leutenegger, Stefan, Davison, Andrew J. +3 more·2022·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2020.3008413
Open this paper in CoChat — free accountOpen PDF

Abstract

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of μs), very high dynamic range (140 dB versus 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2020.3008413

Cites

217 of 217 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 65

  1. C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck, "A 240x180 130dB 3µs latency global shutter spatiotemporal vision sensor," IEEE J. Solid-State Circuits, vol. 49, no. 10, pp. 2333-2341, 2014.
  2. B. Son, Y. Suh, S. Kim, H. Jung, J.-S. Kim, C. Shin, K. Park, K. Lee, J. Park, J. Woo, Y. Roh, H. Lee, Y. Wang, I. Ovsiannikov, and H. Ryu, "A 640x480 dynamic vision sensor with a 9µm pixel and 300Meps address-event representation," in IEEE Intl. Solid-State Circuits Conf. (ISSCC), 2017.
  3. T. Delbruck, "Neuromorphic vision sensing and processing," in Eur. Solid-State Device Research Conf. (ESSDERC), 2016, pp. 7-14.
  4. P. Lichtsteiner and T. Delbruck, "64x64 event-driven logarithmic temporal derivative silicon retina," in IEEE Workshop on Charge- Coupled Devices and Advanced Image Sensors, 2005, pp. 157-160.
  5. P. Lichtsteiner, "An AER temporal contrast vision sensor," Ph. D. Thesis, ETH Zurich, Dept. of Physics (D-PHYS), Zurich, Switzer- land, 2006.
  6. D. Neil, "Deep neural networks and hardware systems for event- driven data," Ph.D. dissertation, ETH-Zurich, Zurich, Switzer- land, 2017.
  7. T. Delbruck, Y. Hu, and Z. He, "V2E: From video frames to realistic DVS event camera streams," arXiv e-prints, 2020.
  8. M. Mahowald, "VLSI analogs of neuronal visual processing: A synthesis of form and function," Ph.D. dissertation, California Institute of Technology, Pasadena, California, May 1992.
  9. R. Berner, C. Brandli, M. Yang, S.-C. Liu, and T. Delbruck, "A 240x180 10mw 12µs latency sparse-output vision sensor for mobile applications," in Proc. Symp. VLSI, 2013, pp. C186-C187.
  10. T. Serrano-Gotarredona and B. Linares-Barranco, "A 128x128 1.5% contrast sensitivity 0.9% FPN 3s latency 4mW asynchronous frame-free dynamic vision sensor using transimpedance pream- plifiers," IEEE J. Solid-State Circuits, vol. 48, no. 3, pp. 827-838, Mar. 2013.
  11. H. Rebecq, D. Gehrig, and D. Scaramuzza, "ESIM: an open event camera simulator," in Conf. on Robotics Learning (CoRL), 2018.
  12. C. Posch and D. Matolin, "Sensitivity and uniformity of a 0.18µm CMOS temporal contrast pixel array," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2011, pp. 1572-1575.
  13. M. Yang, S.-C. Liu, and T. Delbruck, "Comparison of spike encoding schemes in asynchronous vision sensors: Modeling and design," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2014, pp. 2632- 2635.
  14. T. Delbruck, "Frame-free dynamic digital vision," in Proc. Int. Symp. Secure-Life Electron., 2008, pp. 21-26.
  15. A. Khodamoradi and R. Kastner, "O(N)-space spatiotemporal filter for reducing noise in neuromorphic vision sensors," IEEE Trans. Emerg. Topics Comput., vol. PP, no. 99, pp. 1-1, 2018.
  16. D. Czech and G. Orchard, "Evaluating noise filtering for event- based asynchronous change detection image sensors," in IEEE Int. Conf. Biomed. Robot. and Biomechatron. (BioRob), 2016, pp. 19- 24.
  17. V. Padala, A. Basu, and G. Orchard, "A noise filtering algorithm for event-based asynchronous change detection image sensors on TrueNorth and its implementation on TrueNorth," Front. Neurosci., vol. 12, 2018.
  18. R. Berner, "Highspeed USB2.0 AER interfaces," Master's thesis, ETH Zurich, Dept. of Electrical and Information Eng. (D-ITET), Zurich, Switzerland, 2006.
  19. D. B. Fasnacht and T. Delbruck, "Dichromatic spectral measure- ment circuit in vanilla CMOS," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2007, pp. 3091-3094.
  20. R. Berner, P. Lichtsteiner, and T. Delbruck, "Self-timed vertacolor dichromatic vision sensor for low power pattern detection," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2008, pp. 1032-1035.
  21. L. Farian, J. A. Le ñero-Bardallo, and P. Häfliger, "A bio-inspired AER temporal tri-color differentiator pixel array," IEEE Trans. Biomed. Circuits Syst., vol. 9, no. 5, pp. 686-698, Oct. 2015.
  22. R. F. Lyon and P. M. Hubel, "Eyeing the camera: Into the next century," in Proc. IS&T/SID 10th Color Imaging Conf., 2002, pp. 349-355.
  23. C. Li, C. Brandli, R. Berner, H. Liu, M. Yang, S.-C. Liu, and T. Delbruck, "Design of an RGBW color VGA rolling and global shutter dynamic and active-pixel vision sensor," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2015.
  24. D. P. Moeys, C. Li, J. N. P. Martel, S. Bamford, L. Longinotti, V. Motsnyi, D. S. S. Bello, and T. Delbruck, "Color temporal contrast sensitivity in dynamic vision sensors," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2017, pp. 1-4.
  25. A. Marcireau, S.-H. Ieng, C. Simon-Chane, and R. B. Benosman, "Event-based color segmentation with a high dynamic range sensor," Front. Neurosci., vol. 12, 2018.
  26. C. Posch, D. Matolin, R. Wohlgenannt, T. Maier, and M. Litzen- berger, "A microbolometer asynchronous Dynamic Vision Sensor for LWIR," IEEE Sensors J., vol. 9, no. 6, pp. 654-664, Jun. 2009.
  27. M. Liu and T. Delbruck, "Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors," in British Mach. Vis. Conf. (BMVC), 2018.
  28. S. B. Shrestha and G. Orchard, "SLAYER: Spike layer error reassignment in time," in Conf. Neural Inf. Process. Syst. (NIPS), 2018.
  29. J. Binas, D. Neil, S.-C. Liu, and T. Delbruck, "DDD17: End-to-end DAVIS driving dataset," in ICML Workshop on Machine Learning for Autonomous Vehicles, 2017.
  30. C. Ye, A. Mitrokhin, C. Parameshwara, C. Ferm üller, J. A. Yorke, and Y. Aloimonos, "Unsupervised learning of dense optical flow and depth from sparse event data," arXiv e-prints, 2018, 1809.08625.
  31. T. Stoffregen and L. Kleeman, "Simultaneous optical flow and segmentation (SOFAS) using Dynamic Vision Sensor," in Aus- tralasian Conf. Robot. Autom. (ACRA), 2017.
  32. G. Dikov, M. Firouzi, F. R öhrbein, J. Conradt, and C. Richter, "Spiking cooperative stereo-matching at 2ms latency with neu- romorphic hardware," in Conf. Biomimetic and Biohybrid Systems, 2017, pp. 119-137.
  33. M. B. Milde, D. Neil, A. Aimar, T. Delbr ück, and G. Indiveri, "ADaPTION: Toolbox and benchmark for training convolutional neural networks with reduced numerical precision weights and activation," arXiv e-prints, Nov. 2017.
  34. M. B. Milde, O. J. N. Bertrand, H. Ramachandran, M. Egelhaaf, and E. Chicca, "Spiking elementary motion detector in neuromor- phic systems," Neural Computation, vol. 30, no. 9, pp. 2384-2417, Sep. 2018.
  35. L. Salt, D. Howard, G. Indiveri, and Y. Sandamirskaya, "Differ- ential evolution and bayesian optimisation for hyper-parameter selection in mixed-signal neuromorphic circuits applied to UAV obstacle avoidance," arXiv e-prints, 2017, 1704.04853.
  36. B. D. Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision," in Int. Joint Conf. Artificial Intell. (IJCAI), 1981.
  37. --, "Bio-inspired motion estimation with event-driven sen- sors," in Int. Work-Conf. Artificial Neural Netw. (IWANN), Advances in Comput. Intell., 2015, pp. 309-321.
  38. J. Conradt, "On-board real-time optic-flow for miniature event- based vision sensors," in IEEE Int. Conf. Robot. Biomimetics (RO- BIO), 2015.
  39. S. Tschechne, R. Sailer, and H. Neumann, "Bio-inspired optic flow from event-based neuromorphic sensor input," in Artificial Neural Networks in Pattern Recognition (ANNPR), 2014, pp. 171-182.
  40. D. J. Fleet and A. D. Jepson, "Computation of component image velocity from local phase information," Int. J. Comput. Vis., vol. 5, no. 1, pp. 77-104, 1990.
  41. D. Zou, F. Shi, W. Liu, J. Li, Q. Wang, P.-K. J. Park, C.-W. Shi, Y. J. Roh, and H. E. Ryu, "Robust dense depth map estimation from sparse DVS stereos," in British Mach. Vis. Conf. (BMVC), 2017.
  42. R. Szeliski, Computer Vision: Algorithms and Applications, ser. Texts in Computer Science. Springer, 2010.
  43. S. Schraml, A. N. Belbachir, and H. Bischof, "An event-driven stereo system for real-time 3-D 360 panoramic vision," IEEE Trans. Ind. Electron., vol. 63, no. 1, pp. 418-428, Jan. 2016.
  44. --, "Event-driven stereo matching for real-time 3D panoramic vision," in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2015.
  45. R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, "DTAM: Dense tracking and mapping in real-time," in Int. Conf. Comput. Vis. (ICCV), 2011.
  46. C. Brandli, T. Mantel, M. Hutter, M. H öpflinger, R. Berner, R. Sieg- wart, and T. Delbruck, "Adaptive pulsed laser line extraction for terrain reconstruction using a dynamic vision sensor," Front. Neurosci., vol. 7, p. 275, 2014.
  47. J. N. P. Martel, J. M üller, J. Conradt, and Y. Sandamirskaya, "An active approach to solving the stereo matching problem using event-based sensors," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2018.
  48. M. Milford, H. Kim, S. Leutenegger, and A. Davison, "Towards visual SLAM with event-based cameras," in The Problem of Mobile Sensors Workshop in conjunction with RSS, 2015.
  49. Y. Zhou, G. Gallego, and S. Shen, "Event-based stereo visual odometry," arXiv preprint arXiv:2007.15548, 2020. · arXiv
  50. C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, "On- manifold preintegration for real-time visual-inertial odometry," IEEE Trans. Robot., vol. 33, no. 1, pp. 1-21, 2017.
  51. A. I. Mourikis and S. I. Roumeliotis, "A multi-state constraint Kalman filter for vision-aided inertial navigation," in IEEE Int. Conf. Robot. Autom. (ICRA), 2007.
  52. A. Patron-Perez, S. Lovegrove, and G. Sibley, "A spline-based trajectory representation for sensor fusion and rolling shutter cameras," Int. J. Comput. Vis., vol. 113, no. 3, pp. 208-219, 2015.
  53. L. von Stumberg, V. Usenko, and D. Cremers, "Direct sparse visual-inertial odometry using dynamic marginalization," in IEEE Int. Conf. Robot. Autom. (ICRA), 2018.
  54. C. Richter, F. R öhrbein, and J. Conradt, "Bio inspired optic flow detection using neuromorphic hardware," in Bernstein Conf., 2014.
  55. A. Glover, A. B. Stokes, S. Furber, and C. Bartolozzi, "ATIS + SpiNNaker: a fully event-based visual tracking demonstration," in IEEE/RSJ Int. Conf. Intell. Robot. Syst. Workshops (IROSW), 2018.
  56. T. Delbruck, M. Pfeiffer, R. Juston, G. Orchard, E. M üggler, A. Linares-Barranco, and M. W. Tilden, "Human vs. computer slot car racing using an event and frame-based DAVIS vision sensor," in IEEE Int. Symp. Circuits Syst. (ISCAS), 2015.
  57. M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, "ROS: an open-source Robot Operating System," in ICRA Workshop Open Source Softw., vol. 3, no. 2, 2009, p. 5.
  58. A. Glover, V. Vasco, M. Iacono, and C. Bartolozzi, "The event- driven software library for YARP-with algorithms and iCub applications," Front. Robotics and AI, vol. 4, p. 73, 2018.
  59. Y. Hu, J. Binas, D. Nei, S.-C. Liu, and T. Delbruck, "DDD20 End- to-End event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction," in IEEE Int. Conf. Intell. Transportation Systems (ITSC), 2020.
  60. F. Barranco, C. Fermuller, Y. Aloimonos, and T. Delbruck, "A dataset for visual navigation with neuromorphic methods," Front. Neurosci., vol. 10, p. 49, 2016.
  61. J. Delmerico, T. Cieslewski, H. Rebecq, M. Faessler, and D. Scara- muzza, "Are we ready for autonomous drone racing? the UZH- FPV drone racing dataset," in IEEE Int. Conf. Robot. Autom. (ICRA), 2019.
  62. C. Tan, S. Lallee, and G. Orchard, "Benchmarking neuromorphic vision: lessons learnt from computer vision," Front. Neurosci., vol. 9, p. 374, 2015.
  63. E. Calabrese, G. Taverni, C. Awai Easthope, S. Skriabine, F. Cor- radi, L. Longinotti, K. Eng, and T. Delbruck, "DHP19: Dynamic vision sensor 3D human pose dataset," in IEEE Conf. Comput. Vis. Pattern Recog. Workshops (CVPRW), 2019.
  64. G. P. Garcia, P. Camilleri, Q. Liu, and S. B. Furber, "pyDVS: An extensible, real-time dynamic vision sensor emulator using off- the-shelf hardware," in IEEE Symp. Series Comput. Intell. (SSCI), 2016.
  65. J. N. P. Martel, "Unconventional processing with unconventional visual sensing: Parallel, distributed and event based vision al- gorithms & systems," Ph.D. dissertation, ETH-Zurich, Zurich, Switzerland, 2019.

Cited by

Sorted by citations

Authors
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., & Scaramuzza, D.
Affiliations
  • Einstein Center Digital Future (DE)
  • Intel (United States) (US)
  • Italian Institute of Technology (IT)
  • IBM (United States) (US)
  • ETH Zurich (CH)
  • Imperial College London (GB)
  • University of Pennsylvania (US)
  • University of Zurich (CH)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-01-01
Volume
44
Issue
1
Pages
154-180
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2020.3008413
OpenAlex
W3040838455
Cited by
2,566
References
217
Citations by year
2026: 553 · 2025: 648 · 2024: 510 · 2023: 363 · 2022: 258 · 2021: 168 · 2020: 59 · 2019: 6 · 2012: 1
FWCI
17.31
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • SNSF-ERC
  • Swiss National Science Foundation
  • National Center of Competence in Research Robotics
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › CCD and CMOS Imaging Sensors
Keywords
event cameras, event-based vision, bio-inspired sensors, high dynamic range, high temporal resolution, event processing, optical flow, 3D reconstruction, image segmentation, object recognition, spiking neural networks, low-power vision, robot vision
MeSH terms
Algorithms, Robotics, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review