Low-latency automotive vision with event cameras
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
The computer vision algorithms used currently in advanced driver assistance systems rely on image-based RGB cameras, leading to a critical bandwidth–latency trade-off for delivering safe driving experiences. To address this, event cameras have emerged as alternative vision sensors. Event cameras measure the changes in intensity asynchronously, offering high temporal resolution and sparsity, markedly reducing bandwidth and latency requirements 1. Despite these advantages, event-camera-based algorithms are either highly efficient but lag behind image-based ones in terms of accuracy or sacrifice the sparsity and efficiency of events to achieve comparable results. To overcome this, here we propose a hybrid event- and frame-based object detector that preserves the advantages of each modality and thus does not suffer from this trade-off. Our method exploits the high temporal resolution and sparsity of events and the rich but low temporal resolution information in standard images to generate efficient, high-rate object detections, reducing perceptual and computational latency. We show that the use of a 20 frames per second (fps) RGB camera plus an event camera can achieve the same latency as a 5,000-fps camera with the bandwidth of a 45-fps camera without compromising accuracy. Our approach paves the way for efficient and robust perception in edge-case scenarios by uncovering the potential of event cameras 2.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141
A flexible new technique for camera calibration
Zhang, Z. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.888718 · Cited by 15,004
YOLOX: Exceeding YOLO Series in 2021
Ge, Z., Liu, S., Wang, F., Li, Z., & Sun, J. (2021) · arXiv · doi:10.48550/arxiv.2107.08430 · Cited by 2,965
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3008413 · Cited by 2,566
Self-citationA 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2007.914337 · Cited by 2,485
3D Semantic Segmentation with Submanifold Sparse Convolutional Networks
Graham, B., Engelcke, M., & Maaten, L. V. D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00961 · Cited by 1,788
A Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Quasi-Dense Similarity Learning for Multiple Object Tracking
Pang, J., Qiu, L., Li, X., Chen, H., Li, Q., Darrell, T., & Yu, F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00023 · Cited by 480
SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels
Fey, M., Lenssen, J. E., Weichert, F., & Muller, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00097 · Cited by 459
Query-Centric Trajectory Prediction
Zhou, Z., Wang, J., Li, Y., & Huang, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01713 · Cited by 333 · cited 1×
“Moreover, this lies within the tolerance limit of the current agent-forecasting methods 10–12 that are currently finding their way into commercial patents 13 , in which we see displacement errors in prediction of the order of 0.6 m, more than one order of magnitude higher than the worst-case error…”
Influential
Metadata
- Authors
- Gehrig, D., & Scaramuzza, D.
- Venue
- Nature
- Published
- 2024-05-29
- Volume
- 629
- Issue
- 8014
- Pages
- 1034-1040
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W4399121711
- Cited by
- 240
- References
- 14
- FWCI
- 33.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.