On failures of RGB cameras and their effects in autonomous driving applications
Open accessAbstract
RGB cameras are arguably one of the most relevant sensors for autonomous driving applications. It is undeniable that failures of vehicle cameras may compromise the autonomous driving task, possibly leading to unsafe behaviors when images that are subsequently processed by the driving system are altered. To support the definition of safe and robust vehicle architectures and intelligent systems, in this paper we define the failure modes of a vehicle camera, together with an analysis of effects and known mitigations. Further, we build a software library for the generation of the corresponding failed images and we feed them to the trained agent of an autonomous driving simulator: the misbehavior of the trained agent allows a better understanding of failures effects and especially of the resulting safety risk.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RGB cameras are sensitive to light and weather conditions which can lead to failures in outdoor environments.
“Moreover, RGB cameras are sensitive to light and weather conditions which can lead to failures in outdoor environments [37].”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · see citing works
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · see citing works
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · see citing works
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · see citing works
Object scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270 · see citing works
Towards fully autonomous driving: Systems and algorithms
Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S., Kolter, J. Z., Langer, D., Pink, O., Pratt, V., et al. (2011) · 2011 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2011.5940562 · Cited by 1,374 · see citing works
A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos
Schops, T., Schonberger, J. L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., & Geiger, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.272 · Cited by 915 · see citing works
Restoring an Image Taken through a Window Covered with Dirt or Rain
Eigen, D., Krishnan, D., & Fergus, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.84 · Cited by 454 · see citing works
The Architectural Implications of Autonomous Driving
Lin, S.-C., Zhang, Y., Hsu, C.-H., Skach, M., Haque, M. E., Tang, L., & Mars, J. (2018) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3173162.3173191 · Cited by 383 · see citing works
(2017) · doi:10.48550/arxiv.1712.02779 · Cited by 300 · see citing works
Huang, J., You, J., Liu, H., & Song, M. (2020) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2020.106885 · Cited by 279 · see citing works
Failure mode and effects analysis
Teng, S., & Ho, S. (1996) · International Journal of Quality & Reliability Management · doi:10.1108/02656719610118151 · Cited by 274 · see citing works
Blind motion deblurring from a single image using sparse approximation
Cai, J., Ji, H., Liu, C., & Shen, Z. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206743 · Cited by 269 · see citing works
True multi-image alignment and its application to mosaicing and lens distortion correction
Sawhney, H., & Kumar, R. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.754589 · Cited by 244 · see citing works
Line-Based Correction of Radial Lens Distortion
Prescott, B., & McLean, G. F. (1997) · Graphical Models and Image Processing · doi:10.1006/gmip.1996.0407 · Cited by 167 · see citing works
Lens distortion correction for digital image correlation by measuring rigid body displacement
Yoneyama, S., Kikuta, H., Kitagawa, A., et al. (2006) · Optical Engineering · doi:10.1117/1.2168411 · Cited by 157 · see citing works
A STATE-OF-THE-ART REVIEW OF FMEA/FMECA
(1994) · doi:10.1142/s0218539394000362 · Cited by 138 · see citing works
End-to-End Driving Via Conditional Imitation Learning
Codevilla, F., Muller, M., Lopez, A., Koltun, V., & Dosovitskiy, A. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460487 · Cited by 129 · see citing works
Wide-angle camera technology for automotive applications: a review
Hughes, C., Glavin, M., Jones, E., et al. (2009) · IET Intelligent Transport Systems · doi:10.1049/iet-its:20080017 · Cited by 107 · see citing works
Automatic Removal of Chromatic Aberration from a Single Image
Kang, S. B. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383214 · Cited by 80 · see citing works
Benchmarking Image Sensors Under Adverse Weather Conditions for Autonomous Driving
Bijelic, M., Gruber, T., & Ritter, W. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500659 · Cited by 72 · see citing works
Towards Practical Verification of Machine Learning: The Case of Computer Vision SystemsPreprint
Pei, K., Zhu, L., Cao, Y., Yang, J., Vondrick, C., & Jana, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.01785 · Cited by 69 · see citing works
Removing image artifacts due to dirty camera lenses and thin occluders
Gu, J., Ramamoorthi, R., Belhumeur, P. N., & Nayar, S. K. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618490 · Cited by 65 · see citing works
Townsend, H. E. (2002) · doi:10.1520/stp1421-eb · Cited by 56 · see citing works
Camera Image Quality Benchmarking
Phillips, J. B., & Eliasson, H. (2017) · Wiley eBooks · doi:10.1002/9781119054504 · Cited by 50 · see citing works
Deep Validation: Toward Detecting Real-World Corner Cases for Deep Neural Networks
Wu, W., Xu, H. F., Zhong, S., Lyu, M. R.-T., & King, I. (2019) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2019.00026 · Cited by 47 · see citing works
Removing chromatic aberration by digital image processing
Chung, S.-W., Kim, B.-K., & Song, W.-J. (2010) · Optical Engineering · doi:10.1117/1.3455506 · Cited by 46 · see citing works
End to End Vehicle Lateral Control Using a Single Fisheye Camera
Toromanoff, M., Wirbel, É., Wilhelm, F., Vejarano, C., Perrotton, X., & Moutarde, F. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594090 · Cited by 32 · see citing works
Automatic Sensor Correction of Autonomous Vehicles by Human-Vehicle Teaching-and-Learning
Guo, L., Manglani, S., Liu, Y., et al. (2018) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2018.2846593 · Cited by 30 · see citing works
Real-Time Photo Sensor Dead Pixel Detection for Embedded Devices
Cho, C.-Y., Chen, T.-M., Wang, W.-S., et al. (2011) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2011.34 · Cited by 25 · see citing works
Separable Kernel for Image Deblurring
Fang, L., Liu, H., Wu, F., et al. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.369 · Cited by 15 · see citing works
Benchmarks of Low-Level Vision Algorithms for DSP, FPGA, and Mobile PC Processors
Baumgartner, D., Roessler, P., Kubinger, W., et al. (2008) · Advances in computer vision and pattern recognition · doi:10.1007/978-1-84800-304-0_5 · Cited by 12 · see citing works
Noise Reduction for Image Signal Processor in Digital Cameras
Baek, Y.-M., Cho, D.-C., Lee, J.-A., et al. (2008) · doi:10.1109/ichit.2008.233 · Cited by 12 · see citing works
RGB-D Object Classification for Autonomous Driving Perception
Premebida, C., Melotti, G., & Asvadi, A. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-030-28603-3_17 · Cited by 12 · see citing works
An optical model of the appearance of blemishes in digital photographs
Zamfir, A., Drîmbărean, A., Zamfir, M., et al. (2007) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.702807 · Cited by 9 · see citing works
Adversarial Robustness Assessment: Why both $L_0$ and $L_\infty$ Attacks Are NecessaryPreprint
Kotyan, S., & Vargas, D. V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06026 · Cited by 9 · see citing works
Approaching Camera-based Real-World Navigation Using Object Recognition
Zheng, Z., He, X., & Weng, J. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.07.320 · Cited by 7 · see citing works
Elkins, S. D. E. (2013) · Routledge eBooks · doi:10.4324/9780080927947 · Cited by 3 · see citing works
An efficient method for demosaicing
Grosssmann, B., & Eldar, Y. (2004) · IEEE Convention of Electrical and Electronics Engineers in Israel · doi:10.1109/eeei.2004.1361185 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 28
- [3][3] D. Chen et al., "Learning by Cheating," Conference on Robot Learning (CoRL), pp. 66-75, 2019. -
- [6][6] E. Allen, and S. Triantaphillidou, "The manual of photography," CRC Press, 2012. -
- [18][18] K. Ren et al., "The Security of Autonomous Driving: Threats, Defenses, and Future Directions," Proceedings of the IEEE, pp. 357372, 2019. -
- [24][24] V. Behzadan, and A. Munir, "Adversarial reinforcement learning framework for benchmarking collision avoidance mechanisms in autonomous vehicles," IEEE Intelligent Transportation Systems Magazine, vol. 13, no. 2, pp. 236-241, 2019. -
- [30][30] A. Carlson et al., "Modeling Camera Effects to Improve Visual Learning from Synthetic Data," Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018. -
- [32][32] S. Hossain et al., "CAIAS simulator: self-driving vehicle simulator for AI research," International Conference on Intelligent Computing & Optimization, pp. 187-195, Springer, Cham, 2018. -
- [37][37] M.-I. Nicolae et al., "Adversarial Robustness Toolbox v1. 0.0.," arXiv preprint arXiv:1807, 2018. -
- [41][41] T. Northrup, "Tony Northrup's photography buying guide: How to choose a camera, lens, tripod, flash, & more," Waterford, Mason Press, 2016. -
- [59][59] S. Bhagavathy, J. Llach and J. f. Zhai, "Multi-Scale Probabilistic Dithering for Suppressing Banding Artifacts in Digital Images," IEEE International Conference on Image Processing, pp. 397-400, 2007. -
- [62][62] L. Shi, et al., "Demosaicing for RGBZ sensor," Computational Imaging XI, Vol. 8657, International Society for Optics and Photonics, 2013. -
- [69][69] S. Lee, E. Eisemann, and H.-P. Seidel, "Real-time lens blur effects and focus control," ACM Trans. Graph. 29, 4, Article 65, 2010. -
- [70][70] F. F. Sabins, and J. M. Ellis, "Remote Sensing: Principles, Interpretation, and Applications," Fourth Edition, Waveland Press, 2020. -
- [73][73] S. C. Bhargava, "Electrical Measuring Instruments and Measurements," CRC Press, 2012. -
- [74][74] S. Kelby, "Scott Kelby's Digital Photography Boxed Set," Parts 1, 2, 3, 4, and 5, Peachpit Press, 2014. -
- [75][75] A. Geiger, P. Lenz, and R. Urtasun, "Are we ready for autonomous driving? The Kitti vision benchmark suite," 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354-3361, IEEE, 2012. -
- [76][76] M. Menze, and A. Geiger, "Object scene flow for autonomous vehicles," Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3061-3070, 2015. -
- [77][77] M. Everingham, et al. "The pascal visual object classes (voc) challenge," International journal of computer vision 88.2:303-338, 2010. -
- [78][78] F. Secci, and A. Ceccarelli, "On failures of RGB cameras and their effects in autonomous driving applications," The 31st International Symposium on Software Reliability Engineering (ISSRE 2020), pp. 13-24, 2020. -
- [81][81] M. Teichmann et al., "Multinet: Real-time joint semantic reasoning for autonomous driving," IEEE Intelligent Vehicles Symposium (IV), pp. 1013-1020, IEEE, 2018. -
- [83][83] B. Wu et al., "Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 129-137, 2017. -
- [85][85] K. He et al., "Deep residual learning for image recognition," Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. -
- [87][87] Y. Chen et al., "Dsgn: Deep stereo geometry network for 3d object detection," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12536-12545, 2020. -
- [88][88] J. Sun et al., "Disp R-CNN: Stereo 3D Object Detection via Shape Prior Guided Instance Disparity Estimation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10548-10557, 2020. -
- [89][89] K. He et al., "Mask r-cnn," Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017. -
- [90][90] P. Li, X. Chen, and S. Shen, "Stereo r-cnn based 3d object detection for autonomous driving," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 76447652, 2019 -
- [93][93] K. Hogan, "Options for Camera Raw in the Digital Workflow," SMPTE, pp. 1-18, 2014. -
- [94][94] X. Chen, et al. "3d object proposals for accurate object class detection," In Advances in Neural Information Processing Systems, pp. 424-432, 2015 -
- [96][96] J. Su, D. Vargas, and K. Sakurai, "One pixel attack for fooling deep neural networks," IEEE Transactions on Evolutionary Computation 23.5 (2019): 828-841. <!-- image --> Andrea Ceccarelli is an Associate Professor in Computer Science at the University of Florence, Florence, Italy. His primary research interests are in the design, monitoring and experimental evaluation of dependable and secure systems, and systems-of-systems. He has been PC co-chair of the conferences SRDS and LADC, and he is a member of the IFIP WG 10.4 on "Dependable Computing and FaultTolerance". His scientific activities originated more than 100 papers which appeared in international conferences, workshops, and journals. <!-- image --> Francesco Secci graduated from the University of Florence, Florence, Italy, very recently, in 2020. After graduating, he moved to the industry where he is currently working in a book distribution company (Libro Co. Italia s.r.l.) with the qualification of software programmer for business management, for the data manipulation and the management of the company's visibility in the social field.
Cited by
Sorted by citations
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
SELMA: SEmantic Large-Scale Multimodal Acquisitions in Variable Weather, Daytime and Viewpoints
Testolina, P., Barbato, F., Michieli, U., Giordani, M., Zanuttigh, P., & Zorzi, M. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3257086 · Cited by 41 · see citing works
Investigating the Impact of Environmental Factors on Autonomous Vehicle Sensors
Csikor, D., Hollósi, J., & Szauter, F. (2025) · SMTS 2025 · doi:10.3390/engproc2025113015
- Authors
- Secci, F., & Ceccarelli, A.
- Affiliations
- University of Florence (IT)
- ORCID
- Ceccarelli, Andrea
- Venue
- 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE)
- Publisher
- IEEE
- Published
- 2020-10
- Pages
- 13-24
- ISSN
- 1071-9458 (ISSN-L)
- OpenAlex
- W3048668306
- Cited by
- 42
- References
- 39
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 4 · 2023: 12 · 2022: 11 · 2021: 5
- FWCI
- 5.77
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings/Proceedings - International Symposium on Software Reliability Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Florence Research (University of Florence) (submittedVersion)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- failure modes, autonomous driving, in-vehicle camera, safety risk, failure mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
