Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding
Abstract
© 2017. The copyright of this document resides with its authors. We present a deep learning framework for probabilistic pixel-wise semantic segmentation, which we term Bayesian SegNet. Semantic segmentation is an important tool for visual scene understanding and a meaningful measure of uncertainty is essential for decision making. Our contribution is a practical system which is able to predict pixel-wise class labels with a measure of model uncertainty using Bayesian deep learning. We achieve this by Monte Carlo sampling with dropout at test time to generate a posterior distribution of pixel class labels. In addition, we show that modelling uncertainty improves segmentation performance by 2-3% across a number of datasets and architectures such as SegNet, FCN, Dilation Network and DenseNet.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Fully convolutional networks for semantic segmentationPreprint
Long, J., Shelhamer, E., & Darrell, T. (2014) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1605.06211 · Cited by 43,601 · see citing works
(2014) · Cited by 33,419 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Self-citationVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,749 · see citing works
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213 · see citing works
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917 · see citing works
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · see citing works
Real-time human pose recognition in parts from single depth images
Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A. W., Finocchio, M., Blake, A., Cook, M., & Moore, R. (2013) · Communications of the ACM · doi:10.1145/2398356.2398381 · Cited by 2,050 · see citing works
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · see citing works
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528 · see citing works
Practical Variational Inference for Neural Networks
Graves, A. (2011) · Cited by 1,079 · see citing works
Segmentation and Recognition Using Structure from Motion Point Clouds
Brostow, G. J., Shotton, J., Fauqueur, J., & Cipolla, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_5 · Cited by 1,050 · see citing works
Self-citationSemantic texton forests for image categorization and segmentation
Shotton, J., Johnson, M., & Cipolla, R. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587503 · Cited by 1,027 · see citing works
Self-citationShotton, J., Winn, J. M., Rother, C., & Criminisi, A. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0109-1 · Cited by 1,014 · see citing works
Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational InferencePreprint
Gal, Y., & Ghahramani, Z. (2015) · arXiv.org · doi:10.48550/arxiv.1506.02158 · Cited by 839 · see citing works
Scene Text Recognition using Higher Order Language Priors
Mishra, A., Alahari, K., & Jawahar, C. V. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.127 · Cited by 837 · see citing works
Learning Deconvolution Network for Semantic SegmentationPreprint
Noh, H., Hong, S., & Han, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04366 · Cited by 613 · see citing works
Modelling uncertainty in deep learning for camera relocalization
Kendall, A., & Cipolla, R. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487679 · Cited by 548 · cited 1× · see citing works
“This technique has seen success in modelling uncertainty for camera relocalisation [19].”
MethodSelf-citationIndoor Semantic Segmentation using depth information
Couprie, C., Farabet, C., Najman, L. A., & LeCun, Y. (2013) · HAL · doi:10.48550/arxiv.1301.3572 · Cited by 451 · see citing works
RGB-(D) scene labeling: Features and algorithms
Ren, X., Bo, L., & Fox, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247999 · Cited by 410 · see citing works
What, Where and How Many? Combining Object Detectors and CRFs
Ladický, Ľ., Sturgess, P., Alahari, K., Russell, C., & Torr, P. H. S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_31 · Cited by 312 · see citing works
Transforming Neural-Net Output Levels to Probability Distributions
Denker, J. S., & LeCun, Y. (1990) · Cited by 223 · see citing works
Structured class-labels in random forests for semantic image labelling
Kontschieder, P., Bulò, S. R., Bischof, H., & Pelillo, M. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126496 · Cited by 209 · see citing works
Semantic Segmentation of Urban Scenes Using Dense Depth Maps
Zhang, C., Wang, L., & Yang, R. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_51 · Cited by 179 · see citing works
Label propagation in video sequences
Badrinarayanan, V., Galasso, F., & Cipolla, R. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540054 · Cited by 141 · see citing works
Self-citationNeural Decision Forests for Semantic Image Labelling
Bulò, S. R., & Kontschieder, P. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.18 · Cited by 133 · see citing works
Eigen, D., & Fergus, R. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4734 · Cited by 126 · see citing works
Local Label Descriptor for Example Based Semantic Image Labeling
Yang, Y., Li, Z., Zhang, L., Murphy, C., Hoeve, J. V., & Jiang, H. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33786-4_27 · Cited by 28 · see citing works
Hypercolumns for Object Segmentation and Fine-grained LocalizationPreprint
Hariharan, B., Arbeláez, P., Girshick, R., & Malik, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5752 · Cited by 25 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Self-citationImage segmentation using deep learning: A survey
Minaee, S., Boykov, Y. Y., Porikli, F., Plaza, A. J., Kehtarnavaz, N., & Terzopoulos, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059968 · Cited by 3,796 · see citing works
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · see citing works
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287 · see citing works
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 2,720 · see citing works
“[43] 2015 SegNet [44] semantic segmentation √”
BackgroundMethodFeng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169 · see citing works
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · see citing works
3D Graph Neural Networks for RGBD Semantic Segmentation
Qi, X., Liao, R., Jia, J., Fidler, S., & Urtasun, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.556 · Cited by 482 · see citing works
Leveraging uncertainty information from deep neural networks for disease detection
Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-17876-z · Cited by 510 · see citing works
UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders
Zhang, J., Fan, D.-P., Dai, Y., Anwar, S., Saleh, F. S., Zhang, T., & Barnes, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00861 · Cited by 423 · see citing works
Data Uncertainty Learning in Face Recognition
Chang, J., Lan, Z., Cheng, C., & Wei, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00575 · Cited by 320 · see citing works
FuseSeg: Semantic segmentation of urban scenes based on RGB and thermal data fusion
Sun, Y., Zuo, W., Yun, P., Wang, H., & Liu, M. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2993143 · Cited by 270 · see citing works
Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges
Siam, M., Elkerdawy, S., Jagersand, M., & Yogamani, S. (2017) · 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317714 · Cited by 178 · see citing works
Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal OCT
Seebock, P., Orlando, J. I., Schlegl, T., Waldstein, S. M., Bogunovic, H., Klimscha, S., Langs, G., & Schmidt-Erfurth, U. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2919951 · Cited by 164 · see citing works
Hiasa, Y., Otake, Y., Takao, M., Ogawa, T., Sugano, N., & Sato, Y. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2940555 · Cited by 157 · see citing works
Roy, A. G., Conjeti, S., Navab, N., & Wachinger, C. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.042 · Cited by 137 · see citing works
Soldi, G., Gaglione, D., Forti, N., Millefiori, L. M., Braca, P., Carniel, S., Simone, A. D., Iodice, A., Riccio, D., Daffina, F. C., et al. (2021) · IEEE Aerospace and Electronic Systems Magazine · doi:10.1109/maes.2021.3070884 · Cited by 44 · see citing works
“Nowadays, the most used CNN architectures are the SegNet [3], the U-Net [4]–[6], and the Mask R-CNN [7].”
MethodSpinDrop: Dropout-Based Bayesian Binary Neural Networks With Spintronic Implementation
Ahmed, S. T., Danouchi, K., Münch, C., Prenat, G., Anghel, L., & Tahoori, M. B. (2023) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2023.3242146 · Cited by 25 · see citing works
Blind spot detection for safe sim-to-real transfer
Ramakrishnan, R., Kamar, E., Dey, D., Horvitz, E., & Shah, J. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11436 · Cited by 21 · see citing works
Conformal Semantic Image Segmentation: Post-hoc Quantification of Predictive Uncertainty
Mossina, L., Dalmau, J., & Andéol, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00361 · Cited by 18 · see citing works
Scalable Uncertainty for Computer Vision With Functional Variational Inference
Carvalho, E. D. C., Clark, R., Nicastro, A., & Kelly, P. H. J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01202 · Cited by 14 · see citing works
- Authors
- Kendall, A., Badrinarayanan, V., & Cipolla, R.
- Affiliations
- University of Cambridge (GB)
- Venue
- Procedings of the British Machine Vision Conference 2017
- Publisher
- British Machine Vision Association
- Published
- 2017
- OpenAlex
- W2103328396
- Cited by
- 784
- References
- 36
- Citations by year
- 2026: 17 · 2025: 34 · 2024: 38 · 2023: 65 · 2022: 64 · 2021: 127 · 2020: 145 · 2019: 127 · 2018: 96 · 2017: 56 · 2016: 15
- FWCI
- 33.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- British Machine Vision Conference (BMVC) (publishedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- semantic segmentation, model uncertainty, Monte Carlo dropout, encoder-decoder architecture, scene understanding, uncertainty estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
