CoChat
Sign Up
PreprintCited by 292Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningRemote-Sensing Image Classification

Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding

Open access
Kendall, Alex, Badrinarayanan, Vijay, Cipolla, Roberto·2015·arXiv
DOI 10.48550/arxiv.1511.02680arXiv 1511.02680
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a deep learning framework for probabilistic pixel-wise semantic segmentation, which we term Bayesian SegNet. Semantic segmentation is an important tool for visual scene understanding and a meaningful measure of uncertainty is essential for decision making. Our contribution is a practical system which is able to predict pixel-wise class labels with a measure of model uncertainty. We achieve this by Monte Carlo sampling with dropout at test time to generate a posterior distribution of pixel class labels. In addition, we show that modelling uncertainty improves segmentation performance by 2-3% across a number of state of the art architectures such as SegNet, FCN and Dilation Network, with no additional parametrisation. We also observe a significant improvement in performance for smaller datasets where modelling uncertainty is more effective. We benchmark Bayesian SegNet on the indoor SUN Scene Understanding and outdoor CamVid driving scenes datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1511.02680

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The most used CNN architectures are SegNet, U-Net, and Mask R-CNN.

    “Nowadays, the most used CNN architectures are the SegNet [3], the U-Net [4]-[6], and the Mask R-CNN [7].”

    From Space-Based Global Maritime Surveillance. Part II: Artificial Intelligence and Data Fusion Techniques (2021) · cites this paper as [3]

  • Techniques for uncertainty estimation can also lessen deep neural networks' susceptibility to adversarial attacks.

    “In addition to producing valuable measures that function as uncertainty measures [358, 359, 66], these techniques can also lessen deep neural networks susceptibility to adversarial attacks [357, 360].”

    From An overview of deep learning in medical imaging focusing on MRI (2019) · cites this paper as [359]

  • The intraobserver agreement of the cartilage lesion detection system was far from perfect, likely due to multiple factors including inherent model uncertainty of the network structure, the stochastic nature of network training, and variation between the training and testing image data sets.

    “However, the intraobserver agreement was far from perfect, which is likely due to multiple factors, including the inherent model uncertainty of the network structure (43,44), the stochastic nature of network training, and the variation between the training and testing image data sets (24,45).”

    From Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. (2018) · cites this paper as [43]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 292

Research works printed but not linked: 42

  1. [1][1] V. Badrinarayanan, F. Galasso, and R. Cipolla. Label propagation in video sequences. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on , pages 3265-3272. IEEE, 2010. 2 -
  2. [2][2] V. Badrinarayanan, A. Handa, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling. arXiv preprint arXiv:1505.07293 , 2015. 3, 5 - · arXiv
  3. [3][3] V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561 , 2015. 2, 3, 4, 5, 6, 9 - · arXiv
  4. [4][4] G. J. Brostow, J. Fauqueur, and R. Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters , 30(2):88-97, 2009. 1, 2, 4, 5, 6, 7, 8 -
  5. [5][5] G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla. Segmentation and recognition using structure from motion point clouds. In Computer Vision-ECCV 2008 , pages 44-57. Springer, 2008. 5 -
  6. [6][6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062 , 2014. 2, 5, 6, 8 - · arXiv
  7. [7][7] D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. Journal of artificial intelligence research , 1996. 2 -
  8. [8][8] C. Couprie, C. Farabet, L. Najman, and Y. LeCun. Indoor semantic segmentation using depth information. arXiv preprint arXiv:1301.3572 , 2013. 1 - · arXiv
  9. [9][9] J. Denker and Y. Lecun. Transforming neural-net output levels to probability distributions. In Advances in Neural Information Processing Systems 3 . Citeseer, 1991. 2, 3 -
  10. [10][10] D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. arXiv preprint arXiv:1411.4734 , 2014. 5, 6 - · arXiv
  11. [11][11] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303-338, 2010. 1, 6, 8, 9 -
  12. [12][12] Y. Gal and Z. Ghahramani. Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv:1506.02158 , 2015. 2, 3 - · arXiv
  13. [13][13] Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. arXiv:1506.02142 , 2015. 2 - · arXiv
  14. [14][14] A. Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems , pages 23482356, 2011. 2, 3 -
  15. [15][15] S. Gupta, R. Girshick, P. Arbel´ aez, and J. Malik. Learning rich features from rgb-d images for object detection and segmentation. In Computer Vision-ECCV 2014 , pages 345-360. Springer, 2014. 6 -
  16. [16][16] B. Hariharan, P. Arbel´ aez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and fine-grained localization. arXiv preprint arXiv:1411.5752 , 2014. 2 - · arXiv
  17. [17][17] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. 5 - · arXiv
  18. [18][18] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 , 2014. 5 - · arXiv
  19. [19][19] A. Kendall and R. Cipolla. Modelling uncertainty in deep learning for camera relocalization. arXiv preprint arXiv:1509.05909 , 2015. 2 - · arXiv
  20. [20][20] P. Kontschieder, S. Rota Bul´ o, H. Bischof, and M. Pelillo. Structured class-labels in random forests for semantic image labelling. In Computer Vision (ICCV), 2011 IEEE International Conference on , pages 2190-2197. IEEE, 2011. 5 -
  21. [21][21] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012. 2 -
  22. [22][22] L. Ladick` y, P. Sturgess, K. Alahari, C. Russell, and P. H. Torr. What, where and how many? combining object detectors and crfs. In Computer Vision-ECCV 2010 , pages 424-437. Springer, 2010. 5 -
  23. [23][23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014 , pages 740-755. Springer, 2014. 8 -
  24. [24][24] C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. T. Freeman. Sift flow: Dense correspondence across different scenes. In Computer VisionECCV 2008 , pages 28-42. Springer, 2008. 6 -
  25. [25][25] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. arXiv preprint arXiv:1411.4038 , 2014. 1, 2, 5, 6, 8 - · arXiv
  26. [26][26] D. J. MacKay. A practical bayesian framework for backpropagation networks. Neural computation , 4(3):448-472, 1992. 2 -
  27. [27][27] H. Noh, S. Hong, and B. Han. Learning deconvolution network for semantic segmentation. arXiv preprint arXiv:1505.04366 , 2015. 2, 5, 6, 8 - · arXiv
  28. [28][28] X. Ren, L. Bo, and D. Fox. Rgb-(d) scene labeling: Features and algorithms. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on , pages 2759-2766. IEEE, 2012. 6 -
  29. [29][29] S. Rota Bulo and P. Kontschieder. Neural decision forests for semantic image labelling. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on , pages 81-88. IEEE, 2014. 5 -
  30. [30][30] J. Shotton, M. Johnson, and R. Cipolla. Semantic texton forests for image categorization and segmentation. In Computer vision and pattern recognition, 2008. CVPR 2008. IEEE Conference on , pages 1-8. IEEE, 2008. 2 -
  31. [31][31] J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore. Real-time human pose recognition in parts from single depth images. Communications of the ACM , 56(1):116-124, 2013. 1, 2 -
  32. [32][32] J. Shotton, J. Winn, C. Rother, and A. Criminisi. Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. International Journal of Computer Vision , 81(1):2-23, 2009. 1, 2 -
  33. [33][33] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In Computer VisionECCV 2012 , pages 746-760. Springer, 2012. 6 -
  34. [34][34] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 3 - · arXiv
  35. [35][35] S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 567-576, 2015. 1, 2, 6, 7, 8, 9 -
  36. [36][36] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014. 2, 3, 4 -
  37. [37][37] P. Sturgess, K. Alahari, L. Ladicky, and P. H. Torr. Combining appearance and structure from motion features for road scene understanding. In BMVC , volume 1, page 6, 2009. 5 -
  38. [38][38] J. Tighe and S. Lazebnik. Superparsing. International Journal of Computer Vision , 101(2):329-349, 2013. 5 -
  39. [39][39] Y. Yang, Z. Li, L. Zhang, C. Murphy, J. Ver Hoeve, and H. Jiang. Local label descriptor for example based semantic image labeling. In Computer Vision-ECCV 2012 , pages 361-375. Springer, 2012. 5 -
  40. [40][40] F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR , 2016. 2, 6, 8 -
  41. [41][41] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014 , pages 818-833. Springer, 2014. 4 -
  42. [42][42] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. Torr. Conditional random fields as recurrent neural networks. arXiv preprint arXiv:1502.03240 , 2015. 2, 8 · arXiv

Authors
Kendall, A., Badrinarayanan, V., & Cipolla, R.
Affiliations
University of Cambridge (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1511.02680
arXiv
1511.02680
OpenAlex
W4300126339
Cited by
292
References
0
Citations by year
2026: 1 · 2025: 8 · 2024: 23 · 2023: 28 · 2022: 35 · 2021: 35 · 2020: 44 · 2019: 51 · 2018: 36 · 2017: 26 · 2016: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Media Technology › Remote-Sensing Image Classification
Keywords
model uncertainty, Monte Carlo dropout, encoder-decoder architecture, pixel classification, scene understanding, uncertainty estimation, small datasets, SUN RGB-D, CamVid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review