CoChat
Sign Up
PreprintCited by 22,392Advanced Neural Network ApplicationsMedical Image Segmentation TechniquesDomain Adaptation and Few-Shot Learning

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Open access
Chen, Liang-Chieh, Papandreou, George, Kokkinos, Iasonas, Murphy, Kevin, Yuille, Alan L.·2016·arXiv
DOI 10.48550/arxiv.1606.00915arXiv 1606.00915
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution', as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7% mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1606.00915

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Depth estimation is necessary for 3D perception from cameras, which is possible with a single camera though stereo or multi-view systems are more robust.

    “Depth estimation is therefore necessary, which is in fact possible with a single camera [158] though stereo or multi-view systems are more robust [159].”

    From A Survey of Autonomous Driving: <i>Common Practices and Emerging Technologies</i> (2020) · cites this paper as [158]

    The Introduction describes DeepLab as a semantic image segmentation method, not a depth-estimation method. Searches for monocular depth and 3D perception found only titles of depth-prediction studies in the References (page not provided); no passage states that this paper estimates depth from a single camera. The stereo or multi-view comparison is attached to a different citation and is not judged here. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 95

  1. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," in Proc. IEEE, 1998.
  2. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "Overfeat: Integrated recognition, localization and detection using convolutional networks," arXiv:1312.6229, 2013. · arXiv
  3. K. Simonyan and A. Zisserman, "Very deep convolutional net- works for large-scale image recognition," in ICLR, 2015.
  4. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," arXiv:1409.4842, 2014. · arXiv
  5. G. Papandreou, I. Kokkinos, and P.-A. Savalle, "Modeling local and global deformations in deep learning: Epitomic convolution, multiple instance learning, and sliding window detection," in CVPR, 2015.
  6. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmenta- tion," in CVPR, 2014.
  7. D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov, "Scalable object detection using deep neural networks," in CVPR, 2014.
  8. R. Girshick, "Fast r-cnn," in ICCV, 2015.
  9. S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real- time object detection with region proposal networks," in NIPS, 2015.
  10. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed, "SSD: Single shot multibox detector," arXiv:1512.02325, 2015. · arXiv
  11. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in ECCV, 2014.
  12. J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional net- works for semantic segmentation," in CVPR, 2015.
  13. M. Holschneider, R. Kronland-Martinet, J. Morlet, and P. Tchamitchian, "A real-time algorithm for signal analysis with the help of the wavelet transform," in Wavelets: Time-Frequency Methods and Phase Space, 1989, pp. 289-297.
  14. L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, "Attention to scale: Scale-aware semantic image segmentation," in CVPR, 2016.
  15. I. Kokkinos, "Pushing the boundaries of boundary detection using deep learning," in ICLR, 2016.
  16. S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene cate- gories," in CVPR, 2006.
  17. K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition," in ECCV, 2014.
  18. B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik, "Hyper- columns for object segmentation and fine-grained localization," in CVPR, 2015.
  19. P. Krähenb ühl and V. Koltun, "Efficient inference in fully con- nected crfs with gaussian edge potentials," in NIPS, 2011.
  20. C. Rother, V. Kolmogorov, and A. Blake, "GrabCut: Interactive foreground extraction using iterated graph cuts," in SIGGRAPH, 2004.
  21. J. Shotton, J. Winn, C. Rother, and A. Criminisi, "Textonboost for image understanding: Multi-class object recognition and segmen- tation by jointly modeling texture, layout, and context," IJCV, 2009.
  22. A. Lucchi, Y. Li, X. Boix, K. Smith, and P. Fua, "Are spatial and global constraints really necessary for segmentation?" in ICCV, 2011.
  23. X. He, R. S. Zemel, and M. Carreira-Perpindn, "Multiscale condi- tional random fields for image labeling," in CVPR, 2004.
  24. L. Ladicky, C. Russell, P. Kohli, and P. H. Torr, "Associative hierarchical crfs for object class image segmentation," in ICCV, 2009.
  25. V. Lempitsky, A. Vedaldi, and A. Zisserman, "Pylon model for semantic segmentation," in NIPS, 2011.
  26. A. Delong, A. Osokin, H. N. Isack, and Y. Boykov, "Fast approxi- mate energy minimization with label costs," IJCV, 2012.
  27. J. M. Gonfaus, X. Boix, J. Van de Weijer, A. D. Bagdanov, J. Serrat, and J. Gonzalez, "Harmony potentials for joint classification and segmentation," in CVPR, 2010.
  28. P. Kohli, P. H. Torr et al., "Robust higher order potentials for enforcing label consistency," IJCV, vol. 82, no. 3, pp. 302-324, 2009.
  29. L.-C. Chen, G. Papandreou, and A. Yuille, "Learning a dictionary of shape epitomes with applications to image labeling," in ICCV, 2013.
  30. P. Wang, X. Shen, Z. Lin, S. Cohen, B. Price, and A. Yuille, "Towards unified depth and semantic prediction from a single image," in CVPR, 2015.
  31. M. Everingham, S. M. A. Eslami, L. V. Gool, C. K. I. Williams, J. Winn, and A. Zisserma, "The pascal visual object classes challenge a retrospective," IJCV, 2014.
  32. R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille, "The role of context for object detection and semantic segmentation in the wild," in CVPR, 2014.
  33. X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille, "Detect what you can: Detecting and representing objects using holistic models and body parts," in CVPR, 2014.
  34. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The cityscapes dataset for semantic urban scene understanding," in CVPR, 2016.
  35. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Semantic image segmentation with deep convolutional nets and fully connected crfs," in ICLR, 2015.
  36. G. Lin, C. Shen, I. Reid et al., "Efficient piecewise training of deep structured models for semantic segmentation," arXiv:1504.01013, 2015. · arXiv
  37. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, "Caffe: Convolutional architecture for fast feature embedding," arXiv:1408.5093, 2014. · arXiv
  38. Z. Tu and X. Bai, "Auto-context and its application to high- level vision tasks and 3d brain image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 10, pp. 1744-1757, 2010.
  39. J. Shotton, M. Johnson, and R. Cipolla, "Semantic texton forests for image categorization and segmentation," in CVPR, 2008.
  40. B. Fulkerson, A. Vedaldi, and S. Soatto, "Class segmentation and object localization with superpixel neighborhoods," in ICCV, 2009.
  41. J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu, "Semantic segmentation with second-order pooling," in ECCV, 2012.
  42. J. Carreira and C. Sminchisescu, "CPMC: Automatic object seg- mentation using constrained parametric min-cuts," PAMI, vol. 34, no. 7, pp. 1312-1328, 2012.
  43. P. Arbeláez, J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik, "Multiscale combinatorial grouping," in CVPR, 2014.
  44. J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders, "Selective search for object recognition," IJCV, 2013.
  45. B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik, "Simultane- ous detection and segmentation," in ECCV, 2014.
  46. M. Mostajabi, P. Yadollahpour, and G. Shakhnarovich, "Feedfor- ward semantic segmentation with zoom-out features," in CVPR, 2015.
  47. J. Dai, K. He, and J. Sun, "Convolutional feature masking for joint object and stuff segmentation," arXiv:1412.1283, 2014. · arXiv
  48. D. Eigen and R. Fergus, "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture," arXiv:1411.4734, 2014. · arXiv
  49. M. Cogswell, X. Lin, S. Purushwalkam, and D. Batra, "Combining the best of graphical models and convnets for semantic segmen- tation," arXiv:1412.4313, 2014. · arXiv
  50. D. Geiger and F. Girosi, "Parallel and deterministic algorithms from mrfs: Surface reconstruction," PAMI, vol. 13, no. 5, pp. 401- 412, 1991.
  51. D. Geiger and A. Yuille, "A common framework for image segmentation," IJCV, vol. 6, no. 3, pp. 227-243, 1991.
  52. I. Kokkinos, R. Deriche, O. Faugeras, and P. Maragos, "Computa- tional analysis and learning for a biologically motivated model of boundary detection," Neurocomputing, vol. 71, no. 10, pp. 1798- 1812, 2008.
  53. S. Bell, P. Upchurch, N. Snavely, and K. Bala, "Material recog- nition in the wild with the materials in context database," arXiv:1412.0623, 2014. · arXiv
  54. G. Papandreou, L.-C. Chen, K. Murphy, and A. L. Yuille, "Weakly-and semi-supervised learning of a dcnn for semantic image segmentation," in ICCV, 2015.
  55. S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. Torr, "Conditional random fields as recurrent neural networks," in ICCV, 2015.
  56. J. Dai, K. He, and J. Sun, "Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation," in ICCV, 2015.
  57. H. Noh, S. Hong, and B. Han, "Learning deconvolution network for semantic segmentation," in ICCV, 2015.
  58. Z. Liu, X. Li, P. Luo, C. C. Loy, and X. Tang, "Semantic image segmentation via deep parsing network," in ICCV, 2015.
  59. L.-C. Chen, J. T. Barron, G. Papandreou, K. Murphy, and A. L. Yuille, "Semantic image segmentation with task-specific edge detection using cnns and a discriminatively trained domain transform," in CVPR, 2016.
  60. L.-C. Chen, A. Schwing, A. Yuille, and R. Urtasun, "Learning deep structured models," in ICML, 2015.
  61. A. G. Schwing and R. Urtasun, "Fully connected deep structured networks," arXiv:1503.02351, 2015. · arXiv
  62. S. Chandra and I. Kokkinos, "Fast, exact and multi-scale inference for semantic image segmentation with deep Gaussian CRFs," arXiv:1603.08358, 2016. · arXiv
  63. E. S. L. Gastal and M. M. Oliveira, "Domain transform for edge- aware image and video processing," in SIGGRAPH, 2011.
  64. G. Bertasius, J. Shi, and L. Torresani, "High-for-low and low-for- high: Efficient boundary detection from deep object features and its applications to high-level vision," in ICCV, 2015.
  65. P. O. Pinheiro and R. Collobert, "Weakly supervised seman- tic segmentation with convolutional networks," arXiv:1411.6228, 2014. · arXiv
  66. S. Hong, H. Noh, and B. Han, "Decoupled deep neural network for semi-supervised semantic segmentation," in NIPS, 2015.
  67. A. Vezhnevets, V. Ferrari, and J. M. Buhmann, "Weakly su- pervised semantic segmentation with a multi-image model," in ICCV, 2011.
  68. X. Liang, Y. Wei, X. Shen, J. Yang, L. Lin, and S. Yan, "Proposal- free network for instance-level object segmentation," arXiv preprint arXiv:1509.02636, 2015. · arXiv
  69. J. E. Fowler, "The redundant discrete wavelet transform and additive noise," IEEE Signal Processing Letters, vol. 12, no. 9, pp. 629-632, 2005.
  70. P. P. Vaidyanathan, "Multirate digital filters, filter banks, polyphase networks, and applications: a tutorial," Proceedings of the IEEE, vol. 78, no. 1, pp. 56-93, 1990.
  71. F. Yu and V. Koltun, "Multi-scale context aggregation by dilated convolutions," in ICLR, 2016.
  72. J. Dai, Y. Li, K. He, and J. Sun, "R-fcn: Object detection via region- based fully convolutional networks," arXiv:1605.06409, 2016. · arXiv
  73. J. Dai, K. He, Y. Li, S. Ren, and J. Sun, "Instance-sensitive fully convolutional networks," arXiv:1603.08678, 2016. · arXiv
  74. K. Chen, J. Wang, L.-C. Chen, H. Gao, W. Xu, and R. Nevatia, "Abc-cnn: An attention based convolutional neural network for visual question answering," arXiv:1511.05960, 2015. · arXiv
  75. L. Sevilla-Lara, D. Sun, V. Jampani, and M. J. Black, "Op- tical flow with semantic segmentation and localized layers," arXiv:1603.03911, 2016. · arXiv
  76. Z. Wu, C. Shen, and A. van den Hengel, "High-performance semantic segmentation using very deep fully convolutional net- works," arXiv:1604.04339, 2016. · arXiv
  77. M. J. Shensa, "The discrete wavelet transform: wedding the a trous and mallat algorithms," Signal Processing, IEEE Transactions on, vol. 40, no. 10, pp. 2464-2482, 1992.
  78. M. Abadi, A. Agarwal et al., "Tensorflow: Large-scale machine learning on heterogeneous distributed systems," arXiv:1603.04467, 2016. · arXiv
  79. B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik, "Semantic contours from inverse detectors," in ICCV, 2011.
  80. W. Liu, A. Rabinovich, and A. C. Berg, "Parsenet: Looking wider to see better," arXiv:1506.04579, 2015. · arXiv
  81. T.-Y. Lin et al., "Microsoft COCO: Common objects in context," in ECCV, 2014.
  82. R. Vemulapalli, O. Tuzel, M.-Y. Liu, and R. Chellappa, "Gaussian conditional random field network for semantic segmentation," in CVPR, 2016.
  83. Z. Yan, H. Zhang, Y. Jia, T. Breuel, and Y. Yu, "Combining the best of convolutional layers and recurrent layers: A hybrid network for semantic segmentation," arXiv:1603.04871, 2016. · arXiv
  84. G. Ghiasi and C. C. Fowlkes, "Laplacian reconstruction and refinement for semantic segmentation," arXiv:1605.02264, 2016. · arXiv
  85. A. Arnab, S. Jayasumana, S. Zheng, and P. Torr, "Higher order potentials in end-to-end trainable conditional random fields," arXiv:1511.08119, 2015. · arXiv
  86. F. Shen and G. Zeng, "Fast semantic image segmentation with high order context and guided filtering," arXiv:1605.04068, 2016. · arXiv
  87. Z. Wu, C. Shen, and A. van den Hengel, "Bridging category-level and instance-level semantic image segmentation," arXiv:1605.06885, 2016. · arXiv
  88. K. He, X. Zhang, S. Ren, and J. Sun, "Identity mappings in deep residual networks," arXiv:1603.05027, 2016. · arXiv
  89. F. Xia, P. Wang, L.-C. Chen, and A. L. Yuille, "Zoom better to see clearer: Huamn part segmentation with auto zoom net," arXiv:1511.06881, 2015. · arXiv
  90. X. Liang, X. Shen, D. Xiang, J. Feng, L. Lin, and S. Yan, "Se- mantic object parsing with local-global long short-term memory," arXiv:1511.04510, 2015. · arXiv
  91. X. Liang, X. Shen, J. Feng, L. Lin, and S. Yan, "Semantic object parsing with graph lstm," arXiv:1603.07063, 2016. · arXiv
  92. J. Wang and A. Yuille, "Semantic part segmentation using com- positional model combining shape and appearance," in CVPR, 2015.
  93. P. Wang, X. Shen, Z. Lin, S. Cohen, B. Price, and A. Yuille, "Joint object and part segmentation using deep learned potentials," in ICCV, 2015.
  94. V. Badrinarayanan, A. Kendall, and R. Cipolla, "Segnet: A deep convolutional encoder-decoder architecture for image segmenta- tion," arXiv:1511.00561, 2015. · arXiv
  95. J. Uhrig, M. Cordts, U. Franke, and T. Brox, "Pixel-level en- coding and depth layering for instance-level semantic labeling," arXiv:1604.05096, 2016. · arXiv

Cited by

Sorted by citations

Authors
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L.
Affiliations
  • Google (United States) (US)
  • University College London (GB)
  • Johns Hopkins University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
Volume
40
Issue
4
Pages
834-848
ISSN
0162-8828 (ISSN-L)
DOI
10.48550/arxiv.1606.00915
arXiv
1606.00915
OpenAlex
W2412782625
Cited by
22,392
References
3
Citations by year
2026: 1655 · 2025: 2789 · 2024: 3153 · 2023: 3262 · 2022: 3039 · 2021: 2885 · 2020: 2282 · 2019: 1914 · 2018: 1035 · 2017: 332 · 2016: 40 · 2015: 1 · 2014: 2
FWCI
631.17
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • Horizon 2020 Framework Programme
  • Army Research Office
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
semantic segmentation, dilated convolution, atrous spatial pyramid pooling, conditional random fields, dense prediction, boundary localization, Pascal VOC 2012, Pascal Context, cityscape, probabilistic graphical models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review