CoChat
Sign Up
PreprintCited by 4,213Advanced Neural Network ApplicationsMedical Image Segmentation TechniquesDomain Adaptation and Few-Shot Learning

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Open access
Chen, Liang-Chieh, Papandreou, George, Kokkinos, Iasonas, Murphy, Kevin, Yuille, Alan L.·2014·arXiv
DOI 10.48550/arxiv.1412.7062arXiv 1412.7062
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep Convolutional Neural Networks (DCNNs) have recently shown state of the art performance in high level vision tasks, such as image classification and object detection. This work brings together methods from DCNNs and probabilistic graphical models for addressing the task of pixel-level classification (also called "semantic image segmentation"). We show that responses at the final layer of DCNNs are not sufficiently localized for accurate object segmentation. This is due to the very invariance properties that make DCNNs good for high level tasks. We overcome this poor localization property of deep networks by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF). Qualitatively, our "DeepLab" system is able to localize segment boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set. We show how these results can be obtained efficiently: Careful network re-purposing and a novel application of the 'hole' algorithm from the wavelet community allow dense computation of neural net responses at 8 frames per second on a modern GPU.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1412.7062

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The FCN architecture has a fundamental limitation that learning long-range dependency information becomes challenging due to still limited receptive fields.

    “However, it also raises a fundamental limitation that learning long-range dependency information, critical for semantic segmentation in unconstrained scene images [1,48], becomes challenging due to still limited receptive fields.”

    From Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2021) · cites this paper as [48]

    The Introduction (p. 2) says the fully connected CRF caters for long-range dependencies, supporting their relevance to semantic segmentation. But Section 3.2 (p. 4) describes the network’s receptive fields as large, and Section 4.1 (p. 5) attributes its localization challenge to increased invariance and large receptive fields. The paper does not attribute difficulty learning long-range dependencies to still-limited receptive fields. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Learning long-range dependency information is critical for semantic segmentation in unconstrained scene images.

    “learning long-range dependency information, critical for semantic segmentation in unconstrained scene images [1,48], becomes challenging due to still limited receptive fields.”

    From Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2021) · cites this paper as [48]

  • When a neural network is adapted to a new task, it forgets previously learned knowledge, referred to as catastrophic interference or forgetting.

    “This issue, where a neural network forgets previously learned knowledge when adapted to a new task, is referred to as catastrophic interference or forgetting.”

    From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [30]

    Section 3 (p. 3) describes fine-tuning a pretrained image-classification network for semantic segmentation, but does not say that the network forgets previously learned knowledge. Searches for catastrophic interference, forgetting, and loss of prior knowledge when adapting to a new task found no passage addressing that claim. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • With a few non-local blocks, non-local neural networks are more accurate for video classification than 2D and 3D convolutional networks (including the inflated variant).

    “With a few non-local blocks, our architecures called non-local neural networks are more accurate for video classification than 2D and 3D convolutional networks [48] (including the inflated variant [7]).”

    From Non-local Neural Networks (2018) · cites this paper as [48]

    The paper’s Abstract (p. 1) describes semantic image segmentation using a deep convolutional network and a fully connected CRF, not video classification. Searches for non-local blocks and comparisons with 2D, 3D, or inflated convolutional networks found no passage addressing the claimed accuracy comparison; the mention of frames per second concerns processing speed. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 62 · Cited by 4,213

Research works printed but not linked: 43

  1. Arbel´ aez, P., Pont-Tuset, J., Barron, J. T., Marques, F., and Malik, J. Multiscale combinatorial grouping. In CVPR , 2014.
  2. Bell, S., Upchurch, P., Snavely, N., and Bala, K. Material recognition in the wild with the materials in context database. arXiv:1412.0623 , 2014. · arXiv
  3. Carreira, J., Caseiro, R., Batista, J., and Sminchisescu, C. Semantic segmentation with second-order pooling. In ECCV , 2012.
  4. Chen, L.-C., Papandreou, G., and Yuille, A. Learning a dictionary of shape epitomes with applications to image labeling. In ICCV , 2013.
  5. Chen, L.-C., Schwing, A., Yuille, A., and Urtasun, R. Learning deep structured models. arXiv:1407.2538 , 2014. · arXiv
  6. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv:1606.00915 , 2016. · arXiv
  7. Chen, X. and Yuille, A. L. Articulated pose estimation by a graphical model with image dependent pairwise relations. In NIPS , 2014.
  8. Cogswell, M., Lin, X., Purushwalkam, S., and Batra, D. Combining the best of graphical models and convnets for semantic segmentation. arXiv:1412.4313 , 2014. · arXiv
  9. Dai, J., He, K., and Sun, J. Convolutional feature masking for joint object and stuff segmentation. arXiv:1412.1283 , 2014. · arXiv
  10. Delong, A., Osokin, A., Isack, H. N., and Boykov, Y. Fast approximate energy minimization with label costs. IJCV , 2012.
  11. Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. arXiv:1411.4734 , 2014. · arXiv
  12. Geiger, D. and Girosi, F. Parallel and deterministic algorithms from mrfs: Surface reconstruction. PAMI , 13(5):401-412, 1991.
  13. Geiger, D. and Yuille, A. A common framework for image segmentation. IJCV , 6(3):227-243, 1991.
  14. Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014.
  15. Gonfaus, J. M., Boix, X., Van de Weijer, J., Bagdanov, A. D., Serrat, J., and Gonzalez, J. Harmony potentials for joint classification and segmentation. In CVPR , 2010.
  16. Hariharan, B., Arbel´ aez, P., Bourdev, L., Maji, S., and Malik, J. Semantic contours from inverse detectors. In ICCV , 2011.
  17. Hariharan, B., Arbel´ aez, P., Girshick, R., and Malik, J. Simultaneous detection and segmentation. In ECCV , 2014b.
  18. He, X., Zemel, R. S., and Carreira-Perpindn, M. Multiscale conditional random fields for image labeling. In CVPR , 2004.
  19. Jia, Y ., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093 , 2014. · arXiv
  20. Kohli, P., Ladicky, L., and Torr, P. H. Robust higher order potentials for enforcing label consistency. IJCV , 2009.
  21. Kokkinos, I., Deriche, R., Faugeras, O., and Maragos, P. Computational analysis and learning for a biologically motivated model of boundary detection. Neurocomputing , 71(10):1798-1812, 2008.
  22. Kr¨ ahenb¨ uhl, P. and Koltun, V. Efficient inference in fully connected crfs with gaussian edge potentials. In NIPS , 2011.
  23. Kr¨ ahenb¨ uhl, P. and Koltun, V. Parameter learning and convergent inference for dense random fields. In ICML , 2013.
  24. Ladicky, L., Russell, C., Kohli, P., and Torr, P. H. Associative hierarchical crfs for object class image segmentation. In ICCV , 2009.
  25. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. In Proc. IEEE , 1998.
  26. Lempitsky, V., Vedaldi, A., and Zisserman, A. Pylon model for semantic segmentation. In NIPS , 2011.
  27. Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. arXiv:1411.4038 , 2014. · arXiv
  28. Lucchi, A., Li, Y., Boix, X., Smith, K., and Fua, P. Are spatial and global constraints really necessary for segmentation? In ICCV , 2011.
  29. Mallat, S. A Wavelet Tour of Signal Processing . Acad. Press, 2 edition, 1999.
  30. Mostajabi, M., Yadollahpour, P., and Shakhnarovich, G. Feedforward semantic segmentation with zoom-out features. arXiv:1412.0774 , 2014. · arXiv
  31. Papandreou, G., Kokkinos, I., and Savalle, P.-A. Untangling local and global deformations in deep convolutional networks for image classification and sliding window detection. arXiv:1412.0296 , 2014. · arXiv
  32. Papandreou, G., Chen, L.-C., Murphy, K., and Yuille, A. L. Weakly- and semi-supervised learning of a DCNN for semantic image segmentation. arXiv:1502.02734 , 2015. · arXiv
  33. Rother, C., Kolmogorov, V., and Blake, A. Grabcut: Interactive foreground extraction using iterated graph cuts. In SIGGRAPH , 2004.
  34. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv:1312.6229 , 2013. · arXiv
  35. Shotton, J., Winn, J., Rother, C., and Criminisi, A. Textonboost for image understanding: Multiclass object recognition and segmentation by jointly modeling texture, layout, and context. IJCV , 2009.
  36. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 , 2014. · arXiv
  37. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. arXiv:1409.4842 , 2014. · arXiv
  38. Tompson, J., Jain, A., LeCun, Y., and Bregler, C. Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation. In NIPS , 2014.
  39. Uijlings, J., van de Sande, K., Gevers, T., and Smeulders, A. Selective search for object recognition. IJCV , 2013.
  40. Wang, P., Shen, X., Lin, Z., Cohen, S., Price, B., and Yuille, A. Towards unified depth and semantic prediction from a single image. In CVPR , 2015.
  41. Yadollahpour, P., Batra, D., and Shakhnarovich, G. Discriminative re-ranking of diverse segmentations. In CVPR , 2013.
  42. Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In ECCV , 2014.
  43. Zhang, N., Donahue, J., Girshick, R., and Darrell, T. Part-based r-cnns for fine-grained category detection. In ECCV , 2014.

Authors
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2014
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1412.7062
arXiv
1412.7062
OpenAlex
W1923697677
Cited by
4,213
References
62
Citations by year
2026: 23 · 2025: 186 · 2024: 376 · 2023: 413 · 2022: 402 · 2021: 520 · 2020: 565 · 2019: 525 · 2018: 388 · 2017: 464 · 2016: 285 · 2015: 63 · 2014: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
semantic segmentation, deep convolutional neural networks, conditional random fields, pixel classification, boundary localization, DeepLab, probabilistic graphical models, GPU acceleration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review