Object Instance Segmentation and Fine-Grained Localization Using Hypercolumns
Abstract
Recognition algorithms based on convolutional networks (CNNs) typically use the output of the last layer as a feature representation. However, the information in this layer may be too coarse spatially to allow precise localization. On the contrary, earlier layers may be precise in localization but will not capture semantics. To get the best of both worlds, we define the hypercolumn at a pixel as the vector of activations of all CNN units above that pixel. Using hypercolumns as pixel descriptors, we show results on three fine-grained localization tasks: simultaneous detection and segmentation, where we improve state-of-the-art from 49.7 mean APr to 62.4, keypoint localization, where we get a 3.3 point boost over a strong regression baseline using CNN features, and part labeling, where we show a 6.6 point gain over a strong baseline.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work has shown that CNNs have a remarkable ability to localize objects in CONV layers, an ability weakened in FC layers.
“Recent work has shown that CNNs have a remarkable ability to localize objects in CONV layers (Zhou et al. 2015(Zhou et al. , 2016a;;Cinbis et al. 2017;Oquab et al. 2015;Hariharan et al. 2016), an 11 ability which is weakened in the FC layers.”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Hariharan et al. 2016
SSD effectively combines ideas from RPN in Faster RCNN, YOLO and multiscale CONV features to achieve fast detection speed while retaining high detection quality.
“SSD effectively combines ideas from RPN in Faster RCNN (Ren et al. 2015), YOLO (Redmon et al. 2016) and multiscale CONV features (Hariharan et al. 2016) to achieve fast detection speed, while still retaining high detection quality.”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Hariharan et al. 2016
The sub-sampling layers in a CNN feature hierarchy lead to an inherent multiscale pyramid, producing feature maps at different spatial resolutions, but subject to challenges.
“Since a CNN computes its feature hierarchy layer by layer, the sub-sampling layers in the feature hierarchy already lead to an inherent multiscale pyramid, producing feature maps at different spatial resolutions, but subject to challenges (Hariharan et al. 2016;Long et al. 2015;Shrivastava et al. 2017).”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Hariharan et al. 2016
Features from different layers are complementary to each other and can improve detection accuracy.
“Features from different layers are complementary to each other and can improve detection accuracy, as shown by Hypercolumns (Hariharan et al. 2016), HyperNet (Kong et al. 2016) and ION (Bell et al. 2016).”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Hariharan et al. 2016
Combining features from multiple CNN layers before making a prediction is commonly accomplished via concatenation.
“Such feature combination is commonly accomplished via concatenation, a classic neural network idea that concatenates features from different layers, architectures which have recently become popular for semantic segmentation (Long et al. 2015;Shelhamer et al. 2017;Hariharan et al. 2016).”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Hariharan et al. 2016
The citation lists haven’t been fetched yet
References 57 · Cited by 102
- Authors
- Hariharan, B., Arbelaez, P., Girshick, R., & Malik, J.
- Affiliations
- Meta (United States) (US)
- University of California, Berkeley (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2017-04-01
- Volume
- 39
- Issue
- 4
- Pages
- 627-639
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W2418346537
- Cited by
- 102
- References
- 57
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 4 · 2023: 15 · 2022: 13 · 2021: 18 · 2020: 17 · 2019: 11 · 2018: 9 · 2017: 5
- FWCI
- 5.66
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- ONR
- MURI (N000141010933)
- Google Research
- Microsoft Research
- NVIDIA
- GPUs
- University of California Berkeley
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- hypercolumns, fine-grained localization, keypoint detection, spatial localization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
