CoChat
Sign Up
Cited by 657Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningVisual Attention and Saliency Detection

UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory

Open access
Kokkinos, Iasonas·2017·2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2017.579
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work we train in an end-to-end manner a convolutional neural network (CNN) that jointly handles low-, mid-, and high-level vision tasks in a unified architecture. Such a network can act like a swiss knife for vision tasks, we call it an UberNet to indicate its overarching nature. The main contribution of this work consists in handling challenges that emerge when scaling up to many tasks. We introduce techniques that facilitate (i) training a deep architecture while relying on diverse training sets and (ii) training many (potentially unlimited) tasks with a limited memory budget. This allows us to train in an end-to-end manner a unified CNN architecture that jointly handles (a) boundary detection (b) normal estimation (c) saliency estimation (d) semantic segmentation (e) human part segmentation (f) semantic boundary detection, (g) region proposal generation and object detection. We obtain competitive performance while jointly addressing all tasks in 0.7 seconds on a GPU. Our system will be made publicly available.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2017.579

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The UPerNet head used in BEiT and MAE operates on a feature pyramid.

    “The UPerNet [90] head used in BEiT [9] and MAE [35] operates on a feature pyramid [50].”

    From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [35]

    In §2, UberNet describes pooling and fusing features from multiple VGG-16 layers; the Figure 2 caption also describes an image pyramid. These passages support a related multiscale approach, but they do not establish that the UPerNet head operates on a feature pyramid. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

  • Tasks in UberNet do not form the hierarchy of visual concepts.

    “Moreover, although tasks in [25] are formed from low level to high level, such as boundary detection, semantic segmentation and object detection, these tasks do not form the hierarchy of visual concepts.”

    From Unified Perceptual Parsing for Scene Understanding (2018) · cites this paper as [25]

  • Masked Autoencoders (MAEs) are a powerful, conceptually simple and efficient self-supervised pre-training strategy for Vision Transformers (ViTs).

    “Masked Autoencoders (MAEs) [35] have recently been demonstrated to be a powerful, yet conceptually simple and efficient, self-supervised pre-training strategy for Vision Transformers [26] (ViTs).”

    From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [35]

    The Method passages describe training UberNet, a convolutional network, and the supplied passages do not address masked autoencoders or self-supervised pre-training of Vision Transformers. Searches for “Masked Autoencoders” and “masked autoencoder transformer self-supervised pretraining” found no relevant passage in our copy; page numbers were not provided. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • The MAE pre-training objective has been limited to a single modality, namely RGB images, and does not make use of any other modalities that are optionally present.

    “So far, however, the MAE pre-training objective has been limited to a single modality, namely RGB images, and does not make use of any other modalities that are optionally present.”

    From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [35]

    The Multi-Task Training section describes UberNet’s per-task losses, and the Experimental Settings section describes pretraining for segmentation and detection. Searches for MAE pretraining, RGB-only input, and optional modalities found no passage addressing whether the MAE objective uses other modalities. Our copy therefore cannot verify the attributed claim; no page numbers were provided for these passages. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Predicting per-patch standardized RGB patches slightly improves transfer performance.

    “They find that predicting standardized patches slightly improves transfer performance.”

    From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [35]

    The Method passages report boundary-detection results and discuss batch normalization, but do not describe predicting standardized RGB patches. Searches for standardized patches and RGB patch normalization found no passage linking such predictions to improved transfer performance. Page numbers were unavailable in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 98

  1. A. Bansal, B. Russell, and A. Gupta. Marr revisited: 2d-3d alignment via surface normal prediction. In Proc. CVPR, 2016.
  2. V. Belagiannis and A. Zisserman. Recurrent human pose estimation. CoRR, abs/1605.02914, 2016. · arXiv
  3. S. Bell, P. Upchurch, N. Snavely, and K. Bala. Mate- rial recognition in the wild with the materials in con- text database. In Proc. CVPR, 2015.
  4. S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. Proc. CVPR, 2016.
  5. G. Bertasius, J. Shi, and L. Torresani. Deepedge: A multi-scale bifurcated deep network for top-down contour detection. In Proc. CVPR, 2015.
  6. G. Bertasius, J. Shi, and L. Torresani. High-for-low and low-for-high: Efficient boundary detection from deep object features and its applications to high-level vision. In Proc. ICCV, 2015.
  7. H. Bilen and A. Vedaldi. Integrated perception with recurrent multi-task neural networks. In Proc. NIPS, 2016.
  8. L. Bottou, Y. Bengio, and Y. LeCun. Global training of document processing systems using graph trans- former networks. In Proc. CVPR, 1997.
  9. L. D. Bourdev, S. Maji, and J. Malik. Describing people: A poselet-based approach to attribute classi- fication. In Proc. ICCV, 2011.
  10. S. Chandra and I. Kokkinos. Fast, exact and multi- scale inference for semantic image segmentation with deep gaussian crfs. In Proc. ECCV, 2016.
  11. L. Chen, J. T. Barron, G. Papandreou, K. Murphy, and A. L. Yuille. Semantic image segmentation with task-specific edge detection using cnns and a dis- criminatively trained domain transform. In Proc. CVPR, 2016.
  12. L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In Proc. ICLR, 2015.
  13. L. Chen, G. Papandreou, I. Kokkinos, K. Mur- phy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. CoRR, abs/1606.00915, 2016. · arXiv
  14. L. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille. Attention to scale: Scale-aware semantic image seg- mentation. In Proc. CVPR, 2015.
  15. T. Chen, B. Xu, C. Zhang, and C. Guestrin. Train- ing deep nets with sublinear memory cost. CoRR, abs/1604.06174, 2016. · arXiv
  16. X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proc. CVPR, 2014.
  17. M. Cimpoi, S. Maji, I. Kokkinos, and A. Vedaldi. Deep filter banks for texture recognition, description, and segmentation. IJCV, 2016.
  18. J. Dai, K. He, and J. Sun. Boxsup: Exploiting bound- ing boxes to supervise convolutional networks for se- mantic segmentation. In Proc. ICCV, 2015.
  19. J. Dai, Y. Li, K. He, and J. Sun. R-FCN: object detec- tion via region-based fully convolutional networks. In Proc. NIPS, 2016.
  20. D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multi- scale convolutional architecture. In Proc. ICCV, 2015.
  21. Y. Ganin and V. Lempitsky. Nˆ4-fields: Neural net- work nearest neighbor fields for image transforms. In Proc. ACCV, 2014.
  22. G. Ghiasi and C. C. Fowlkes. Laplacian reconstruc- tion and refinement for semantic segmentation. In Proc. ECCV, 2016.
  23. S. Gidaris and N. Komodakis. Attend refine repeat: Active box proposal generation via in-out localiza- tion. CoRR, abs/1606.04446, 2016. · arXiv
  24. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
  25. R. B. Girshick. Fast R-CNN. In Proc. ICCV, 2015.
  26. G. Gkioxari, R. B. Girshick, and J. Malik. Contextual action recognition with r*cnn. In Proc. ICCV, 2015.
  27. A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. · arXiv
  28. X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg. Matchnet: Unifying feature and metric learn- ing for patch-based matching. In Proc. CVPR, 2015.
  29. B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In Proc. ICCV, 2011.
  30. B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In Proc. ECCV, 2014.
  31. A. W. Harley, K. G. Derpanis, and I. Kokkinos. Learning dense convolutional embeddings for se- mantic segmentation. CoRR, abs/1511.04377, 2015. · arXiv
  32. J.-J. Hwang and T.-L. Liu. Pixel-wise deep learning for contour detection. In Proc. ICLR, 2015.
  33. E. Insafutdinov, L. Pishchulin, B. Andres, M. An- driluka, and B. Schiele. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. CoRR, abs/1605.03170, 2016. · arXiv
  34. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. B. Girshick, S. Guadarrama, and T. Dar- rell. Caffe: Convolutional architecture for fast fea- ture embedding. In Proceedings of the ACM, 2014.
  35. P. Jiang, H. Ling, J. Yu, and J. Peng. Salient region detection by UFO: uniqueness, focusness and object- ness. In Proc. ICCV, 2013.
  36. A. Kanazawa, A. Sharma, and D. W. Jacobs. Lo- cally scale-invariant convolutional neural networks. CoRR, abs/1412.5104, 2014. · arXiv
  37. J. D. Keeler, D. E. Rumelhart, and W. K. Leow. Inte- grated segmentation and recognition of hand-printed numerals. In Proc. NIPS, 1990.
  38. J. J. Kivinen, C. K. I. Williams, and N. Heess. Vi- sual boundary prediction: A deep neural prediction network and quality dissection. In AISTATS, 2014.
  39. I. Kokkinos. Pushing the boundaries of boundary de- tection using deep learning. ICLR, 2016.
  40. I. Kokkinos and P. Maragos. An expectation max- imization approach to the synergy between image segmentation and object categorization. In Proc. ICCV, volume I, pages 617-624, 2005.
  41. P. Krähenbühl and V. Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NIPS, 2011.
  42. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Im- agenet classification with deep convolutional neural networks. In NIPS, 2013.
  43. M. P. Kumar, P. Torr, and A. Zisserman. Obj-cut. Proc. CVPR, 2005.
  44. L. Ladicky, B. Zeisl, and M. Pollefeys. Discrimi- natively trained dense surface normal estimation. In Proc. ECCV, 2014.
  45. G. Larsson, M. Maire, and G. Shakhnarovich. Learning representations for automatic colorization. CoRR, abs/1603.06668, 2016. · arXiv
  46. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. In Proc. IEEE, 1998.
  47. G. Li and Y. Yu. Visual saliency based on multiscale deep features. In Proc. CVPR, 2015.
  48. G. Li and Y. Yu. Deep contrast learning for salient object detection. In Proc. CVPR, 2016.
  49. Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille. The secrets of salient object segmentation. In Proc. CVPR, 2014.
  50. X. Liang, X. Shen, J. Feng, L. Lin, and S. Yan. Se- mantic object parsing with graph LSTM. In Proc. CVPR, 2016.
  51. T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In Proc. ECCV, 2014.
  52. F. Liu, C. Shen, and G. Lin. Deep convolutional neu- ral fields for depth estimation from a single image. In Proc. CVPR, 2015.
  53. F. Liu, C. Shen, G. Lin, and I. D. Reid. Learn- ing depth from single monocular images using deep convolutional neural fields. CoRR, abs/1502.07411, 2015. · arXiv
  54. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. E. Reed. SSD: single shot multibox detector. CoRR, abs/1512.02325, 2015. · arXiv
  55. W. Liu, A. Rabinovich, and A. C. Berg. Parsenet: Looking wider to see better. CoRR, abs/1506.04579, 2015. · arXiv
  56. J. Long, E. Shelhamer, and T. Darrell. Fully convolu- tional networks for semantic segmentation. In Proc. CVPR, 2015.
  57. M. Maire, S. X. Yu, and P. Perona. Object detection and segmentation from joint embedding of parts and pixels. In ICCV, 2011.
  58. D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. ICCV, 2001.
  59. I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In Proc. CVPR, 2016.
  60. R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of con- text for object detection and semantic segmentation in the wild. In Proc. CVPR, 2014.
  61. D. Mumford. Neuronal Architectures for Pattern Theoretic Problems. In Large Scale Theories of the Cortex. MIT Press, 1994.
  62. T. Narihira, M. Maire, and S. X. Yu. Direct intrinsics: Learning albedo-shading decomposition by convolu- tional regression. In Proc. ICCV, 2015.
  63. H. Noh, S. Hong, and B. Han. Learning deconvo- lution network for semantic segmentation. In Proc. ICCV, 2015.
  64. M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Is object localization for free? -weakly-supervised learning with convolutional neural networks. In Proc. CVPR, 2015.
  65. G. Papandreou, L. Chen, K. Murphy, and A. L. Yuille. Weakly-and semi-supervised learning of a DCNN for semantic image segmentation. In Proc. ICCV, 2015.
  66. G. Papandreou, I. Kokkinos, and P. Savalle. Mod- eling local and global deformations in deep learn- ing: Epitomic convolution, multiple instance learn- ing, and sliding window detection. In Proc. CVPR, 2015.
  67. F. Perazzi, P. Krähenbühl, Y. Pritch, and A. Hornung. Saliency filters: Contrast based filtering for salient region detection. In Proc. CVPR, 2012.
  68. T. Pfister, J. Charles, and A. Zisserman. Flowing con- vnets for human pose estimation in videos. In Proc. ICCV, 2015.
  69. P. H. O. Pinheiro, R. Collobert, and P. Dollár. Learn- ing to segment object candidates. In Proc. NIPS, 2015.
  70. Y. Qin, H. Lu, Y. Xu, and H. Wang. Saliency detec- tion via cellular automata. In Proc. CVPR, 2015.
  71. R. Ranjan, V. M. Patel, and R. Chellappa. Hyperface: A deep multi-task learning framework for face de- tection, landmark localization, pose estimation, and gender recognition. CoRR, abs/1603.01249, 2016. · arXiv
  72. S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R- CNN: towards real-time object detection with region proposal networks. In Proc. NIPS, 2015.
  73. X. Ren and L. Bo. Discriminatively trained sparse code gradients for contour detection. In Proc. NIPS, 2012.
  74. O. Ronneberger, P. Fischer, and T. Brox. U-net: Con- volutional networks for biomedical image segmenta- tion. In Proc. MICCAI, 2015.
  75. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recog- nition, localization and detection using convolutional networks. In Proc. ICLR, 2014.
  76. W. Shen, X. Wang, Y. Wang, X. Bai, and Z. Zhang. Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. In Proc. CVPR, 2015.
  77. W. Shen, K. Zhao, Y. Jiang, Y. Wang, Z. Zhang, and X. Bai. Object skeleton extraction in natural images by fusing scale-associated deep side outputs. In Proc. CVPR, 2016.
  78. E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer. Discriminative learn- ing of deep convolutional feature point descriptors. Proc. ICCV, 2015.
  79. K. Simonyan and A. Zisserman. Very deep convolu- tional networks for large-scale image recognition. In Proc. ICLR, 2015.
  80. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Ra- binovich. Going deeper with convolutions. In Proc. CVPR, 2015.
  81. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013. · arXiv
  82. A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In Proc. CVPR, 2015.
  83. S. Tsogkas and I. Kokkinos. Learning-based sym- metry detection in natural images. In Proc. ECCV, 2012.
  84. S. Tsogkas, I. Kokkinos, G. Papandreou, and A. Vedaldi. Deep learning for semantic part segmentation with high-level guidance. CoRR, abs/1505.02438, 2015. · arXiv
  85. Z. W. Tu, X. Chen, A. Yuille, and S. C. Zhu. Im- age Parsing: Unifying Segmentation, Detection, and Recognition. In Proc. ICCV, 2003.
  86. J. R. R. Uijlings and V. Ferrari. Situational object boundary detection. In Proc. CVPR, 2015.
  87. K. Wang, L. Lin, J. Lu, C. Li, and K. Shi. PISA: pixelwise image saliency by aggregating comple- mentary appearance contrast measures with edge- preserving coherence. IEEE Trans. Im. Proc., 2015.
  88. L. Wang, H. Lu, X. Ruan, and M. Yang. Deep net- works for saliency detection via local estimation and global search. In Proc. CVPR, 2015.
  89. X. Wang, D. F. Fouhey, and A. Gupta. Designing deep networks for surface normal estimation. In Proc. CVPR, 2015.
  90. S. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. In Proc. CVPR, 2016.
  91. A. P. Witkin. Scale-space filtering. In IJCAI, 1983.
  92. F. Xia, P. Wang, L. Chen, and A. L. Yuille. Zoom better to see clearer: Human part segmentation with auto zoom net. In Proc. ECCV, 2016.
  93. S. Xie and Z. Tu. Holistically-nested edge detection. In Proc. ICCV, 2015.
  94. T. F. Yago Vicente, M. Hoai, and D. Samaras. Noisy label recovery for shadow detection in unfamiliar do- mains. In Proc. CVPR, 2016.
  95. K. M. Yi, E. Trulls, V. Lepetit, and P. Fua. Lift: Learned invariant feature transform. In Proc. ECCV, 2016.
  96. S. Zagoruyko and N. Komodakis. Learning to com- pare image patches via convolutional neural net- works. In Proc. CVPR, 2015.
  97. Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Facial landmark detection by deep multi-task learning. In Proc. ECCV, 2014.
  98. R. Zhao, W. Ouyang, H. Li, and X. Wang. Saliency detection by multi-context deep learning. In Proc. CVPR, 2015.

Cited by

Sorted by citations

Authors
Kokkinos, I.
Affiliations
University College London (GB)
Corresponding author
Kokkinos, Iasonas
ORCID
Kokkinos, Iasonas
Venue
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2017-07
Pages
5454-5463
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2017.579
OpenAlex
W2510182739
Cited by
657
References
16
Citations by year
2026: 10 · 2025: 51 · 2024: 53 · 2023: 80 · 2022: 83 · 2021: 107 · 2020: 99 · 2019: 91 · 2018: 57 · 2017: 24 · 2016: 2
FWCI
21.67
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • Horizon 2020 Framework Programme
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
multi-task learning, convolutional neural networks, low-level vision, mid-level vision, high-level vision, boundary detection, surface normal estimation, saliency detection, semantic segmentation, region proposal generation, object detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review