CoChat
Sign Up
Cited by 1,128Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsAdvanced Vision and Imaging

Taskonomy: Disentangling Task Transfer Learning

Open access
Zamir, Amir R., Sax, Alexander, Shen, William, Guibas, Leonidas, Malik, Jitendra, Savarese, Silvio·2018·2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
DOI 10.1109/cvpr.2018.00391
Open this paper in CoChat — free accountOpen PDF

Abstract

Do visual tasks have relationships, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a certain structure among visual tasks. Knowing this structure has notable values; it provides a principled way for identifying relationships across tasks, for instance, in order to reuse supervision among tasks with redundancies or solve many tasks in one system without piling up the complexity. We propose a fully computational approach for modeling the transfer learning structure of the space of visual tasks. This is done via finding transfer learning dependencies across tasks in a dictionary of twenty-six 2D, 2.5D, 3D, and semantic tasks. The product is a computational taxonomic map among tasks for transfer learning, and we exploit it to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and visualizing this taxonomical structure at http://taskonomy.vision.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2018.00391

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The pretraining approach has been successful for various machine vision problems.

    “The pretraining approach has been successful for various machine vision problems28–30.”

    From Cellpose 2.0: how to train your own model (2022) · cites this paper as [28]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 82

  1. P. Agrawal, J. Carreira, and J. Malik. Learning to see by moving. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 37-45, 2015. 8
  2. M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pages 3981-3989, 2016. 2
  3. I. Armeni, S. Sax, A. R. Zamir, and S. Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017. 4 · arXiv
  4. S. Arora, A. Bhaskara, R. Ge, and T. Ma. Provable bounds for learning some deep representations. In International Conference on Machine Learning, pages 584-592, 2014. 2
  5. Y. Aytar and A. Zisserman. Tabula rasa: Model transfer for object category detection. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2252-2259. IEEE, 2011. 2
  6. J. Baxter. A bayesian/information theoretic model of learn- ing to learn viamultiple task sampling. Mach. Learn., 28(1):7-39, July 1997. 3
  7. S. Ben-David and R. S. Borbely. A notion of task relat- edness yielding provable multiple-task learning guarantees. Machine Learning, 73(3):273-287, Dec 2008.
  8. Y. Bengio, A. Courville, and P. Vincent. Representa- tion learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013. 2
  9. E. Bienenstock, S. Geman, and D. Potter. Compositionality, mdl priors, and object recognition. In Advances in neural information processing systems, pages 838-844, 1997. 2
  10. H. Bilen and A. Vedaldi. Integrated perception with re- current multi-task neural networks. In Advances in neural information processing systems, pages 235-243, 2016. 2
  11. J. Bingel and A. Søgaard. Identifying beneficial task rela- tions for multi-task learning in deep neural networks. arXiv preprint arXiv:1702.08303, 2017. 2 · arXiv
  12. O. Boiman and M. Irani. Similarity by composition. In Advances in neural information processing systems, pages 177-184, 2007. 2
  13. I. I. CPLEX. V12. 1: Users manual for cplex. International Business Machines Corporation, 46(53):157, 2009. 5
  14. C. Doersch, A. Gupta, and A. A. Efros. Unsupervised vi- sual representation learning by context prediction. In Pro- ceedings of the IEEE International Conference on Com- puter Vision, pages 1422-1430, 2015. 2
  15. C. Doersch and A. Zisserman. Multi-task self-supervised visual learning. arXiv preprint arXiv:1708.07860, 2017. 2 · arXiv
  16. J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial fea- ture learning. arXiv preprint arXiv:1605.09782, 2016. 2 · arXiv
  17. Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. Rl2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016. 2 · arXiv
  18. D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vin- cent, and S. Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Re- search, 11(Feb):625-660, 2010. 2
  19. A. Faktor and M. Irani. clustering by composition- unsupervised discovery of image categories. In European Conference on Computer Vision, pages 474-487. Springer, 2012. 2
  20. L. Fe-Fei et al. A bayesian approach to unsupervised one- shot learning of object categories. In Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, pages 1134-1141. IEEE, 2003. 2
  21. L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611, 2006. 2
  22. B. Fernando, A. Habrard, M. Sebban, and T. Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE international con- ference on computer vision, pages 2960-2967, 2013. 2
  23. C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta- learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400, 2017. 2 · arXiv
  24. C. Finn, S. Levine, and P. Abbeel. Guided cost learn- ing: Deep inverse optimal control via policy optimization. CoRR, abs/1603.00448, 2016. 2 · arXiv
  25. C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learn- ing. In Robotics and Automation (ICRA), 2016 IEEE Inter- national Conference on, pages 512-519. IEEE, 2016. 2
  26. C. Finn, T. Yu, J. Fu, P. Abbeel, and S. Levine. Generalizing skills with semi-supervised reinforcement learning. CoRR, abs/1612.00429, 2016. 2 · arXiv
  27. C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One- shot visual imitation learning via meta-learning. CoRR, abs/1709.04905, 2017. 2 · arXiv
  28. R. M. French. Catastrophic forgetting in connectionist net- works: Causes, consequences and solutions. Trends in Cog- nitive Sciences, 3(4):128-135, 1999. 2
  29. R. Ge. Provable algorithms for machine learning problems. PhD thesis, Princeton University, 2013. 2
  30. S. Geman, D. F. Potter, and Z. Chi. Composition systems. Quarterly of Applied Mathematics, 60(4):707-736, 2002. 2
  31. R. Gopalan, R. Li, and R. Chellappa. Domain adaptation for object recognition: An unsupervised approach. In Com- puter Vision (ICCV), 2011 IEEE International Conference on, pages 999-1006. IEEE, 2011. 2
  32. A. Gopnik, C. Glymour, D. M. Sobel, L. E. Schulz, T. Kushnir, and D. Danks. A theory of causal learning in children: causal maps and bayes nets. Psychological re- view, 111(1):3, 2004. 2
  33. A. Graves, G. Wayne, and I. Danihelka. Neural turing ma- chines. CoRR, abs/1410.5401, 2014. 2 · arXiv
  34. K. Henry. The theory and applications of homomorphic cryptography. Master's thesis, University of Waterloo, 2008. 2
  35. G. Hinton, O. Vinyals, and J. Dean. Distilling the knowl- edge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 4 · arXiv
  36. J. Hoffman, T. Darrell, and K. Saenko. Continuous mani- fold based adaptation for evolving visual domains. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 867-874, 2014. 2
  37. Y. Hoshen and S. Peleg. Visual learning of arithmetic oper- ations. CoRR, abs/1506.02264, 2015. 2 · arXiv
  38. F. Hu, G.-S. Xia, J. Hu, and L. Zhang. Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Remote Sensing, 7(11):14680-14707, 2015. 2
  39. I.-H. Jhuo, D. Liu, D. Lee, and S.-F. Chang. Robust visual domain adaptation with low-rank reconstruction. In Com- puter Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2168-2175. IEEE, 2012. 2
  40. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 2 · arXiv
  41. I. Kokkinos. Ubernet: Training auniversal'convolutional neural network for low-, mid-, and high-level vision us- ing diverse datasets and limited memory. arXiv preprint arXiv:1609.02132, 2016. 2 · arXiv
  42. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097-1105, 2012. 8
  43. B. Kulis, K. Saenko, and T. Darrell. What you saw is not what you get: Domain adaptation using asymmetric ker- nel transforms. In Computer Vision and Pattern Recogni- tion (CVPR), 2011 IEEE Conference on, pages 1785-1792. IEEE, 2011. 2
  44. I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab. Deeper depth prediction with fully convolutional residual networks. In 3D Vision (3DV), 2016 Fourth Inter- national Conference on, pages 239-248. IEEE, 2016. 7
  45. B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Ger- shman. Building machines that learn and think like people. Behavioral and Brain Sciences, pages 1-101, 2016. 2
  46. Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully con- volutional instance-aware semantic segmentation. arXiv preprint arXiv:1611.07709, 2016. 4 · arXiv
  47. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Dollár, and C. L. Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014. 4
  48. F. Liu, G. Lin, and C. Shen. CRF learning with CNN features for image segmentation. CoRR, abs/1503.08263, 2015. 2 · arXiv
  49. Z. Luo, Y. Zou, J. Hoffman, and L. F. Fei-Fei. Label ef- ficient learning of transferable representations acrosss do- mains and tasks. In Advances in Neural Information Pro- cessing Systems, pages 164-176, 2017. 2
  50. M. M. H. Mahmud. On Universal Transfer Learning, pages 135-149. Springer Berlin Heidelberg, Berlin, Heidelberg, 2007. 3
  51. L. Mihalkova, T. Huynh, and R. J. Mooney. Mapping and revising markov logic networks for transfer learning. In AAAI, volume 7, pages 608-614, 2007. 2
  52. T. Mikolov, Q. V. Le, and I. Sutskever. Exploiting simi- larities among languages for machine translation. CoRR, abs/1309.4168, 2013. 2 · arXiv
  53. I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross- stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994-4003, 2016. 2
  54. A. Niculescu-Mizil and R. Caruana. Inductive transfer for bayesian network structure learning. In Artificial Intelli- gence and Statistics, pages 339-346, 2007. 2
  55. M. Noroozi and P. Favaro. Unsupervised learning of vi- sual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84. Springer, 2016. 2, 8
  56. M. Noroozi, H. Pirsiavash, and P. Favaro. Represen- tation learning by learning to count. arXiv preprint arXiv:1708.06734, 2017. 2 · arXiv
  57. M. Norouzi, T. Mikolov, S. Bengio, Y. Singer, J. Shlens, A. Frome, G. S. Corrado, and J. Dean. Zero-shot learn- ing by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650, 2013. 2 · arXiv
  58. M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, and L. Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (TOG), 31(4):30, 2012. 2
  59. D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544, 2016. 2
  60. J. Piaget and M. Cook. The origins of intelligence in chil- dren, volume 8. International Universities Press New York, 1952. 2
  61. L. Y. Pratt. Discriminability-based transfer between neural networks. In Advances in neural information processing systems, pages 204-211, 1993. 2
  62. S. R. Richter, Z. Hayder, and V. Koltun. Playing for bench- marks. In International Conference on Computer Vision (ICCV), 2017. 2
  63. S. T. Roweis and L. K. Saul. Nonlinear dimension- ality reduction by locally linear embedding. science, 290(5500):2323-2326, 2000. 2
  64. R. W. Saaty. The analytic hierarchy process -what it is and how it is used. Mathematical Modeling, 9(3-5):161-176, 1987. 5
  65. R. Salakhutdinov, J. Tenenbaum, and A. Torralba. One-shot learning with a hierarchical nonparametric bayesian model. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, pages 195-206, 2012. 2
  66. J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015. 2 · arXiv
  67. D. L. Silver and K. P. Bennett. Guest editors introduction: special issue on inductive transfer learning. Machine Learn- ing, 73(3):215-220, 2008. 2
  68. D. L. Silver, Q. Yang, and L. Li. Lifelong machine learning systems: Beyond learning algorithms. In in AAAI Spring Symposium Series, 2013. 2
  69. R. Socher, M. Ganjoo, C. D. Manning, and A. Ng. Zero- shot learning through cross-modal transfer. In NIPS, pages 935-943, 2013. 2
  70. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014. 2
  71. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013. 2 · arXiv
  72. J. B. Tenenbaum and T. L. Griffiths. Generalization, sim- ilarity, and bayesian inference. Behavioral and Brain Sci- ences, 24(4):629640, 2001. 2
  73. C. Tessler, S. Givony, T. Zahavy, D. J. Mankowitz, and S. Mannor. A deep hierarchical approach to lifelong learn- ing in minecraft. In AAAI, pages 1553-1561, 2017. 2
  74. S. Thrun and L. Pratt. Learning to learn. Springer Science & Business Media, 2012. 2
  75. X. Wang, K. He, and A. Gupta. Transitive invariance for self-supervised visual representation learning. arXiv preprint arXiv:1708.02901, 2017. 2 · arXiv
  76. T. Winograd. Thinking machines: Can there be? Are we, volume 200. University of California Press, Berkeley, 1991. 2
  77. J. Yang, R. Yan, and A. G. Hauptmann. Adapting svm clas- sifiers to data with shifted distributions. In Data Mining Workshops, 2007. ICDM Workshops 2007. Seventh IEEE International Conference on, pages 69-76. IEEE, 2007. 2
  78. A. R. Zamir, T. Wekel, P. Agrawal, C. Wei, J. Malik, and S. Savarese. Generic 3d representation via pose estimation and matching. In European Conference on Computer Vi- sion, pages 535-553. Springer, 2016. 2, 8
  79. A. R. Zamir, F. Xia, J. He, A. Sax, J. Malik, and S. Savarese. Gibson Env: Real-world perception for embodied agents. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018. 4
  80. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking general- ization. CoRR, abs/1611.03530, 2016. 2 · arXiv
  81. R. Zhang, P. Isola, and A. A. Efros. Colorful image col- orization. In European Conference on Computer Vision, pages 649-666. Springer, 2016. 2, 8
  82. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Advances in neural information processing systems, pages 487-495, 2014. 2, 4, 8, 9

Cited by

Sorted by citations

Authors
Zamir, A. R., Sax, A., Shen, W., Guibas, L., Malik, J., & Savarese, S.
Affiliations
  • University of California, Berkeley (US)
  • Stanford University (US)
ORCID
Venue
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Publisher
IEEE
Published
2018-06
Pages
3712-3722
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2018.00391
OpenAlex
W2798512429
Cited by
1,128
References
12
Citations by year
2026: 36 · 2025: 94 · 2024: 136 · 2023: 164 · 2022: 137 · 2021: 211 · 2020: 165 · 2019: 158 · 2018: 27
FWCI
57.35
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (N00014-13-1-0341, 1521608)
  • Nvidia
  • Amazon Web Services
  • Tencent
  • Multidisciplinary University Research Initiative (N00014-13-1-0341)
  • Office of Naval Research (N00014-13-1-0341, N000141410671, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
visual task, task dependencies, latent space, surface normal estimation, depth estimation, data efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review