The Curious Robot: Learning Visual Representations via Physical Interactions
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
What is the right supervisory signal to train visual representations? Current approaches in computer vision use category labels from datasets such as ImageNet to train ConvNets. However, in case of biological agents, visual representation learning does not require millions of semantic labels. We argue that biological agents use physical interactions with the world to learn visual representations unlike current vision systems which just use passive observations (images and videos downloaded from web). For example, babies push objects, poke them, put them in their mouth and throw them to learn representations. Towards this goal, we build one of the first systems on a Baxter platform that pushes, pokes, grasps and observes objects in a tabletop environment. It uses four different types of physical interactions to collect more than 130K datapoints, with each datapoint providing supervision to a shared ConvNet architecture allowing us to learn visual representations. We show the quality of learned representations by observing neuron activations and performing nearest neighbor retrieval on this learned representation. Quantitatively, we evaluate our learned ConvNet on image classification tasks and show improvements compared to learning without external data. Finally, on the task of instance retrieval, our network outperforms the ImageNet network on recall@1 by 3%
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629
Self-citationLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
Movement-produced stimulation in the development of visually guided behavior.
Held, R., & Hein, A. (1963) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0040546 · Cited by 1,161 · cited 1×
“In fact, this has been supported by several psychological behavior experiments as well [16].”
BackgroundUnsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005
Self-citationAloimonos, J., Weiss, I., & Bandyopadhyay, A. (1988) · International Journal of Computer Vision · doi:10.1007/bf00133571 · Cited by 931 · cited 1×
“This is similar to the idea of active vision [33,34].”
BackgroundData-Driven Grasp Synthesis—A Survey
Bohg, J., Morales, A., Asfour, T., & Kragic, D. (2014) · IEEE Transactions on Robotics · doi:10.1109/tro.2013.2289018 · Cited by 929
Stable Pushing: Mechanics, Controllability, and Planning
Lynch, K. M., & Mason, M. T. (1996) · The International Journal of Robotics Research · doi:10.1177/027836499601500602 · Cited by 463 · cited 1×
“The origins of pushing as a manipulation task can be traced to the task of aligning objects to reduce pose uncertainty [26,27,28].”
Background
Cited by
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
Self-citationThe “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471
Temporal Relational Reasoning in Videos
Zhou, B., Andonian, A., Oliva, A., & Torralba, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_49 · Cited by 1,067
Electronic skins and machine learning for intelligent soft robots
Shih, B., Shah, D., Li, J., Thuruthel, T. G., Park, Y.-L., Iida, F., Bao, Z., Kramer-Bottiglio, R., & Tolley, M. T. (2020) · Science Robotics · doi:10.1126/scirobotics.aaz9239 · Cited by 715
“This property makes soft robots appealing in contexts such as HRI and robotic manipulation, where safety around fragile objects can be important (11, 80).”
BackgroundSelf-citationMulti-task Self-Supervised Visual Learning
Doersch, C., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.226 · Cited by 598
Metadata
- Authors
- Pinto, L., Gandhi, D., Han, Y., Park, Y.-L., & Gupta, A.
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2016
- Pages
- 3-18
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2338684808
- Cited by
- 156
- References
- 11
- FWCI
- 16.88
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Curious Robot: Learning Visual Representations via Physical Interactions. https://app.cochat.ai/papers/10.1007/978-3-319-46475-6_1
- The paper
- Pinto, L., Gandhi, D., Han, Y., Park, Y.-L., & Gupta, A. (2016). The Curious Robot: Learning Visual Representations via Physical Interactions. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-319-46475-6_1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.