The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
State representation learning for control: An overview
Open accessAbstract
Representation learning algorithms are designed to learn abstract features that characterize data. State representation learning (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent. The representation is learned to capture the variation in the environment generated by the agent's actions; this kind of representation is particularly suitable for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning. This survey aims at covering the state-of-the-art on state representation learning in the most recent years. It reviews different SRL methods that involve interaction with the environment, their implementations and their applications in robotics control tasks (simulated or real). In particular, it highlights how generic learning objectives are differently exploited in the reviewed algorithms. Finally, it discusses evaluation methods to assess the representation learned and summarizes current and future lines of research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036
(2017) · International Conference on Learning Representations · Cited by 3,248
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 2,405
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 1×
“• Atari games [Bellemare et al., 2013]: mostly low (2D) dimensional simulated environments with different agents and goals.”
BackgroundEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131
Conditional Image Synthesis With Auxiliary Classifier GANs
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 1,837
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815 · cited 10×
“An example using inverse models to learn state representations is the Intrinsic Curiosity Module (ICM) [Pathak et al., 2017].”
BackgroundMethodInfluentialDeep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Slow Feature Analysis: Unsupervised Learning of Invariances
Wiskott, L., & Sejnowski, T. J. (2002) · Neural Computation · doi:10.1162/089976602317318938 · Cited by 1,350
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03657 · Cited by 1,156
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
A Survey of Dimension Reduction Techniques
Fodor, I. K. (2002) · doi:10.2172/15002155 · Cited by 1,116
Auto-encoder based dimensionality reduction
Wang, Y., Yao, H., & Zhao, S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.08.104 · Cited by 976
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460875 · Cited by 636
Donahue, J., Krähenbühl, P., & Darrell, T. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09782 · Cited by 616
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · cited 1×
“Many of the latter simulated scenarios are part of Universe and OpenAI Gym [ Brockman et al., 2016] or DeepMind Labs [Beattie et al., 2016].”
BackgroundUnsupervised Feature Learning and Deep Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · Cited by 581
Adversarially Learned Inference
Dumoulin, V., Belghazi, M. I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00704 · Cited by 579
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497
Deep spatial autoencoders for visuomotor learning
Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487173 · Cited by 430
Learning to Poke by Poking: Experiential Learning of Intuitive PhysicsPreprint
Agrawal, P., Nair, A., Abbeel, P., Malik, J., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07419 · Cited by 404
Empowerment: A Universal Agent-Centric Measure of Control
Klyubin, A. S., Polani, D., & Nehaniv, C. L. (2005) · 2005 IEEE Congress on Evolutionary Computation · doi:10.1109/cec.2005.1554676 · Cited by 274
Algorithmic applications of low-distortion geometric embeddings
Indyk, P. (2001) · Proceedings 42nd IEEE Symposium on Foundations of Computer Science · doi:10.1109/sfcs.2001.959878 · Cited by 253
Interactive Perception: Leveraging Action in Perception and Perception in Action
Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragić, D., Schaal, S., & Sukhatme, G. (2017) · MPG.PuRe (Max Planck Society) · Cited by 242
Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
Karl, M., Soelch, M., Bayer, J., & van der Smagt, P. (2016) · arXiv · doi:10.48550/arxiv.1605.06432 · Cited by 232
Interactive perception: Leveraging action in perception and perception in action
Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., & Sukhatme, G. S. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2721939 · Cited by 227 · cited 1×
“This is for instance the case of representation learning from different temporal signals in [Duan, 2017, Bohg et al., 2017].”
BackgroundDarla Improving zero-shot transfer in reinforcement learning
Higgins, I., Pal, A., Rusu, A. A., Matthey, L., Burgess, C. P., Pritzel, A., Botvinick, M., Blundell, C., & Lerchner, A. (2017) · arXiv · doi:10.48550/arxiv.1707.08475 · Cited by 219
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
Watter, M., Springenberg, J. T., Boedecker, J., & Riedmiller, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.07365 · Cited by 212
Krishnan, R. G., Shalit, U., & Sontag, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05121 · Cited by 206 · cited 3×
“This method is also used in [Karl et al., 2016] and [Krishnan et al., 2015].”
MethodLoss is its own Reward: Self-Supervision for Reinforcement LearningPreprint
Shelhamer, E., Mahmoudieh, P., Argus, M. J., & Darrell, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07307 · Cited by 205
The Mario AI Benchmark and Competitions
Karakovskiy, S., & Togelius, J. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2188528 · Cited by 186
Stable reinforcement learning with autoencoders for tactile and visual data
van Hoof, H., Chen, N., Karl, M., van der Smagt, P., & Peters, J. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759578 · Cited by 184
From Pixels to Torques: Policy Learning with Deep Dynamical ModelsPreprint
Wahlström, N., Schön, T. B., & Deisenroth, M. P. (2015) · arXiv (Cornell University) · doi:10.25561/21120 · Cited by 177
Robot Skill Learning: From Reinforcement Learning to Evolution Strategies
Stulp, F., & Sigaud, O. (2013) · Paladyn Journal of Behavioral Robotics · doi:10.2478/pjbr-2013-0003 · Cited by 171
The Curious Robot: Learning Visual Representations via Physical Interactions
Pinto, L., Gandhi, D., Han, Y., Park, Y., & Gupta, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_1 · Cited by 162
Learning state representations with robotic priors
Jonschkowski, R., & Brock, O. (2015) · Autonomous Robots · doi:10.1007/s10514-015-9459-7 · Cited by 158
Deep Multimodal Representation Learning from Temporal Data
Yang, X., Ramesh, P. S., Chitta, R., Madhvanath, S., Bernal, E. A., & Luo, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.538 · Cited by 126
Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation
Sermanet, P., Lynch, C., Hsu, J., & Levine, S. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.69 · Cited by 112
Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal ExplorationPreprint
Péré, A., Forestier, S., Sigaud, O., & Oudeyer, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00781 · Cited by 100
Böhmer, W., Springenberg, J. T., Boedecker, J., Riedmiller, M., & Obermayer, K. (2015) · Künstliche Intell. · doi:10.1007/s13218-015-0356-1 · Cited by 83
Independently Controllable FactorsPreprint
Thomas, V., Pondard, J., Bengio, E., Sarfati, M., Beaudoin, P., Meurs, M., Pineau, J., Precup, D., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01289 · Cited by 77
Decoupling Dynamics and Reward for Transfer LearningPreprint
Zhang, A., Satija, H., & Pineau, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.10689 · Cited by 72
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., et al. (2016) · arXiv · doi:10.48550/arxiv.1612.03801 · Cited by 68 · cited 1×
“Many of the latter simulated scenarios are part of Universe and OpenAI Gym [ Brockman et al., 2016] or DeepMind Labs [Beattie et al., 2016].”
BackgroundA Folded Neural Network Autoencoder for Dimensionality Reduction
Wang, J., He, H., & Prokhorov, D. V. (2012) · Procedia Computer Science · doi:10.1016/j.procs.2012.09.120 · Cited by 68
PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State RepresentationsPreprint
Jonschkowski, R., Hafner, R., Scholz, J., & Riedmiller, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09805 · Cited by 62
Autoencoder-augmented neuroevolution for visual doom playing
Alvernaz, S., & Togelius, J. (2017) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2017.8080408 · Cited by 62
Learning to Control an Octopus Arm with Gaussian Process Temporal Difference Methods
Engel, Y., Szabo, P., & Volkinshtein, D. (2005) · Cited by 60
Learning state representation for deep actor-critic control
Munk, J., Kober, J., & Babuška, R. (2016) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2016.7798980 · Cited by 58
Learn to Swing Up and Balance a Real Pole Based on Raw Visual Input Data
Mattner, J., Lange, S., & Riedmiller, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34500-5_16 · Cited by 43
Unsupervised state representation learning with robotic priors: a robustness benchmarkPreprint
Lesort, T., Seurin, M., Li, X., Díaz-Rodríguez, N., & Filliat, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.05185 · Cited by 26
Self-citationData-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical ModelsPreprint
Assael, J.-A. M., Wahlström, N., Schön, T. B., & Deisenroth, M. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.02173 · Cited by 25
Curiosity-Driven Reinforcement Learning with Homeostatic Regulation
Magrans, I., & Kanai, R. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489075 · Cited by 24
A Separation Principle for Control in the Age of Deep Learning
Achille, A., & Soatto, S. (2018) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-060117-105140 · Cited by 19
Kompella, V. R., Luciw, M., & Schmidhuber, J. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1112.2113 · Cited by 7
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Self-citationBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Self-citationDeep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Self-citationOverview and comparative study of dimensionality reduction techniques for high dimensional data
Ayesha, S., Hanif, M. K., & Talib, R. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.005 · Cited by 569
Metadata
- Authors
- Lesort, T., Díaz-Rodríguez, N., Goudou, J.-F., & Filliat, D.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2018-12
- Volume
- 108
- Pages
- 379-392
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2787666871
- Cited by
- 301
- References
- 74
- FWCI
- 19.52
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.