Learning Invariant Representations for Reinforcement Learning without Reconstruction
Open accessAbstract
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction. Our goal is to learn representations that both provide for effective downstream control and invariance to task-irrelevant details. Bisimulation metrics quantify behavioral similarity between states in continuous MDPs, which we propose using to learn robust latent representations which encode only the task-relevant information from observations. Our method trains encoders such that distances in latent space equal bisimulation distances in state space. We demonstrate the effectiveness of our method at disregarding task-irrelevant information using modified visual MuJoCo tasks, where the background is replaced with moving distractors and natural videos, while achieving SOTA performance. We also test a first-person highway driving task where our method learns invariance to clouds, weather, and time of day. Finally, we provide generalization results drawn from properties of bisimulation metrics, and links to causal inference.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Experiments suggest that Random Overlay is the best data augmentation technique to improve data efficiency of pixel-based deep reinforcement learning.
“Experiments suggest that Random Overlay [137] (which interpolates the observation with an unrelated image and achieves Bisimulation in practice) is the best DA technique to improve data efficiency of pixel-based DRL.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [137]
The Experiments section compares a bisimulation-based representation method with SLAC, DeepMDP, reconstruction, and contrastive baselines; it does not evaluate Random Overlay. Searches for Random Overlay, interpolation with an unrelated image, and comparisons of data-augmentation techniques found no passage supporting the claim that Random Overlay is best for data efficiency. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIn order to perfectly clone a behavior that depends on the state, the learner must implicitly learn a task-relevant dynamics model.
“This shows that in order to perfectly clone a behavior that depends on the state, the learner must implicitly learn a (task-relevant) dynamics model Subramanian and Mahajan (2019); Zhang et al. (2020).”
From Diffusion policy: Visuomotor policy learning via action diffusion (2025) · cites this paper as Zhang et al. (2020)
The Introduction describes a representation that predicts task-relevant future states, and the method trains a latent dynamics model for reinforcement learning. Neither passage says that perfectly cloning state-dependent behavior requires a learner to implicitly learn such a model. Searches for imitation learning, behavior cloning, and that necessity claim found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredTo perfectly clone a behavior that depends on the state, the learner must implicitly learn a task-relevant dynamics model.
“This shows that in order to perfectly clone a behavior that depends on the state, the learner must implicitly learn a (task-relevant) dynamics model Subramanian and Mahajan (2019); Zhang et al. (2020).”
From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as Zhang et al. (2020)
The Introduction describes a task-relevant representation that predicts reward and future representations, and the Discussion says the authors train a latent dynamics model. Neither section addresses behavior cloning or establishes that perfectly cloning state-dependent behavior requires implicitly learning a dynamics model. Searches for behavior cloning and for that claimed necessity found no supporting passage. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [0]0. If only future reward changes, then those future states will have nonzero bisimilarity, and maxa∈AW (d̃)(P a si , P a sj ) > 0, giving us d̃(si, sj) > 0. B Definition of State Since we are concerned primarily with learning from image observations, we could explicitly distinguish the image observation space O from an unknown state space S. However, since we are not tackling the general POMDP problem, we consider the Block MDP (Du et al., 2019), which assumes the state space is latent, and that we are instead given access to an observation space O and rendering function q : S 7→ O. The crucial assumption that distinguishes the Block MDP from partially observable MDPs is the following: Assumption 1 (Block structure (Du et al., 2019)). Each observation o uniquely determines its generating state s. That is, the observation space O can be partitioned into disjoint blocks Os, each containing the support of the conditional distribution q(o|s). This assumption gives us the Markov property in the observation space o ∈ O. As an example, one can think of the proprioceptive state consisting of positions and velocities of actuators as the underlying state, and stacked pixel observations from a specific camera angle as a particular rendering function and corresponding observation space. C Additional DMC Results In Figure 11 we show performance on the default setting on 9 different environments from DMC. Figures 12 and 13 give performance on the simple distractors and natural video settings for all 9 environments. 13 Published as a conference paper at ICLR 2021
Cited by
Sorted by citations
Diffusion policy: Visuomotor policy learning via action diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241273668 · Cited by 613 · see citing works
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
Self-citationA Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182 · see citing works
The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
Stone, A., Ramirez, O., Konolige, K., & Jonschkowski, R. (2021) · arXiv · doi:10.48550/arxiv.2101.02722 · Cited by 31 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“It has been argued that a good representation should be able to (1) learn task-relevant information [137], (2) preserve most information to predict the future [138], and/or (3) capture the posterior distribution of the underlying explanatory factors for the observation [139].”
BackgroundMethodInfluentialDiffusion Policy: Visuomotor Policy Learning via Action Diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2303.04137 · Cited by 12 · see citing works
“But interestingly, it need only learn the task-relevant dynamics (Subramanian and Mahajan, 2019; Zhang et al., 2020); in the linear dynamics example here, if K has a column that is all zeros, then leftmost multiplication by K means that the associated state dynamics are not relevant to the task and…”
Measuring Visual Generalization in Continuous Control from Pixels
Grigsby, J., & Qi, Y. (2020) · arXiv · doi:10.48550/arxiv.2010.06740 · Cited by 10 · see citing works
- Authors
- Zhang, A., McAllister, R., Calandra, R., Gal, Y., & Levine, S.
- Affiliations
- Meta (United Kingdom) (GB)
- University of California, Berkeley (US)
- Meta (Israel) (IL)
- University of Oxford (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.10742
- OpenAlex
- W3036185205
- Cited by
- 165
- References
- 1
- Citations by year
- 2025: 13 · 2024: 32 · 2023: 32 · 2022: 25 · 2021: 51 · 2020: 12
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- invariant representations, representation learning, reinforcement learning, bisimulation metrics, latent representations, visual servoing, causal inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
