Observational Overfitting in Reinforcement Learning
Open accessAbstract
A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The examples of goal misgeneralization discussed in Section 3.1 were caused by spurious correlations between rewards and environmental features on smallscale tasks.
“The examples of goal misgeneralization discussed in Section 3.1 were caused by spurious correlations between rewards and environmental features on smallscale tasks (also known as 'observational overfitting') [Song et al., 2019].”
From The alignment problem from a deep learning perspective (2022) · cites this paper as Song et al., 2019
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 61 · Cited by 48
- Authors
- Song, X., Jiang, Y., Tu, S., Du, Y., & Neyshabur, B.
- Affiliations
- Google (United States) (US)
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1912.02975
- OpenAlex
- W2994066474
- Cited by
- 48
- References
- 61
- Citations by year
- 2025: 4 · 2024: 7 · 2023: 7 · 2022: 6 · 2021: 18 · 2020: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- model-free reinforcement learning, spurious correlations, reward sparsity, observation space, synthetic benchmark, implicit regularization, Markov decision process
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
