Generalizable Causal Reinforcement Learning for Out-of-Distribution Environments
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Out-of-distribution (OOD) generalization is critical for applying reinforcement learning algorithms to real-world applications. To address the OOD problem, recent works focus on learning an OOD adaptation policy by capturing the causal factors affecting the environmental dynamics. However, these works recover the causal factors with only an entangled or binary form, resulting in a limited generalization of the policy that requires extra data from the testing environments. To break this limitation, we propose generalizable causal reinforcement learning (GCRL) to learn a disentangled representation of causal factors, on the basis of which we learn a policy that achieves the OOD generalization without extra training. For capturing the causal factors, GCRL deploys a weakly supervised signal with a two-stage constraint to ensure that all factors can be disentangled. Then, to achieve the OOD generalization through causal factors, we establish the dependence of actions on the learned representation and optimize the policy model across multiple environments. Experimental results show that the established dependence recovers the correct relationship between causal factors and actions when the learned policy could address the target tasks in training environments. Benefiting from the recovered relationship, GCRL achieves the OOD generalization on eight benchmarks from Causal World and Mujoco. Moreover, the policy learned by our model is more explainable and can be controlled to generat…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · cited 1×
“I NTRODUCTION R EINFORCEMENT learning (RL) provides a powerful framework for training agents to take proper sequential actions based on states [1], [2], [3], [4], such as robot control [5], [6], [7], [8], [9], self-driving cars [10], [11], and human–machine interaction [12].”
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · cited 1×
“To verify the performance of the OOD generalization, we carry out benchmark tasks on Causal World [29] and Mujoco Control Suite [30].”
Reinforcement Learning for Real-Time Pricing and Scheduling Control in EV Charging Stations
Wang, S., Bi, S., & Zhang, Y. A. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2950809 · Cited by 281 · cited 2×
“However, the strict OOD generalization could not interact with testing environments because the testing distribution is commonly un-knowninreal-worldapplications[8],[17],[18].”
Stable learning establishes some common ground between causal inference and machine learning
Cui, P., & Athey, S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00445-z · Cited by 241 · cited 1×
“Along with progress in industry and academia, RL agents need to generalize factors of variation that are not in the training distribution, which is called the out-of-distribution (OOD) problem [13], [14].”
Self-citationA Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
Cui, Y., Peng, L., & Li, H. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3142323 · Cited by 81
Xing, Q., Xu, Y. H., Chen, Z., Zhang, Z., & Shi, Z. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3210264 · Cited by 67 · cited 1×
“However, the strict OOD generalization could not interact with testing environments because the testing distribution is commonly un-knowninreal-worldapplications[8],[17],[18].”
Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process
Zhu, L., Cui, Y., Takami, G., Kanokogi, H., & Matsubara, T. (2020) · Control Engineering Practice · doi:10.1016/j.conengprac.2020.104331 · Cited by 65 · cited 1×
“I NTRODUCTION R EINFORCEMENT learning (RL) provides a powerful framework for training agents to take proper sequential actions based on states [1], [2], [3], [4], such as robot control [5], [6], [7], [8], [9], self-driving cars [10], [11], and human–machine interaction [12].”
Research works printed but not linked: 27
- Fujimoto, 2024, For SALE: State-action representation learning for deep reinforcement learning, 61573
- Bhatt, 2024, CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity
- Hiraoka, 2022, Dropout Q-functions for doubly efficient reinforcement learning
- Sontakke, 2021, Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning, 9848
- Zintgraf, 2020, VariBAD: A very good method for Bayes-adaptive deep RL via meta-learning
- Yao, Direct policy transfer via hidden parameter Markov decision processes
- Higgins, 2017, beta-VAE: Learning basic visual concepts with a constrained variational framework
- Kim, 2018, Disentangling by factorising, 2649
- Locatello, 2020, Weakly-supervised disentanglement without compromises, 6348
- Shu, 2020, Weakly-supervised disentanglement with guarantees, 1094
- Zhang, 2021, Learning robust state abstractions for hidden parameter block
- Bica, 2021, Invariant causal imitation learning for generalizable policies, 34, 3952
- Thomas, 2017, Disentangling the independently controllable factors of variation by interacting with the world
- Haarnoja, 2018, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 1861
- Ahmed, 2021, CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning
- Ding, 2022, Generalizing goal-conditioned reinforcement learning with variational causal reasoning, 26532
- Truble, The role of pretained representations for the OOD generalization of RL agents, 2022
- Agarwal, 2021, Deep reinforcement learning at the edge of the statistical precipice, 29304
- Seitzer, 2021, Causal influence detection for improving efficiency in reinforcement learning, 22905
- Lee, 2020, Characterizing optimal mixed policies: Where to intervene and what to observe, 8565
- Sonar, 2021, Invariant policy optimization: Towards stronger generalization in reinforcement learning, 21
- Ahuja, 2020, Invariant risk minimization games, 145
- Zhang, 2020, Invariant causal prediction for block MDPs, 11214
- Turchetta, 2021, Safe reinforcement learning via curriculum induction
- Zhang, 2021, Learning invariant representations for reinforcement learning without reconstruction
- Sodhani, 2021, Multi-task reinforcement learning with context-based representations, 9767
- Yarats, 2021, Reinforcement learning with prototypical representations, 11920
Cited by
A Quality Prediction Approach for Compressor Blade Manufacturing With Various Geometric Dimensions
Liu, Q., Yang, Z., Dong, X., Dai, Y., Zeng, Z., Qiao, X., & Cheng, C. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3611990 · Cited by 1
“Pre-train and fine-tune is to train a network in the source domain (historical blade manufacturing data), directly use it on the data in the target domain (new iterative products), and perform fine-tuning on the target domain data [20].”
Automated UAV Controller Synthesis via LLM-Generated Control Logic and Particle Swarm Optimization
Carr, C., Martínez-García, M., Marshall, B. J., Coombes, M., & Zhang, Y. (2026) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2026.3675738 · Cited by 1
Rawal, A. (2026) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera69989.2026.11618581
“A few studies have highlighted that causal representations are essential for robust out-of-distribution generalization in autonomous systems [17], [18].”
Reconstruction-Free Vision-Based Control via Differentiable Long-Horizon Influence
Zhao, C., Tang, W., Huang, H., Li, D., & Zeng, Z. (2026) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2026.3734242
“Existing approaches address this issue through visual adaptation [45], task-oriented attention [46], self-supervised representation learning [47], or causal representation learning [48].”
Metadata
- Authors
- Huang, S., Hu, J., Chen, H., Cui, P., Piao, H., Sun, L., & Yang, B.
- Affiliations
- Jilin University (CN)
- Tsinghua University (CN)
- Lehigh University (US)
- Venue
- IEEE Transactions on Industrial Informatics
- Published
- 2025-07
- Volume
- 21
- Issue
- 7
- Pages
- 5514-5524
- ISSN
- 1551-3203 (ISSN-L)
- 1941-0050
- OpenAlex
- W4409683812
- Cited by
- 4
- References
- 16
- Citations by year
- 2026: 3 · 2025: 1
- FWCI
- 1.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Industrial Informatics (publishedVersion)
- Funders
- National Key R&D Program of China (2021ZD0112500)
- National Natural Science Foundation of China (U22A2098, U2341229, 62476110, 62172185, 61976102, 62206105, 62202200)
- Key R&D Project of Jilin Province (20240304200SF)
- International Cooperation Project of Jilin Province (20220402009GH)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- out-of-distribution generalization, causal reinforcement learning, disentangled representation learning, causal factors, weakly supervised learning, MuJoCo
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Generalizable Causal Reinforcement Learning for Out-of-Distribution Environments. https://app.cochat.ai/papers/10.1109/tii.2025.3556029
- The paper
- Huang, S., Hu, J., Chen, H., Cui, P., Piao, H., Sun, L., & Yang, B. (2025). Generalizable Causal Reinforcement Learning for Out-of-Distribution Environments. IEEE Transactions on Industrial Informatics. https://doi.org/10.1109/tii.2025.3556029
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.