Approximate information state for partially observed systems
Abstract
The standard approach for modeling partially observed systems is to model them as partially observable Markov decision processes (POMDPs) and obtain a dynamic program in terms of a belief state. The belief state formulation works well for planning but is not ideal for online reinforcement learning because the belief state depends on the model and, as such, is not observable when the model is unknown.In this paper, we present an alternative notion of an information state for obtaining a dynamic program in partially observed models. In particular, an information state is a sufficient statistic for the current reward which evolves in a controlled Markov manner. We show that such an information state leads to a dynamic programming decomposition. Then we present a notion of an approximate information state and present an approximate dynamic program based on the approximate information state. Approximate information state is defined in terms of properties that can be estimated using sampled trajectories. Therefore, they provide a constructive method for reinforcement learning in partially observed systems. We present one such construction and show that it performs better than the state of the art for three benchmark models.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In order to perfectly clone a behavior that depends on the state, the learner must implicitly learn a task-relevant dynamics model.
“This shows that in order to perfectly clone a behavior that depends on the state, the learner must implicitly learn a (task-relevant) dynamics model Subramanian and Mahajan (2019); Zhang et al. (2020).”
From Diffusion policy: Visuomotor policy learning via action diffusion (2025) · cites this paper as Subramanian and Mahajan (2019)
To perfectly clone a behavior that depends on the state, the learner must implicitly learn a task-relevant dynamics model.
“This shows that in order to perfectly clone a behavior that depends on the state, the learner must implicitly learn a (task-relevant) dynamics model Subramanian and Mahajan (2019); Zhang et al. (2020).”
From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as Subramanian and Mahajan (2019)
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · see citing works
Optimal Transport: Old and New
Villani, C. (2013) · Grundlehren der mathematischen Wissenschaften · doi:10.1007/978-3-540-71050-9 · Cited by 3,838 · see citing works
The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
Smallwood, R., & Sondik, E. (1973) · Operational Research · doi:10.1287/opre.21.5.1071 · Cited by 1,787 · see citing works
Deep Recurrent Q-Learning for Partially Observable MDPsPreprint
Hausknecht, M., & Stone, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06527 · Cited by 1,224 · see citing works
Discrete-Time Markov Control Processes
Hernández–Lerma, O., & Lasserre, J. B. (1996) · Springer eBooks · doi:10.1007/978-1-4612-0729-0 · Cited by 1,085 · see citing works
Partially observable Markov decision processes for spoken dialog systems
Williams, J. D., & Young, S. J. (2006) · Computer Speech & Language · doi:10.1016/j.csl.2006.06.008 · Cited by 853 · see citing works
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Journal of Artificial Intelligence Research · doi:10.1613/jair.806 · Cited by 641 · see citing works
Acting Optimally in Partially Observable Stochastic Domains
Cassandra, A. R., Kaelbling, L. P., & Littman, M. L. (1994) · Cited by 570 · see citing works
Linear automaton transformations
Nerode, A. (1958) · Proceedings of the American Mathematical Society · doi:10.1090/s0002-9939-1958-0135681-9 · Cited by 541 · see citing works
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Cited by 525 · see citing works
A survey of point-based POMDP solvers
Shani, G., Pineau, J., & Kaplow, R. (2012) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-012-9200-2 · Cited by 427 · see citing works
Predictive Representations of State
Littman, M. L., & Sutton, R. S. (2001) · Cited by 424 · see citing works
Stochastic approximation with two time scales
Borkar, V. S. (1997) · Systems & Control Letters · doi:10.1016/s0167-6911(97)90015-3 · Cited by 394 · see citing works
Memory-based control with recurrent neural networks
Heess, N., Hunt, J. J., Lillicrap, T. P., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1512.04455 · Cited by 211 · see citing works
Reinforcement Learning with Long Short-Term Memory
Bakker, B. (2001) · Cited by 209 · see citing works
Discrete-Time Markov Control Processes: Basic Optimality Criteria.
PALE, Hernández–Lerma, O., & Lasserre, J. B. (1996) · Journal of the American Statistical Association · doi:10.2307/2291618 · Cited by 190 · see citing works
Convergence of discretization procedures in dynamic programming
Bertsekas, D. P. (1975) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1975.1100984 · Cited by 181 · see citing works
Sufficient statistics in the optimum control of stochastic systems
Striebel, C. T. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90027-2 · Cited by 180 · see citing works
Solving Deep Memory POMDPs with Recurrent Policy Gradients
Wierstra, D., Foerster, A., Peters, J., & Schmidhuber, J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74690-4_71 · Cited by 143 · see citing works
Wierstra, D., Förster, A., Peters, J., & Schmidhuber, J. (2009) · Logic Journal of IGPL · doi:10.1093/jigpal/jzp049 · Cited by 105 · see citing works
Metrics for Finite Markov Decision ProcessesPreprint
Ferns, N., Panangaden, P., & Precup, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4114 · Cited by 55 · see citing works
Equivalence relations in fully and partially observable Markov decision processes
Castro, P. S., Panangaden, P., & Precup, D. (2009) · Cited by 29 · see citing works
Some Remarks on the Concept of State
Witsenhausen, H. S. (1976) · Springer eBooks · doi:10.1007/978-1-4684-2259-7_6 · Cited by 24 · see citing works
Recurrent Predictive State Policy NetworksPreprint
Hefny, A., Marinho, Z., Sun, W. Y., Srinivasa, S. S., & Gordon, G. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01489 · Cited by 12 · see citing works
Learning in the presence of partial observability and concept drifts
Seraj, R. (2019) · eScholarship@McGill (McGill) · doi:10.82308/11151 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 31 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Diffusion policy: Visuomotor policy learning via action diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241273668 · Cited by 613 · see citing works
“But interestingly, it need only learn the task-relevant dynamics (Subramanian and Mahajan, 2019; Zhang et al., 2020); in the linear dynamics example here, if K has a column that is all zeros, then leftmost multiplication by K means that the associated state dynamics are not relevant to the task and…”
Active Sensing for Communications by Learning
Sohrabi, F., Jiang, T., Cui, W., & Yu, W. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3155496 · Cited by 68 · see citing works
Subramanian, J., Sinha, A. K., Seraj, R., & Mahajan, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08843 · Cited by 44 · see citing works
Self-citationKara, A. D., & Yüksel, S. (2022) · Mathematics of Operations Research · doi:10.1287/moor.2022.1331 · Cited by 29 · see citing works
Information State Embedding in Partially Observable Cooperative Multi-Agent Reinforcement Learning
Mao, W., Zhang, K., Miehling, E., & Başar, T. (2020) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc42340.2020.9303801 · Cited by 22 · see citing works
Separation of learning and control for cyber–physical systems
Malikopoulos, A. A. (2023) · Automatica · doi:10.1016/j.automatica.2023.110912 · Cited by 19 · see citing works
Kim, H., Park, C., Kim, J., Jang, S., & Lee, H. K. (2024) · IEEE Access · doi:10.1109/access.2024.3405009 · Cited by 15 · see citing works
Medical Dead-ends and Learning to Identify High-risk States and TreatmentsPreprint
Fatemi, M., Killian, T. W., Subramanian, J., & Ghassemi, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04186 · Cited by 15 · see citing works
Self-citationDiffusion Policy: Visuomotor Policy Learning via Action Diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2303.04137 · Cited by 12 · see citing works
“But interestingly, it need only learn the task-relevant dynamics (Subramanian and Mahajan, 2019; Zhang et al., 2020); in the linear dynamics example here, if K has a column that is all zeros, then leftmost multiplication by K means that the associated state dynamics are not relevant to the task and…”
Near Optimality of Finite Memory Feedback Policies in Partially Observed Markov Decision ProcessesPreprint
Kara, A. D., & Yüksel, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07452 · Cited by 13 · see citing works
Approximate Information States for Worst-case Control of Uncertain Systems
Dave, A., Nishanth Venkatesh, S., & Malikopoulos, A. A. (2022) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc51059.2022.9992631 · Cited by 8 · see citing works
Approximate Information States for Worst Case Control and Learning in Uncertain Systems
Dave, A., Nishanth Venkatesh, S., & Malikopoulos, A. A. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3422889 · Cited by 7 · see citing works
A Dynamic Program for a Team of Two Agents with Nested Information
Dave, A., & Malikopoulos, A. A. (2021) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc45484.2021.9683339 · Cited by 6 · see citing works
Decentralized Multi-agent Reinforcement Learning with Shared Actions
Mishra, R. K., Vasal, D., & Vishwanath, S. (2021) · Conference on Information Sciences and Systems · doi:10.1109/ciss50987.2021.9400275 · Cited by 5 · see citing works
A Framework for Effective AI Recommendations in Cyber-Physical-Human Systems
Dave, A., Bang, H., & Malikopoulos, A. A. (2024) · IEEE Control Systems Letters · doi:10.1109/lcsys.2024.3410145 · Cited by 5 · see citing works
Variance Decay Property for Filter Stability
Kim, J. W., & Mehta, P. G. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3413573 · Cited by 5 · see citing works
Common Information based Approximate State Representations in Multi-Agent Reinforcement LearningPreprint
Kao, H., & Subramanian, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12603 · Cited by 5 · see citing works
Semiparametric Information State Embedding for Policy Search under Imperfect Information
Bhatt, S., Mao, W., Koppel, A., & Başar, T. (2021) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc45484.2021.9682964 · Cited by 4 · see citing works
Decentralized Learning of Finite-Memory Policies in Dec-POMDPs
Mao, W., Zhang, K., Yang, Z., & Başar, T. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.1346 · Cited by 3 · see citing works
Convergence and Near Optimality of Q-Learning with Finite Memory for Partially Observed Models
Kara, A. D., & Yüksel, S. (2021) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc45484.2021.9682777 · Cited by 2 · see citing works
A Multi-Agent View of Wireless Video Streaming with Delayed Client-Feedback
Khan, N., Dinesha, U., Arunachalam, S., Narasimha, D., Subramanian, V., & Shakkottai, S. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621216 · Cited by 1 · see citing works
Time Block Decomposition of Multistage Stochastic Optimization Problems
Carpentier, P., Chancelier, J., Lara, M. D., Martin, T., & Rigaut, T. (2023) · Journal of convex analysis · doi:10.68381/jca30034 · Cited by 1 · see citing works
Zhang, Q., Lin, B., & Liu, Y. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07393-w
- Authors
- Subramanian, J., & Mahajan, A.
- Affiliations
- McGill University (CA)
- Venue
- 2019 IEEE 58th Conference on Decision and Control (CDC)
- Publisher
- IEEE
- Published
- 2019-12
- Pages
- 1629-1636
- ISSN
- 0191-2216 (ISSN-L)
- OpenAlex
- W3011813374
- Cited by
- 23
- References
- 31
- Citations by year
- 2026: 1 · 2024: 6 · 2023: 3 · 2022: 3 · 2021: 7 · 2020: 3
- FWCI
- 1.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Conference on Decision and Control (CDC) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- partially observed systems, partially observable Markov decision processes, belief states, online reinforcement learning, sufficient statistics, reward modeling, model-free learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
