The DOI matches its Crossref record, with a small difference in title or year.
Risk-sensitive reinforcement learning: a martingale approach to reward uncertainty
Abstract
We introduce a novel framework to account for sensitivity to rewards uncertainty in sequential decision-making problems. While risk-sensitive formulations for Markov decision processes studied so far focus on the distribution of the cumulative reward as a whole, we aim at learning policies sensitive to the uncertain/stochastic nature of the rewards, which has the advantage of being conceptually more meaningful in some cases. To this end, we present a new decomposition of the randomness contained in the cumulative reward based on the Doob decomposition of a stochastic process, and introduce a new conceptual tool - the chaotic variation - which can rigorously be interpreted as the risk measure of the martingale component associated to the cumulative reward process. We innovate on the reinforcement learning side by incorporating this new risk-sensitive approach into model-free algorithms, both policy gradient and value function based, and illustrate its relevance on grid world and portfolio optimization problems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep reinforcement learning (DRL) is a promising approach to automate trading in quantitative finance.
“Deep reinforcement learning (DRL), that balances exploration (of uncharted territory) and exploitation (of current knowledge), is a promising approach to automate trading in quantitative finance [50][51][47][54][21][13].”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [47]
Deep reinforcement learning (DRL) has a competitive edge in quantitative finance.
“Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance.”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [47]
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 26
- Marc G Bellemare Will Dabney and Rémi Munos. 2017. A Distributional Perspective on Reinforcement Learning. In ICML. Marc G Bellemare Will Dabney and Rémi Munos. 2017. A Distributional Perspective on Reinforcement Learning. In ICML .
- 10.1016/j.automatica.2009.07.008 · DOI
- 10.1287/moor.27.2.294.324 · DOI
- 10.1016/j.sysconle.2004.08.007 · DOI
- Vivek S Borkar . 2010 . Learning Algorithms for Risk-Sensitive Control . Proceedings of the Nineteenth International Symposium on Mathematical Theory of Networks and Systems (2010), 1327--1332. Vivek S Borkar. 2010. Learning Algorithms for Risk-Sensitive Control. Proceedings of the Nineteenth International Symposium on Mathematical Theory of Networks and Systems (2010), 1327--1332.
- Yinlam Chow , Mohammad Ghavamzadeh , Lucas Janson , and Marco Pavone . 2018 . Risk-Constrained Reinforcement Learning with Percentile Risk Criteria . J. Mach. Learn. Res. 18 , 167 (2018), 1 -- 51 . Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. 2018. Risk-Constrained Reinforcement Learning with Percentile Risk Criteria. J. Mach. Learn. Res. 18, 167 (2018), 1--51.
- 10.1023/A:1022632907294 · DOI
- Boris Defourny , Damien Ernst , and Louis Wehenkel . 2008 . Risk-Aware Decision Making and Dynamic Programming. In NIPS workshop. Boris Defourny, Damien Ernst, and Louis Wehenkel. 2008. Risk-Aware Decision Making and Dynamic Programming. In NIPS workshop.
- 10.1007/s00780-005-0159-6 · DOI
- S. Ganesh , N. Vadori , M. Xu , H. Zheng , P. Reddy , and M. Veloso . 2019. Reinforcement Learning for Market Making in a Multi-agent Dealer Market . In NeurIPS Workshop on Robust AI in Financial Services. S. Ganesh, N. Vadori, M. Xu, H. Zheng, P. Reddy, and M. Veloso. 2019. Reinforcement Learning for Market Making in a Multi-agent Dealer Market. In NeurIPS Workshop on Robust AI in Financial Services.
- 10.5555/1622519.1622522 · DOI
- Olivier Guéant and Iuliia Manziuk. 2019. Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality. (2019). arXiv:1910.13205 [q-fin.TR] Olivier Guéant and Iuliia Manziuk. 2019. Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality. (2019). arXiv:1910.13205 [q-fin.TR] · arXiv
- 10.1214/105051604000000116 · DOI
- Shie Mannor and John Tsitsiklis. 2011. Mean-Variance Optimization in Markov Decision Processes. In ICML. Shie Mannor and John Tsitsiklis. 2011. Mean-Variance Optimization in Markov Decision Processes. In ICML .
- 10.1023/A:1017940631555 · DOI
- Tetsuro Morimura Masashi Sugiyama and Hisashi Kashima. 2010. Parametric Return Density Estimation for Reinforcement Learning. In UAI. Tetsuro Morimura Masashi Sugiyama and Hisashi Kashima. 2010. Parametric Return Density Estimation for Reinforcement Learning. In UAI .
- Tetsuro Morimura Masashi Sugiyama Hisashi Kashima Hirotaka Hachiya and Toshiyuki Tanaka. 2010. Nonparametric Return Distribution Approximation for Reinforcement Learning. In ICML. Tetsuro Morimura Masashi Sugiyama Hisashi Kashima Hirotaka Hachiya and Toshiyuki Tanaka. 2010. Nonparametric Return Distribution Approximation for Reinforcement Learning. In ICML .
- L.a. Prashanth and Mohammad Ghavamzadeh . 2013 . Actor-Critic Algorithms for Risk-Sensitive MDPs . Advances in Neural Information Processing Systems 26 (2013), 252 -- 260 . L.a. Prashanth and Mohammad Ghavamzadeh. 2013. Actor-Critic Algorithms for Risk-Sensitive MDPs. Advances in Neural Information Processing Systems 26 (2013), 252--260.
- L A Prashanth and Michael Fu. 2018. Risk-Sensitive Reinforcement Learning: A Constrained Optimization Viewpoint. (Oct . 2018 ). arXiv:1810.09126 [cs.LG] L A Prashanth and Michael Fu. 2018. Risk-Sensitive Reinforcement Learning: A Constrained Optimization Viewpoint. (Oct. 2018). arXiv:1810.09126 [cs.LG] · arXiv
- 10.1007/s10994-016-5569-5 · DOI
- Yun Shen , Michael J Tobia , Tobias Sommer , and Klaus Obermayer . 2014. Risk-sensitive Reinforcement Learning. Neural Computation 26, 7 ( 2014 ). Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. 2014. Risk-sensitive Reinforcement Learning. Neural Computation 26, 7 (2014).
- Sobel MJ. 1982. The Variance of discounted Markov Decision Processes. J. Appl. Probab. (1982). Sobel MJ. 1982. The Variance of discounted Markov Decision Processes. J. Appl. Probab . (1982). · DOI
- Aviv Tamar , Dotan Di Castro , and Shie Mannor . 2016 . Learning the Variance of the Reward-To-Go . Journal of Machine Learning Research 17 , 13 (2016), 1 -- 36 . Aviv Tamar, Dotan Di Castro, and Shie Mannor. 2016. Learning the Variance of the Reward-To-Go. Journal of Machine Learning Research 17, 13 (2016), 1--36.
- Aviv Tamar Yinla Chow Mohammad Ghavamzadeh and Shie Mannor. 2015. Policy Gradient for Coherent Risk Measures. In NIPS. Aviv Tamar Yinla Chow Mohammad Ghavamzadeh and Shie Mannor. 2015. Policy Gradient for Coherent Risk Measures. In NIPS .
- Aviv Tamar , Dotan Di Castro, and Shie Mannor . 2012 . Policy Gradients with Variance Related Risk Criteria. In ICML. Aviv Tamar, Dotan Di Castro, and Shie Mannor. 2012. Policy Gradients with Variance Related Risk Criteria. In ICML.
- Tamar A. Glassner Y. Manor S. 2015. Optimizing the CVaR via Sampling. In AAAI. Tamar A. Glassner Y. Manor S. 2015. Optimizing the CVaR via Sampling. In AAAI . · DOI
Cited by
Sorted by citations
Liu, X.-Y., Yang, H., Gao, J., & Wang, C. D. (2021) · Proceedings of the Second ACM International Conference on AI in Finance · doi:10.1145/3490354.3494366 · Cited by 87
“More applications include stock trading [35, 47, 51, 54], futures contracts [54], alternative data (news sentiments) [22, 35], high frequency trading [15], liquidation strategy analysis [3], and hedging [6].”
Background
Metadata
- Authors
- Vadori, N., Ganesh, S., Reddy, P., & Veloso, M.
- Affiliations
- JPMorgan Chase & Co (United States) (US)
- Venue
- Proceedings of the First ACM International Conference on AI in Finance
- Publisher
- ACM
- Published
- 2020-10-15
- Pages
- 1-9
- OpenAlex
- W4288079541
- Cited by
- 25
- References
- 0
- Citations by year
- 2024: 2 · 2023: 4 · 2022: 4 · 2021: 4 · 2020: 2
- FWCI
- 1.34
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on AI in Finance (ICAIF) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- risk-sensitive reinforcement learning, reward uncertainty, sequential decision making, Markov decision process, risk measures, martingales, model-free reinforcement learning, policy gradient, portfolio optimization, grid world
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
