The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Recent advances in reinforcement learning in finance
Open accessAbstract
The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision‐making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value‐ and policy‐based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. We then discuss in detail the application of these RL algorithms in a variety of decision‐making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo‐advising. Our survey concludes by pointing out a few possible future directions for research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
The pricing of options and corporate liabilities
Black, F., & Scholes, M. (1973) · Journal of Political Economy · doi:10.1086/260062 · Cited by 29,074
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 19×
“Examples of policy-based methods include REINFORCE (Williams, 1992), Actor–Critic methods (Konda . . . Tsitsiklis, 2000), TRPO (Schulman et al., 2015), proximal policy optimization (PPO) (Schulman et al., 2017) among others.”
BackgroundMethodInfluentialWatkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 4×
“In addition, since introducing an entropy term in the objective function encourages policy exploration [105] and speeds the learning process [106, 162] (as discussed in Section 2.”
BackgroundMethodInfluentialSharpe, W. F. (1966) · The Journal of Business · doi:10.1086/294846 · Cited by 3,937
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455
Myopia and Inconsistency in Dynamic Utility Maximization
Strotz, R. H. (1955) · The Review of Economic Studies · doi:10.2307/2295722 · Cited by 2,834 · cited 1×
“The mean–variance framework is of particular interest because it not only captures both the portfolio return and the risk, but also suffers from the time-inconsistency problem (Vigna, 2016; Strotz, 1955; Xiao et al., 2020).”
BackgroundTHOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Some aspects of the sequential design of experiments
Robbins, H. (1952) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1952-09620-8 · Cited by 2,184
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 2×
“Both these ideas dramatically improve the empirical performance of the algorithm and DQN has been shown to have a strong performance for a variety of ATARI games [21].”
BackgroundSelf-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.01815 · Cited by 1,102 · cited 1×
“Breakthrough achievements include reaching human-level performance in such complex tasks as Go (Silver et al., 2017) and multiplayer StarCraft II (Vinyals et al., 2019), as well as the development of ChatGPT (Radford et al., 2019).”
BackgroundDeep Direct Reinforcement Learning for Financial Signal Representation and Trading
Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2522401 · Cited by 822 · cited 10×
“…functions include cash inflow or outflow (depending on whether we sell or buy) (Nevmyvaka et al., 2006; Shen et al., 2014), implementation shortfall (Hendricks &Wilcox, 2014), profit (Deng et al., 2017), Sharpe ratio (Deng et al., 2017), return (Jeong . . . Kim, 2019), and PnL (Wei et al., 2019).”
BackgroundMethodInfluentialRule of Law and Constitutional Democracy
(2015) · DergiPark (Istanbul University) · doi:10.2139/ssrn · Cited by 804
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Buehler, H., Gonon, L., Teichmann, J., & Wood, B. (2019) · Quantitative Finance · doi:10.1080/14697688.2019.1571683 · Cited by 333
A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
Jiang, Z., Xu, D., & Liang, J. (2017) · arXiv · doi:10.48550/arxiv.1706.10059 · Cited by 321 · cited 41×
“The benchmark strategies include Constantly Rebalanced Portfolio (CRP) (Yu et al., 2019; Jiang et al., 2017) where at each period the portfolio is rebalanced to the initial wealth distribution among the assets, and the buy-and-hold or do-nothing strategy (Park et al., 2020; Aboussalah, 2020), which…”
BackgroundMethodInfluentialDeep Reinforcement Learning for Trading
Zhang, Z., Zohren, S., & Roberts, S. (2020) · The Journal of Financial Data Science · doi:10.3905/jfds.2020.1.030 · Cited by 227
Minimax Regret Bounds for Reinforcement Learning
Azar, M. G., Osband, I., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.05449 · Cited by 124
Liu, X.-Y., Yang, H., Gao, J., & Wang, C. D. (2021) · Proceedings of the Second ACM International Conference on AI in Finance · doi:10.1145/3490354.3494366 · Cited by 87 · cited 1×
“It is worth noting that there are some open source projects that provide full pipelines for implementing different RL algorithms in financial applications (Liu et al., 2021, 2022).”
BackgroundSelf-citationRisk-sensitive reinforcement learning: a martingale approach to reward uncertainty
Vadori, N., Ganesh, S., Reddy, P., & Veloso, M. (2020) · Proceedings of the First ACM International Conference on AI in Finance · doi:10.1145/3383455.3422519 · Cited by 25
Metadata
- Authors
- Hambly, B., Xu, R., & Yang, H.
- Venue
- Mathematical Finance
- Publisher
- Wiley
- Published
- 2023-04-07
- Volume
- 33
- Issue
- 3
- Pages
- 437-503
- ISSN
- 0960-1627 (ISSN-L)
- 1467-9965
- OpenAlex
- W4362722548
- Cited by
- 205
- References
- 26
- FWCI
- 42.61
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.