FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
As deep reinforcement learning (DRL) has been recognized as an effective approach in quantitative finance, getting hands-on experiences is attractive to beginners. However, to train a practical DRL trading agent that decides where to trade, at what price, and what quantity involves error-prone and arduous development and debugging. In this paper, we introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance and to develop their own stock trading strategies. Along with easily-reproducible tutorials, FinRL library allows users to streamline their own developments and to compare with existing schemes easily. Within FinRL, virtual environments are configured with stock market datasets, trading agents are trained with neural networks, and extensive backtesting is analyzed via trading performance. Moreover, it incorporates important trading constraints such as transaction cost, market liquidity and the investor's degree of risk-aversion. FinRL is featured with completeness, hands-on tutorial and reproducibility that favors beginners: (i) at multiple levels of time granularity, FinRL simulates trading environments across various stock markets, including NASDAQ-100, DJIA, S&P 500, HSI, SSE 50, and CSI 300; (ii) organized in a layered architecture with modular structure, FinRL provides fine-tuned state-of-the-art DRL algorithms (DQN, DDPG, PPO, SAC, A2C, TD3, etc.), commonly-used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the reproducibility, and (iii) being highly extendable, FinRL reserves a complete set of user-import interfaces. Furthermore, we incorporated three application demonstrations, namely single stock trading, multiple stock trading, and portfolio allocation. The FinRL library will be available on Github at link https://github.com/AI4Finance-Foundation/FinRL.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most existing libraries [13, 19, 25, 26] send the gradients of learners by following a traditional synchronization approach on supervised learning.”
From FinRL-podracer: high performance and scalable deep reinforcement learning for quantitative finance (2021) · cites this paper as [25]
In the citing paper
“For DRL strategy design, existing works studied algorithmic trading on a daily time-frame [2, 25, 26, 42-45] or hourly time-frame [14], which is hard to fully capture the dynamics of a highly volatile market.”
From FinRL-podracer: high performance and scalable deep reinforcement learning for quantitative finance (2021) · cites this paper as [25]
In the citing paper
“both FinRL [25] and Stable Baseline 3 [9] do not support the training on multiple GPUs”
From FinRL-podracer: high performance and scalable deep reinforcement learning for quantitative finance (2021) · cites this paper as [25]
In the citing paper
“We evaluate the training time of three popular DRL libraries, FinRL [25, 26], RLlib [19] and Stable Baseline3 [9], on NASDAQ-100 constituent stocks with minutelevel data (described in Section 5.2) in Table 1, which shows that it is difficult for them to effectively train a profitable trading agent in a short cycle time.”
From FinRL-podracer: high performance and scalable deep reinforcement learning for quantitative finance (2021) · cites this paper as [25]
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 2×
“FinRL library includes fine-tuned standard DRL algorithms, namely, DQN [29][34], DDPG [29], Multi-Agent DDPG [30], PPO [40], SAC [18], A2C [33] and TD3 [11].”
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · cited 6×
“In the middle is the agent layer that provides fine-tuned standard DRL algorithms (DQN [29][34], DDPG [29], Adaptive DDPG [27], Multi-Agent DDPG [30], PPO [40], SAC [18], A2C [33] and TD3 [11], etc.), commonly used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the r…”
InfluentialProximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 4×
“In the middle is the agent layer that provides fine-tuned standard DRL algorithms (DQN [29][34], DDPG [29], Adaptive DDPG [27], Multi-Agent DDPG [30], PPO [40], SAC [18], A2C [33] and TD3 [11], etc.), commonly used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the r…”
InfluentialSoft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 4×
“In the middle is the agent layer that provides fine-tuned standard DRL algorithms (DQN [29][34], DDPG [29], Adaptive DDPG [27], Multi-Agent DDPG [30], PPO [40], SAC [18], A2C [33] and TD3 [11], etc.), commonly used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the r…”
InfluentialMulti-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017) · arXiv · doi:10.48550/arxiv.1706.02275 · Cited by 3,377 · cited 5×
“In the middle is the agent layer that provides fine-tuned standard DRL algorithms (DQN [29][34], DDPG [29], Adaptive DDPG [27], Multi-Agent DDPG [30], PPO [40], SAC [18], A2C [33] and TD3 [11], etc.”
BackgroundMethodInfluentialPortfolio theory and capital markets
Breen, W. J., & Sharpe, W. F. (1972) · The Journal of Finance · doi:10.2307/2978700 · Cited by 1,221 · cited 1×
“Investors usually have two objectives for their decisions: the highest possible profits and the lowest possible risks of uncertainty [41].”
Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2522401 · Cited by 822 · cited 3×
“Deng et al. [12] showed that applying deep neural networks profits more.”
Learning to trade via direct reinforcement
Moody, J., & Saffell, M. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935097 · Cited by 535 · cited 1×
“Moody and Saffell [36] utilized policy search and direct RL for stock trading.”
RLlib: Abstractions for Distributed Reinforcement Learning
Liang, E., Liaw, R., Moritz, P., Nishihara, R., Fox, R., Goldberg, K., Gonzalez, J. E., Jordan, M. I., & Stoica, I. (2017) · arXiv · doi:10.48550/arxiv.1712.09381 · Cited by 473
How effective are neural networks at forecasting and prediction? A review and evaluation
Adya, M., & Collopy, F. (1998) · Journal of Forecasting · doi:10.1002/(sici)1099-131x(1998090)17:5/6<481::aid-for709>3.0.co;2-q · Cited by 451
A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
Jiang, Z., Xu, D., & Liang, J. (2017) · arXiv · doi:10.48550/arxiv.1706.10059 · Cited by 321
(2017) · doi:10.1109/intellisys.2017.8324237 · Cited by 298
Deep Reinforcement Learning for Trading
Zhang, Z., Zohren, S., & Roberts, S. (2020) · The Journal of Financial Data Science · doi:10.3905/jfds.2020.1.030 · Cited by 227 · cited 5×
“As a result, DRL provides useful toolkits for stock trading [21, 44, 48, 45, 10, 8, 26].”
InfluentialSkulls, Financial Turbulence, and Risk Management
Kritzman, M., & Li, Y. (2010) · Financial Analysts Journal · doi:10.2469/faj.v66.n5.3 · Cited by 214 · cited 1×
“To control the risk in a worst-case scenario, such as financial crisis of 2007–2008, FinRL employs the financial turbulence index turbulencet that measures extreme asset price fluctuation [25]: turbulencet = (yt − μ)Σ −1(yt − μ) ′ ∈ R, (1)”
Deep reinforcement learning for automated stock trading: an ensemble strategy
Yang, H., Liu, X.-Y., Zhong, S., & Walid, A. (2020) · Proceedings of the First ACM International Conference on AI in Finance · doi:10.1145/3383455.3422540 · Cited by 196
Self-citationPassive investment strategies and efficient markets
Malkiel, B. G. (2003) · European Financial Management · doi:10.1111/1468-036x.00205 · Cited by 181 · cited 1×
“FinRL uses five conventional strategies, namely passive buy-and-hold trading strategy [32], mean-variance strategy [1], and min-variance strategy [1], momentum trading strategy [15], and equal-weighted strategy to address these two mutually limiting objectives and the industrial standards.”
Dankwa, S., & Zheng, W. (2019) · Proceedings of the 3rd International Conference on Vision, Image and Signal Processing · doi:10.1145/3387168.3387199 · Cited by 173 · cited 5×
“In the middle is the agent layer that provides fine-tuned standard DRL algorithms (DQN [29][34], DDPG [29], Adaptive DDPG [27], Multi-Agent DDPG [30], PPO [40], SAC [18], A2C [33] and TD3 [11], etc.), commonly used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the r…”
InfluentialDopamine: A Research Framework for Deep Reinforcement Learning
Castro, P. S., Moitra, S., Gelada, C., Kumar, S., & Bellemare, M. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06110 · Cited by 154 · cited 1×
“• Google Dopamine [7] is a research framework for fast prototyping of reinforcement learning algorithms.”
Experience Replay Optimization
Zha, D., Lai, K.-H., Zhou, K., & Hu, X. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/589 · Cited by 153 · cited 1×
“2 Standard and User Import Datasets The application of DRL in finance is different from that in other fields, such as playing chess and card games [42, 46]; the latter inherently have clearly defined rules for environments.”
Deep Reinforcement Learning for Automated Stock Trading: An Ensemble StrategyPreprint
Yang, H., Liu, X.-Y., Zhong, S., & Walid, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3690996 · Cited by 115
Self-citationFinancial Trading as a Game: A Deep Reinforcement Learning ApproachPreprint
Huang, C. Y. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1807.02787 · Cited by 114
Deep Hedging of Derivatives Using Reinforcement Learning
Cao, J., Chen, J., Hull, J., & Poulos, Z. (2020) · The Journal of Financial Data Science · doi:10.3905/jfds.2020.1.052 · Cited by 111 · cited 1×
“Deep Hedging [5, 6] represents hedging strategies with neural networks learned by modern DRL policy search.”
Practical Deep Reinforcement Learning Approach for Stock Trading
Liu, X., Xiong, Z., Zhong, S., Yang, H., & Walid, A. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1811.07522 · Cited by 93 · cited 3×
“…commonly used ones [14] as follows: • The change of the portfolio value when action a is taken at state s and arriving at new state s (cid:48) [12, 44, 10, 37, 45], i.e., r ( s, a, s (cid:48) ) = v (cid:48) − v , where v (cid:48) and v represent the portfolio values at state s (cid:48) and s ,…”
Self-citationHorizon: Facebook's Open Source Applied Reinforcement Learning Platform
Gauci, J., Conti, E., Liang, Y., Virochsiri, K., He, Y., Kaden, Z., Narayanan, V., Ye, X., Chen, Z., & Fujimoto, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00260 · Cited by 83 · cited 1×
“• Horizon [17] is a DL-focused framework dominated by PyTorch, whose main use case is to train RL models in the batch setting.”
Application of deep reinforcement learning on automated stock trading
Chen, L., & Gao, Q. (2019) · 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess47205.2019.9040728 · Cited by 78
(2018) · arXiv (Cornell University) · Cited by 74
Self-citationKim, Y., Ahn, W., Oh, K. J., & Enke, D. (2017) · Applied Soft Computing · doi:10.1016/j.asoc.2017.02.006 · Cited by 64 · cited 1×
“Taking many complex financial factors into account, DRL trading agents build a multi-factor model and provide algorithmic trading strategies, which are difficult for human traders [3, 47, 24, 22].”
Reinforcement Learning in Stock Trading
Dang, Q.-V. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-030-38364-0_28 · Cited by 57 · cited 4×
“We provide commonly used ones [14] as follows: • The change of the portfolio value when action a is taken at state s and arriving at new state s [12, 44, 10, 37, 45], i.”
InfluentialLi, X., Li, Y., Zhan, Y., & Liu, X. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1907.01503 · Cited by 54 · cited 4×
“For example, DRL has been implemented on sentimental analysis on portfolio allocation [27, 22] and liquidation strategy analysis [2], showing the potential of DRL on various financial tasks.”
InfluentialSelf-citationSentiment and Knowledge Based Algorithmic Trading with Deep Reinforcement Learning
Nan, A., Perumal, A., & Zai͏̈ane, O. R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-12423-5_13 · Cited by 54
Fuzzy adaptive decision-making for boundedly rational traders in speculative stock markets
Bekiros, S. D. (2010) · European Journal of Operational Research · doi:10.1016/j.ejor.2009.04.015 · Cited by 53 · cited 1×
“Taking many complex financial factors into account, DRL trading agents build a multi-factor model and provide algorithmic trading strategies, which are difficult for human traders [3, 47, 24, 22].”
Multi-Agent Deep Reinforcement Learning for Liquidation Strategy AnalysisPreprint
Bao, W., & Liu, X. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1906.11046 · Cited by 53 · cited 1×
“For example, DRL has been implemented on sentimental analysis on portfolio allocation [27, 22] and liquidation strategy analysis [2], showing the potential of DRL on various financial tasks.”
Self-citationDeep Hedging: Hedging Derivatives Under Generic Market Frictions Using Reinforcement Learning
Buehler, H., Gonon, L., Teichmann, J., Wood, B., Mohan, B., & Kochems, J. (2019) · doi:10.2139/ssrn.3355706 · Cited by 48 · cited 3×
“Deep Hedging [5, 6] represents hedging strategies with neural networks learned by modern DRL policy search.”
Sentiment and Knowledge Based Algorithmic Trading with Deep Reinforcement Learning
Nan, A., Perumal, A., & Zaiane, O. R. (2020) · arXiv · doi:10.48550/arxiv.2001.09403 · Cited by 39 · cited 3×
“News headline sentiments and knowledge graphs can also be combined with the time series stock data to train an optimal policy using DRL [37].”
Online portfolio selection strategy based on combining experts' advice
Zhang, Y., & Yang, X. (2017) · Computational Economics · doi:10.1007/s10614-016-9585-0 · Cited by 38 · cited 1×
“ng [21, 44, 48, 45, 10, 8, 26]. Taking many complex financial factors into account, DRL tradingagentsbuilda multi-factormodelandprovidealgorithmictradingstrategies, whichare difficult for human traders [3, 47, 24, 22]. Preceding DRL, conventional reinforcement learning (RL) [43] has been applied to complex financial prob…”
Profitable momentum trading strategies for individual investors
Foltice, B., & Langer, T. (2015) · Financial markets and portfolio management · doi:10.1007/s11408-015-0246-4 · Cited by 34 · cited 1×
“FinRL uses five conventional strategies, namely passive buy-and-hold trading strategy [32], mean-variance strategy [1], and min-variance strategy [1], momentum trading strategy [15], and equal-weighted strategy to address these two mutually limiting objectives and the industrial standards.”
Learning to Trade with Deep Actor Critic Methods
Li, J., Rao, R., & Shi, J. (2018) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid.2018.10116 · Cited by 27 · cited 2×
“As a result, DRL provides useful toolkits for stock trading [21, 44, 48, 45, 10, 8, 26].”
Ang, A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2131932 · Cited by 21 · cited 1×
“FinRL uses five conventional strategies, namely passive buy-and-hold trading strategy [32], mean-variance strategy [1], and min-variance strategy [1], momentum trading strategy [15], and equal-weighted strategy to address these two mutually limiting objectives and the industrial standards.”
Deep Hedging of Derivatives Using Reinforcement LearningPreprint
Cao, J., Chen, J. H. K., Hull, J. C., & Poulos, Z. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3514586 · Cited by 21
Ang, A. (2014) · Asset Management · doi:10.1093/acprof:oso/9780199959327.003.0003 · Cited by 21
Q-Learning-Based Financial Trading Systems with ApplicationsPreprint
Corazza, M., & Bertoluzzo, F. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2507826 · Cited by 10 · cited 1×
“Many researchers have explored various approaches using DRL [38, 37, 10, 9, 48, 16].”
VLSTM: Very Long Short-Term Memory Networks for High-Frequency TradingPreprint
Ganesh, P., & Rakheja, P. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1809.01506 · Cited by 9
Deep Reinforcement Learning in High Frequency Trading.
Ganesh, P., & Rakheja, P. (2018) · Cited by 6
Research works printed but not linked: 7
- G Thomas, 2018, Reinforcement learning in financial markets -a survey. Fau discussion papers in economics
- John Hull, 2009, Options, futures and other derivatives/John C. Hull
- Olivier Jin, 2016, Portfolio management using reinforcement learning
- Ryan Lowe, 2017, Multi-agent actorcritic for mixed cooperative-competitive environments, Advances in Neural Information Processing Systems, 6379
- G David, 1997, Investment science
- Pg Nechchi, 2016, Reinforcement learning for automated trading, Mathematical EngineeringPolitecnico di Milano
- Quantopian, 2019, A toolkit for portfolio and risk analytics in python
Cited by
Huang, Y., Zhou, C., Cui, K., & Lu, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121502 · Cited by 104
GraphSAGE with deep reinforcement learning for financial portfolio optimization
Sun, Q., Wei, X., & Yang, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122027 · Cited by 78
A novel deep reinforcement learning framework with BiLSTM-Attention networks for algorithmic trading
Huang, Y., Wan, X., Zhang, L., & Lu, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122581 · Cited by 70
Dynamic datasets and market environments for financial reinforcement learning
Liu, X.-Y., Xia, Z., Yang, H., Gao, J., Zha, D., Zhu, M., Wang, C. D., Wang, Z., & Guo, J. (2024) · Machine Learning · doi:10.1007/s10994-023-06511-w · Cited by 70
“The first three are technological innovations: • FinRL (Liu et al., 2020, 2021) is the first framework to provide an automatic pipeline for financial reinforcement learning.”
InfluentialSelf-citationDeep reinforcement learning based on transformer and U-Net framework for stock trading
Yang, B., Liang, T., Xiong, J., & Zhong, C. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110211 · Cited by 43
Metadata
- Authors
- Liu, X.-Y., Yang, H., Chen, Q., Zhang, R., Yang, L., Xiao, B., & Wang, C.
- Affiliations
- Columbia University (US)
- Imperial College London (GB)
- New York University (US)
- Corresponding author
- Yang, Hongyang
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2025
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W3162607076
- Semantic Scholar
- 1fc643611860d9cb46df0251c35c2c55a941f3f2
- Cited by
- 53
- References
- 46
- Citations by year
- 2026: 1 · 2025: 11
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
- Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
- Social Sciences › Business, Management and Accounting › Management Information Systems › FinTech, Crowdfunding, Digital Finance
- Keywords
- deep reinforcement learning, automated stock trading, financial mathematics, backtesting, transaction costs, market liquidity, risk aversion, asset allocation, deep Q-network, deep deterministic policy gradient, PPO, SAC, TD3, Advantage Actor-Critic, reward function, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.