Deep reinforcement learning for automated stock trading: an ensemble strategy
The DOI matches its Crossref record, with a small difference in title or year; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Stock trading strategies play a critical role in investment. However, it is challenging to design a profitable strategy in a complex and dynamic stock market. In this paper, we propose an ensemble strategy that employs deep reinforcement schemes to learn a stock trading strategy by maximizing investment return. We train a deep reinforcement learning agent and obtain an ensemble trading strategy using three actor-critic based algorithms: Proximal Policy Optimization (PPO), Advantage Actor Critic (A2C), and Deep Deterministic Policy Gradient (DDPG). The ensemble strategy inherits and integrates the best features of the three algorithms, thereby robustly adjusting to different market situations. In order to avoid the large memory consumption in training networks with continuous action space, we employ a load-on-demand technique for processing very large data. We test our algorithms on the 30 Dow Jones stocks that have adequate liquidity. The performance of the trading agent with different reinforcement learning algorithms is evaluated and compared with both the Dow Jones Industrial Average index and the traditional min-variance portfolio allocation strategy. The proposed deep ensemble strategy is shown to outperform the three individual algorithms and two baselines in terms of the risk-adjusted return measured by the Sharpe ratio.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deep reinforcement learning (DRL), that balances exploration (of uncharted territory) and exploitation (of current knowledge), is a promising approach to automate trading in quantitative finance [50][51][47][54][21][13].”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [51]
In the citing paper
“Therefore, the backtesting performance demonstrates that FinRL successfully reproduces the ensemble strategy [51].”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [51]
In the citing paper
“The ensemble strategy [51] combines three DRL algorithms (PPO [42], A2C [32] and DDPG [26]) to improve the robustness.”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [51]
In the citing paper
“Explainable DRL for portfolio management [17] and DRL ensemble strategy for stock trading [50, 51] are also implemented.”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [51]
In the citing paper
“The performance in terms of multiple metrics is consistent with the results reported in [51] and [50], and here we show results in a recent trading period.”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [51]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · cited 12×
“DDPG performs similar but not as good as PPO, it can be used as a complementary strategy to PPO in a bullish market.”
InfluentialProximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 3×
“The three actor-critic algorithms [24] are Proximal Policy Optimization (PPO) [28, 37], Advantage Actor Critic (A2C) [32, 48], and Deep Deterministic Policy Gradient (DDPG) [28, 29, 44].”
BackgroundMethod(1995) · Cited by 10,558
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 2×
“A2C [32] is a typical actor-critic algorithm and we use it a component in the ensemble strategy.”
BackgroundMethodMarkowitz, H. (1952) · The Journal of Finance · doi:10.1111/j.1540-6261.1952.tb01525.x · Cited by 5,641
(1999) · Cited by 5,147
Sharpe, W. F. (1994) · The Journal of Portfolio Management · doi:10.3905/jpm.1994.409501 · Cited by 2,534
Konda, V. R., & Tsitsiklis, J. N. (2002) · Cited by 2,202
International Journal of Economics and Financial Issues
(2021) · International Journal of Economics and Financial Issues · doi:10.32479/ijefi · Cited by 2,008
International Conference on Learning Representations (ICLR)
Abbas, M. Z. A., Kivinen, J., & Raiko, T. (2016) · Cited by 1,694
Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2522401 · Cited by 822 · cited 1×
“The actor-only approach has been used in [24], [25], [26].”
Conference on Neural Information Processing Systems
Liu, L., Zhou, T., Long, G., Jiang, J., & Zhang, C. (2019) · UTS ePRESS (University of Technology Sydney) · Cited by 593
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · cited 1×
“We employ OpenAI gym to implement our environment and train the agent [32], [33], [34].”
Learning to trade via direct reinforcement
Moody, J., & Saffell, M. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935097 · Cited by 535 · cited 2×
“[24].”
Reinforcement learning for control: Performance, stability, and deep approximators
Buşoniu, L., de Bruin, T., Tolić, D., Kober, J., & Palunko, I. (2018) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2018.09.005 · Cited by 463 · cited 1×
“In contrast with DQN that learns indirectly through Q-values tables and suffers the curse of dimensionality problem [40], DDPG learns directly from the observations through policy gradient.”
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401 · cited 1×
“For future work, it will be interesting to explore more sophisticated model [42], solve empirical challenges [15], deal with large-scale data [7] such as S&P 500 constituent stocks.”
Background(2017) · doi:10.1109/intellisys.2017.8324237 · Cited by 298
Cryptocurrency Portfolio Management with Deep Reinforcement Learning
Jiang, Z., & Liang, J. (2016) · arXiv · doi:10.48550/arxiv.1612.01277 · Cited by 298
Trading Strategies to Exploit Blog and News Sentiment
Zhang, W., & Skiena, S. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14075 · Cited by 252 · cited 1×
“In recent years, machine learning and deep learning algorithms have been widely applied to build prediction and classification models for the financial market.”
Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation
Wang, L., Zhang, W., He, X., & Zha, H. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219961 · Cited by 223 · cited 1×
“For future work, it will be interesting to explore more sophisticated model [44], solve empirical challenges [45], deal with large-scale data [46] such as S&P 500 constituent stocks.”
Skulls, Financial Turbulence, and Risk Management
Kritzman, M., & Li, Y. (2010) · Financial Analysts Journal · doi:10.2469/faj.v66.n5.3 · Cited by 214 · cited 1×
“When turbulence t is higher than a threshold, which indicates extreme market conditions, we simply halt buying and the trading agent sells all shares.”
Jeong, G., & Kim, H. Y. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.09.036 · Cited by 204 · cited 1×
“The critic-only learning approach, which is the most common, solves a discrete action space problem using, for example, Deep Q-learning (DQN) and its improvements, and trains an agent on a single stock or asset [9, 12, 21].”
BackgroundTraining Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning
Wu, Y., & Tian, Y. (2017) · International Conference on Learning Representations · Cited by 136
Expected Returns: An Investor's Guide to Harvesting Market Rewards
Ilmanen, A. S. (2011) · Cited by 122
Revisiting the Performance of MACD and RSI Oscillators
Chong, T., Ng, W.-K., & Liew, V. (2014) · Journal of Risk and Financial Management · doi:10.3390/jrfm7010001 · Cited by 95 · cited 2×
“CCI compares current price to average price over a time window to indicate a buying or selling action [36].”
Application of deep reinforcement learning on automated stock trading
Chen, L., & Gao, Q. (2019) · 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess47205.2019.9040728 · Cited by 78 · cited 1×
“The critic-only learning approach, which is the most common, solves a discrete action space problem using, for example, Deep Q-learning (DQN) and its improvements, and trains an agent on a single stock or asset [21], [22], [23].”
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78 · cited 1×
“For future work, it will be interesting to explore more sophisticated model [42], solve empirical challenges [15], deal with large-scale data [7] such as S&P 500 constituent stocks.”
BackgroundBekiros, S. D. (2010) · Journal of Economic Dynamics and Control · doi:10.1016/j.jedc.2010.01.015 · Cited by 76 · cited 1×
“The actor-critic approach has been recently applied in finance [2, 26, 44, 48].”
MethodOptimal Asset Allocation using Adaptive Dynamic Programming
Neuneier, R. (1995) · Cited by 76
(2018) · arXiv (Cornell University) · Cited by 74
Self-citationReinforcement learning in financial markets - a surveyPreprint
Fischer, T. (2018) · RePEc: Research Papers in Economics · Cited by 69
Kim, Y., Ahn, W., Oh, K. J., & Enke, D. (2017) · Applied Soft Computing · doi:10.1016/j.asoc.2017.02.006 · Cited by 64 · cited 1×
“However, it is challenging for analysts to consider all relevant factors in a complex and dynamic stock market [1], [2], [3].”
Enhancing Q-Learning for Optimal Asset Allocation
Neuneier, R. (1997) · Cited by 57
Reinforcement Learning in Stock Trading
Dang, Q.-V. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-030-38364-0_28 · Cited by 57 · cited 1×
“The critic-only learning approach, which is the most common, solves a discrete action space problem using, for example, Deep Q-learning (DQN) and its improvements, and trains an agent on a single stock or asset [9, 12, 21].”
BackgroundDP-LSTM: Differential Privacy-inspired LSTM for Stock Prediction Using Financial NewsPreprint
Li, X., Li, Y., Yang, H., Yang, L., & Liu, X. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1912.10806 · Cited by 56 · cited 1×
“We can also explore more features for the state space such as adding advanced transaction cost and liquidity model [47], incorporating fundamental analysis indicators [9], natural language processing analysis of financial market news [48], and ESG features [12] to our observations.”
Self-citationFuzzy adaptive decision-making for boundedly rational traders in speculative stock markets
Bekiros, S. D. (2010) · European Journal of Operational Research · doi:10.1016/j.ejor.2009.04.015 · Cited by 53
Multi-Agent Deep Reinforcement Learning for Liquidation Strategy AnalysisPreprint
Bao, W., & Liu, X. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1906.11046 · Cited by 53 · cited 1×
“We can also explore more features for the state space such as adding advanced transaction cost and liquidity model [47], incorporating fundamental analysis indicators [9], natural language processing analysis of financial market news [48], and ESG features [12] to our observations.”
Self-citationBertoluzzo, F., & Corazza, M. (2012) · Procedia Economics and Finance · doi:10.1016/s2212-5671(12)00122-0 · Cited by 47 · cited 1×
“Another approach for stock trading is to model it as a Markov Decision Process (MDP) and use dynamic programming to derive the optimal strategy [4, 5, 34, 35].”
MethodA Practical Machine Learning Approach for Dynamic Stock Recommendation
Yang, H., Liu, X.-Y., & Wu, Q. (2018) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom/bigdatase.2018.00253 · Cited by 47
Self-citationOnline portfolio selection strategy based on combining experts' advice
Zhang, Y., & Yang, X. (2017) · Computational Economics · doi:10.1007/s10614-016-9585-0 · Cited by 38 · cited 1×
“However, it is challenging for analysts to consider all relevant factors in a complex and dynamic stock market [3, 23, 47].”
BackgroundGurrib, I. (2018) · Banks and Bank Systems · doi:10.21511/bbs.13(3).2018.06 · Cited by 28 · cited 1×
“2) Action Space: For a single stock, the action space is defined as {− k, ..., − 1 , 0 , 1 , ..., k } , where k and − k presents the number of shares we can buy and sell, and k ≤ h max while h max is a predefined parameter that sets as the maximum amount of shares for each buying action.”
Learning to Trade with Deep Actor Critic Methods
Li, J., Rao, R., & Shi, J. (2018) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid.2018.10116 · Cited by 27 · cited 1×
“Note that ”hold” may lead to different portfolio values due to the changing stock prices.”
Quantifying ESG alpha using scholar big data
Chen, Q., & Liu, X.-Y. (2020) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3383455.3422529 · Cited by 17 · cited 2×
“In recent years, machine learning and deep learning algorithms have been widely applied to build prediction and classification models for the financial market.”
Self-citationCommodity Channel Index: Evaluation of Trading Rule of Agricultural Commodities
Maitah, M., Procházka, P., Čermák, M., & Šrédl, K. (2016) · International Journal of Economics and Financial Issues · Cited by 16
Practical Machine Learning Approach to Capture the Scholar Data Driven Alpha in AI Industry
Fang, Y., Liu, X.-Y., & Yang, H. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006093 · Cited by 7 · cited 1×
“In recent years, machine learning and deep learning algorithms have been widely applied to build prediction and classification models for the financial market.”
Self-citationA Practical Machine Learning Approach for Dynamic Stock RecommendationPreprint
Yang, H., Liu, X.-Y., & Wu, Q. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3302088 · Cited by 7
Self-citation
Research works printed but not linked: 4
- Zhipeng Liang , Kangkang Jiang , Hao Chen , Junhao Zhu , and Yanran Li. 2018. Adversarial deep reinforcement learning in portfolio management. arXiv: Portfolio Management ( 2018 ). Zhipeng Liang, Kangkang Jiang, Hao Chen, Junhao Zhu, and Yanran Li. 2018. Adversarial deep reinforcement learning in portfolio management. arXiv: Portfolio Management (2018).
- John Schulman , Sergey Levine , Philipp Moritz , Michael Jordan , and Pieter Abbeel . 2015 . Trust region policy optimization . In The 31st International Conference on Machine Learning. John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. 2015. Trust region policy optimization. In The 31st International Conference on Machine Learning.
- 10.5555/551283 · doi:10.5555/551283
- Zihao Zhang . 2019. Deep reinforcement learning for trading. ArXiv 2019 (11 2019 ). Zihao Zhang. 2019. Deep reinforcement learning for trading. ArXiv 2019 (11 2019).
Cited by
Ensemble deep learning: A review
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105151 · Cited by 2,143
“2020 Deep reinforcement learning for automated stock trading: An ensemble strategy[171] Reinforcement learning Stock trading agency”
MethodDeep deterministic policy gradient algorithm: A systematic review
Sumiea, E. H., Abdulkadir, S. J., Alhussian, H. S., Al-Selwi, S. M., Alqushaibi, A., Ragab, M. G., & Fati, S. M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e30697 · Cited by 229
Sahu, S. K., Mokhade, A., & Bokde, N. D. (2023) · Applied Sciences · doi:10.3390/app13031956 · Cited by 208
Lim, T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10708-3 · Cited by 172
FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
Liu, X.-Y. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3737257 · Cited by 90
Self-citation
Metadata
- Authors
- Yang, H., Liu, X.-Y., Zhong, S., & Walid, A.
- Affiliations
- Columbia University (US)
- Wormpex AI Research
- Nokia-Bell Labs
- Venue
- Proceedings of the First ACM International Conference on AI in Finance
- Publisher
- ACM
- Published
- 2020-10-15
- Pages
- 1-8
- OpenAlex
- W3203680104
- Cited by
- 196
- References
- 47
- Citations by year
- 2026: 38 · 2025: 40 · 2024: 51 · 2023: 32 · 2022: 24 · 2021: 10 · 2020: 1
- FWCI
- 22.74
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on AI in Finance (ICAIF) (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
- Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
- Social Sciences › Economics, Econometrics and Finance › Finance › Stochastic processes and financial applications
- Keywords
- automated stock trading, deep reinforcement learning, actor-critic algorithms, proximal policy optimization, Advantage Actor-Critic, deep deterministic policy gradient, continuous action spaces, Sharpe ratio
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.