CoChat

FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance

Open access
Liu, Xiao-Yang, Yang, Hongyang, Gao, Jiechao, Wang, Christina · 2021 · SSRN Electronic Journal

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 4 of the 25 sources its claims cite were checked for support.

Abstract

Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely to decide where to trade, at what price and what quantity, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework FinRL as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and extensibility under the key principles, full-stack framework, customization, reproducibility and hands-on tutoring. Embodied as a three-layer architecture with modular structures, FinRL implements fine-tuned state-of-the-art DRL algorithms and common reward functions, while alleviating the debugging work- loads. Thus, we help users pipeline the strategy design at a high turnover rate. At multiple levels of time granularity, FinRL simu- lates various markets as training environments using historical data and live trading APIs. Being highly extensible, FinRL reserves a set of user-import interfaces and incorporates trading constraints such as market friction, market liquidity and investor’s risk-aversion. Moreover, serving as practitioners’ stepping stones, typical trad- ing tasks are provided as step-by-step tutorials, e.g., stock trading, portfolio allocation, cryptocurrency trading, etc.

Overview

CoChat Summary

## TL;DR

FinRL is an open-source, three-layer deep reinforcement learning framework for automated trading in quantitative finance. The paper presents the framework, its training-testing-trading pipeline, and benchmark/tutorial use cases in stocks, portfolio allocation, and cryptocurrencies.

## Key findings

  • FinRL is described as the first open-source framework for end-to-end DRL-based trading in quantitative finance, with a three-layer architecture: application, agent, and environment.
  • The framework supports fine-tuned DRL libraries including Stable Baselines 3, RLlib, and ElegantRL, and integrates historical data plus live trading APIs into Gym-style environments.
  • In the stock-trading use case on Dow 30 constituent stocks, the ensemble strategy achieved an annualized return of 52.61%, Sharpe ratio of 2.81, and max drawdown of -7.09%; the DJIA benchmark had 32.84% annualized return and 2.02 Sharpe ratio.
  • In the portfolio-allocation use case on the same Dow 30 universe, A2C had the best reported performance with Sharpe ratio 2.36 and annual return 42.57%; TD3, PPO, and DDPG also outperformed the DJIA index and min-variance strategy.
  • In the cryptocurrency use case on the top 10 market-cap cryptocurrencies, PPO from ElegantRL reached the highest cumulative return of 103%, ahead of equally weighted (99%) and BTC buy-and-hold (93%).

## Methods

  • Framework/design paper introducing a modular DRL trading system with three layers: application layer for state-action-reward specification, agent layer for DRL algorithms, and environment layer for market simulators.
  • Uses historical market data and live trading APIs to build training/testing/trading workflows; supported tasks include stock trading, portfolio allocation, and cryptocurrency trading.
  • Evaluates the framework through backtesting on benchmark use cases, comparing DRL agents against baselines such as mean-variance, min-variance, passive buy-and-hold, and equally weighted portfolios, using metrics including final portfolio value, cumulative return, annualized return, annualized standard deviation, Sharpe ratio, and maximum drawdown.

## Limitations

  • The paper states that the development pipeline is error-prone and debugging is arduous for quantitative traders; FinRL is presented as a way to alleviate this, not as evidence of solving all implementation difficulties.
  • The text makes evident that the benchmark evidence is limited to reproduced examples and backtests on selected markets/tasks; this suggests limited external validity beyond the showcased use cases (text-evident).
  • The paper does not report a controlled ablation or statistical significance analysis of the framework components; this is not stated in the paper.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2139/ssrn.3955949

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 8 Not valid · 12 Not checked · 56 Own results and interpretations

12 more claims

Create a free account to see all 80 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 25
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 50

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 13

  1. Joshua Achiam, 2018, Spinning up in deep reinforcement learning
  2. Greg Brockman, 2016, OpenAI gym
  3. 2017, CSI, 300
  4. Prafulla Dhariwal, 2017, OpenAI baselines
  5. 2018, Shanghai Stock Exchange, SSE 180 Index Methodology
  6. G Thomas, 2018, Reinforcement learning in financial markets -a survey. FAU Discussion Papers in Economics
  7. 2020, Hang Seng index and sub-indexes
  8. Xiao-Yang Liu, 2021, ElegantRL: A scalable and elastic deep reinforcement learning library
  9. Xiao-Yang Liu, 2021, Data-driven deep reinforcement learning in quantitative finance
  10. Quantopian, 2019, Pyfolio: A toolkit for portfolio and risk analytics in Python
  11. Antonin Raffin, 2019, Stable Baselines3
  12. Jonathan Sadighian, 2019, Deep reinforcement learning in cryptocurrency market making. arXiv: Trading and Market Microstructure
  13. Nelson Vadori, 2020, Risksensitive reinforcement learning: a martingale approach to reward uncertainty. International Conference on AI in Finance (ICAIF)

Cited by

See all 46

Metadata

Authors
Liu, X.-Y., Yang, H., Gao, J., & Wang, C.
Affiliations
  • Columbia University (US)
  • University of Virginia (US)
  • New York University (US)
Corresponding author
Yang, Hongyang
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2021
ISSN
1556-5068 (ISSN-L)
OpenAlex
W3213398673
Cited by
46
References
43
Citations by year
2026: 16 · 2025: 7 · 2024: 6 · 2023: 9 · 2022: 3 · 2021: 3
Type
posted content
Language
en
Open access
Locations
  • SSRN Electronic Journal (acceptedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
  • Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
  • Social Sciences › Business, Management and Accounting › Management Information Systems › FinTech, Crowdfunding, Digital Finance
Keywords
deep reinforcement learning, algorithmic trading, quantitative trading, open source software framework, reward function, market simulation, trading constraints, asset allocation, cryptocurrency trading, stock trading, market frictions, liquidity, risk aversion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.