The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning
Open accessAbstract
Driven by the recent advances in electric vehicle (EV) technologies, EVs have become important for smart grid economy. When EVs participate in demand response program which has real-time pricing signals, the charging cost can be greatly reduced by taking full advantage of these pricing signals. However, it is challenging to determine an optimal charging strategy due to the existence of randomness in traffic conditions, user's commuting behavior, and the pricing process of the utility. Conventional model-based approaches require a model of forecast on the uncertainty and optimization for the scheduling process. In this paper, we formulate this scheduling problem as a Markov Decision Process (MDP) with unknown transition probability. A model-free approach based on deep reinforcement learning is proposed to determine the optimal strategy for this problem. The proposed approach can adaptively learn the transition probability and does not require any system model information. The architecture of the proposed approach contains two networks: a representation network to extract discriminative features from the electricity prices and a Q network to approximate the optimal action-value function. Numerous experimental results demonstrate the effectiveness of the proposed approach.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For load monitoring and electric vehicle charging scheduling, transfer learning [200] and deep reinforcement learning [215] methods are proposed.”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [215]
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
A Critical Review of Recurrent Neural Networks for Sequence LearningPreprint
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 3,159
Short-Term Residential Load Forecasting Based on LSTM Recurrent Neural Network
Kong, W., Dong, Z. Y., Jia, Y., Hill, D. J., Xu, Y. H., & Zhang, Y. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2753802 · Cited by 2,806
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_3 · Cited by 1,268
Tree-Based Batch Mode Reinforcement Learning
Ernst, D., Geurts, P., & Wehenkel, L. A. (2005) · doi:10.5555/1046920.1088690 · Cited by 854
Short-Term Residential Load Forecasting Based on Resident Behaviour Learning
Kong, W., Dong, Z. Y., Hill, D. J., Luo, F., & Xu, Y. H. (2017) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2017.2688178 · Cited by 616
A Menu of Designs for Reinforcement Learning Over Time
Werbos, P. J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/4939.003.0007 · Cited by 609
Optimal Bidding Strategy for Electric Vehicle Aggregators in Electricity Markets
Vagropoulos, S. I., & Bakirtzis, A. G. (2013) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2013.2274673 · Cited by 469
Erdinc, O., Paterakis, N. G., Mendes, T. D. P., Bakirtzis, A. G., & P. S. Catalao, J. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2352650 · Cited by 462
Paterakis, N. G., Erdinç, O., Bakirtzis, A. G., & Catalào, J. P. S. (2015) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2015.2438534 · Cited by 431
Integration of Vehicle-to-Grid in the Western Danish Power System
Pillai, J. R., & Bak‐Jensen, B. (2010) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2010.2072938 · Cited by 396
Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning
Ruelens, F., Claessens, B., Vandael, S., Schutter, B. D., Babuška, R., & Belmans, R. J. M. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2517211 · Cited by 361
Optimal Operation and Services Scheduling for an Electric Vehicle Battery Swapping Station
Sarker, M. R., Pandžić, H., & Ortega‐Vazquez, M. A. (2014) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2331560 · Cited by 359
Ortega‐Vazquez, M. A. (2014) · IET Generation Transmission & Distribution · doi:10.1049/iet-gtd.2013.0624 · Cited by 327
A Real-Time Charging Scheme for Demand Response in Electric Vehicle Parking Station
Yao, L., Lim, W. H., & Tsai, T. S. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2582749 · Cited by 303
Vayá, M. G., & Andersson, G. (2014) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2363159 · Cited by 294
Optimal Demand Response Using Device-Based Reinforcement Learning
Wen, Z., O’Neill, D. C., & Maei, H. R. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2396993 · Cited by 271
Two-Stage Energy Management for Office Buildings With Workplace EV Charging and Renewable Energy
Wu, D., Zeng, H., Lü, C., & Boulet, B. (2017) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2017.2659626 · Cited by 246
Two-Stage Economic Operation of Microgrid-Like Electric Vehicle Parking Deck
Guo, Y., Xiong, J., Xu, S., & Su, W. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2424912 · Cited by 230
Wu, H., Shahidehpour, M., Alabdulwahab, A., & Abusorrah, A. M. (2015) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2015.2498200 · Cited by 227
He, H., Ni, Z., & Fu, J. Y. (2011) · Neurocomputing · doi:10.1016/j.neucom.2011.05.031 · Cited by 221
Self-citationOptimal Scheduling for Electric Vehicle Charging With Discrete Charging Levels in Distribution Grid
Sun, B., Huang, Z., Tan, X., & Tsang, D. H. K. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2558585 · Cited by 217
Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market
Vandael, S., Claessens, B., Ernst, D., Holvoet, T., & Deconinck, G. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2393059 · Cited by 209
Reinforcement Learning-Based Plug-in Electric Vehicle Charging with Forecasted Price
Chis, A., Lundén, J., & Koivunen, V. (2016) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2016.2603536 · Cited by 180
An Online Learning Algorithm for Demand Response in Smart Grid
Bahrami, S., Wong, V. W. S., & Huang, J. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2667599 · Cited by 147
Risk-Based Day-Ahead Scheduling of Electric Vehicle Aggregator Using Information Gap Decision Theory
Zhao, J., Wan, C., Xu, Z., & Wang, J. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2494371 · Cited by 144
Matching EV Charging Load With Uncertain Wind: A Simulation-Based Policy Improvement Approach
Huang, Q., Jia, Q., Qiu, Z., Guan, X., & Deconinck, G. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2385711 · Cited by 143
Real-Time Pricing Mechanism for Electricity Market With Built-In Incentive for Participation
Namerikawa, T., Okubo, N., Sato, R., Okawa, Y., & Ono, M. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2447154 · Cited by 133
Self Scheduling of Plug-In Electric Vehicle Aggregator to Provide Balancing Services for Wind Power
Vayá, M. G., & Andersson, G. (2015) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2015.2498521 · Cited by 131
Kou, P., Liang, D., Gao, L., & Gao, F. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2475316 · Cited by 130
Dispatch of EV Charging Station Energy Resources for Sustainable Mobility
Liao, Y.-T., & Lu, C. (2015) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2015.2430287 · Cited by 106
Risk Averse Scheduling by a PEV Aggregator Under Uncertainty
Momber, I., Siddiqui, A. S., Román, T. G. S., & Söder, L. (2014) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2330375 · Cited by 104
Scalable Real-Time Electric Vehicles Charging With Discrete Charging Rates
Binetti, G., Davoudi, A. A., Naso, D., Turchiano, B., & Lewis, F. L. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2396772 · Cited by 98
Cost-Constrained Dynamic Optimal Electric Vehicle Charging
Maigha, M., & Crow, M. L. (2016) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2016.2615865 · Cited by 95
Optimal Resource Allocation and Charging Prices for Benefit Maximization in Smart PEV-Parking Lots
Awad, A. S. A., Shaaban, M. F., EL-Fouly, T. H. M., El-Saadany, E. F., & Salama, M. M. A. (2016) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2016.2617679 · Cited by 92
The importance of experience replay database composition in deep reinforcement learning
de Bruin, T., Kober, J., Tuyls, K., & Babuška, R. (2015) · neural information processing systems · Cited by 77
PDE Modeling and Control of Electric Vehicle Fleets for Ancillary Services: A Discrete Charging Case
Floch, C. L., Kara, E. C., & Moura, S. J. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2556643 · Cited by 71
Smart Grid Agent: Plug-in Electric Vehicle
Dallinger, D., Link, J., & Büttner, M. (2014) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2014.2298887 · Cited by 62
ADP: Goals, Opportunities and Principles
Si, J., Barto, A. G., Powell, W. B., & Wunsch, D. (2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544785.ch1 · Cited by 43
Flagged first, then by citations
Research works printed but not linked: 5
- bellman, 1958, Dynamic Programming
- 2018, Grid-Connected Renewable Energy Systems
- goodfellow, 2016, Deep Learning, 1
- 2017, The Egallon How Much Cheaper is it to Drive on Electricity?
- 2017, California Independent System Operator
Cited by
Sorted by citations
Das, H. S., Rahman, M. M., Li, S., & Tan, C. W. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2019.109618 · Cited by 1,140
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
“For load monitoring and electric vehicle charging scheduling, TL [200] and deep RL [215] methods are proposed.”
BackgroundMethodReinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review
Cao, D., Hu, W., Zhao, J., Zhang, G., Zhang, B., Liu, Z., Chen, Z., & Blaabjerg, F. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000552 · Cited by 460
“Reference [80] proposes a DQN-based approach for the optimization of EV charging while [81] extends this work by integrating the DQN algorithm with the long-short term memory (LSTM) NN.”
MethodConstrained EV Charging Scheduling Based on Safe Deep Reinforcement Learning
Li, H., Wan, Z., & He, H. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2955437 · Cited by 442
“If we maximize the lower bound to generate the policy π k + 1 when at the k th iteration, we are guaranteed that an improving policy can be obtained according to Wan et al. [16], i.e., Now, we can use the update rule in (38) to improve the policy π k to maximize the return J (π k ) .”
BackgroundMethodInfluentialSelf-citationA Survey of Recent Advances in Edge-Computing-Powered Artificial Intelligence of Things
Chang, Z., Liu, S., Xiong, X., Cai, Z., & Tu, G. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3088875 · Cited by 431
Metadata
- Authors
- Wan, Z., Li, H., He, H., & Prokhorov, D.
- Affiliations
- University of Rhode Island (US)
- Shenyang Institute of Automation (CN)
- Toyota Motor North America Research & Development (United States) (US)
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2019-09
- Volume
- 10
- Issue
- 5
- Pages
- 5246-5257
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W2899639849
- Cited by
- 548
- References
- 46
- Citations by year
- 2026: 61 · 2025: 71 · 2024: 83 · 2023: 95 · 2022: 87 · 2021: 74 · 2020: 57 · 2019: 20
- FWCI
- 14.43
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- IEEE Transactions on Smart Grid (publishedVersion)
- Journal of Media Literacy Education (submittedVersion, cc-by)
- SIA OpenIR (Chinese Academy of Sciences) (submittedVersion)
- Funders
- Office of Naval Research (N00014-18-1-2396)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Electric Vehicles and Infrastructure
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Smart Grid Energy Management
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Electric Power System Optimization
- Keywords
- electric vehicle charging scheduling, real-time pricing, demand response, deep reinforcement learning, model-free learning, unknown transition probabilities, Markov decision process, Q-network, electricity price forecasting, smart grid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
