Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning
Open accessAbstract
Driven by the recent advances in electric vehicle (EV) technologies, EVs have become important for smart grid economy. When EVs participate in demand response program which has real-time pricing signals, the charging cost can be greatly reduced by taking full advantage of these pricing signals. However, it is challenging to determine an optimal charging strategy due to the existence of randomness in traffic conditions, user's commuting behavior, and the pricing process of the utility. Conventional model-based approaches require a model of forecast on the uncertainty and optimization for the scheduling process. In this paper, we formulate this scheduling problem as a Markov Decision Process (MDP) with unknown transition probability. A model-free approach based on deep reinforcement learning is proposed to determine the optimal strategy for this problem. The proposed approach can adaptively learn the transition probability and does not require any system model information. The architecture of the proposed approach contains two networks: a representation network to extract discriminative features from the electricity prices and a Q network to approximate the optimal action-value function. Numerous experimental results demonstrate the effectiveness of the proposed approach.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transfer learning and deep reinforcement learning methods are proposed for load monitoring and electric vehicle charging scheduling.
“For load monitoring and electric vehicle charging scheduling, transfer learning [200] and deep reinforcement learning [215] methods are proposed.”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [215]
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · see citing works
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · see citing works
A Critical Review of Recurrent Neural Networks for Sequence LearningPreprint
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 2,711 · see citing works
Short-Term Residential Load Forecasting Based on LSTM Recurrent Neural Network
Kong, W., Dong, Z. Y., Jia, Y., Hill, D. J., Xu, Y. H., & Zhang, Y. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2753802 · Cited by 2,678 · see citing works
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · see citing works
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333 · see citing works
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_3 · Cited by 1,268 · see citing works
Tree-Based Batch Mode Reinforcement Learning
Ernst, D., Geurts, P., & Wehenkel, L. A. (2005) · doi:10.5555/1046920.1088690 · Cited by 853 · see citing works
Short-Term Residential Load Forecasting Based on Resident Behaviour Learning
Kong, W., Dong, Z. Y., Hill, D. J., Luo, F., & Xu, Y. H. (2017) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2017.2688178 · Cited by 616 · see citing works
A Menu of Designs for Reinforcement Learning Over Time
Werbos, P. J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/4939.003.0007 · Cited by 609 · see citing works
Erdinc, O., Paterakis, N. G., Mendes, T. D. P., Bakirtzis, A. G., & P. S. Catalao, J. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2352650 · Cited by 462 · see citing works
Optimal Bidding Strategy for Electric Vehicle Aggregators in Electricity Markets
Vagropoulos, S. I., & Bakirtzis, A. G. (2013) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2013.2274673 · Cited by 455 · see citing works
Paterakis, N. G., Erdinç, O., Bakirtzis, A. G., & Catalào, J. P. S. (2015) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2015.2438534 · Cited by 431 · see citing works
Integration of Vehicle-to-Grid in the Western Danish Power System
Pillai, J. R., & Bak‐Jensen, B. (2010) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2010.2072938 · Cited by 389 · see citing works
Optimal Operation and Services Scheduling for an Electric Vehicle Battery Swapping Station
Sarker, M. R., Pandžić, H., & Ortega‐Vazquez, M. A. (2014) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2331560 · Cited by 359 · see citing works
Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning
Ruelens, F., Claessens, B., Vandael, S., Schutter, B. D., Babuška, R., & Belmans, R. J. M. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2517211 · Cited by 339 · see citing works
Ortega‐Vazquez, M. A. (2014) · IET Generation Transmission & Distribution · doi:10.1049/iet-gtd.2013.0624 · Cited by 327 · see citing works
A Real-Time Charging Scheme for Demand Response in Electric Vehicle Parking Station
Yao, L., Lim, W. H., & Tsai, T. S. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2582749 · Cited by 303 · see citing works
Vayá, M. G., & Andersson, G. (2014) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2363159 · Cited by 294 · see citing works
Optimal Demand Response Using Device-Based Reinforcement Learning
Wen, Z., O’Neill, D. C., & Maei, H. R. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2396993 · Cited by 271 · see citing works
Two-Stage Energy Management for Office Buildings With Workplace EV Charging and Renewable Energy
Wu, D., Zeng, H., Lü, C., & Boulet, B. (2017) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2017.2659626 · Cited by 246 · see citing works
Two-Stage Economic Operation of Microgrid-Like Electric Vehicle Parking Deck
Guo, Y., Xiong, J., Xu, S., & Su, W. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2424912 · Cited by 230 · see citing works
Wu, H., Shahidehpour, M., Alabdulwahab, A., & Abusorrah, A. M. (2015) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2015.2498200 · Cited by 227 · see citing works
He, H., Ni, Z., & Fu, J. Y. (2011) · Neurocomputing · doi:10.1016/j.neucom.2011.05.031 · Cited by 221 · see citing works
Self-citationOptimal Scheduling for Electric Vehicle Charging With Discrete Charging Levels in Distribution Grid
Sun, B., Huang, Z., Tan, X., & Tsang, D. H. K. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2558585 · Cited by 217 · see citing works
Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market
Vandael, S., Claessens, B., Ernst, D., Holvoet, T., & Deconinck, G. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2393059 · Cited by 209 · see citing works
Reinforcement Learning-Based Plug-in Electric Vehicle Charging with Forecasted Price
Chis, A., Lundén, J., & Koivunen, V. (2016) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2016.2603536 · Cited by 180 · see citing works
An Online Learning Algorithm for Demand Response in Smart Grid
Bahrami, S., Wong, V. W. S., & Huang, J. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2667599 · Cited by 147 · see citing works
Risk-Based Day-Ahead Scheduling of Electric Vehicle Aggregator Using Information Gap Decision Theory
Zhao, J., Wan, C., Xu, Z., & Wang, J. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2494371 · Cited by 144 · see citing works
Matching EV Charging Load With Uncertain Wind: A Simulation-Based Policy Improvement Approach
Huang, Q., Jia, Q., Qiu, Z., Guan, X., & Deconinck, G. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2385711 · Cited by 143 · see citing works
Real-Time Pricing Mechanism for Electricity Market With Built-In Incentive for Participation
Namerikawa, T., Okubo, N., Sato, R., Okawa, Y., & Ono, M. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2447154 · Cited by 133 · see citing works
Self Scheduling of Plug-In Electric Vehicle Aggregator to Provide Balancing Services for Wind Power
Vayá, M. G., & Andersson, G. (2015) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2015.2498521 · Cited by 131 · see citing works
Kou, P., Liang, D., Gao, L., & Gao, F. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2475316 · Cited by 130 · see citing works
Dispatch of EV Charging Station Energy Resources for Sustainable Mobility
Liao, Y.-T., & Lu, C. (2015) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2015.2430287 · Cited by 105 · see citing works
Risk Averse Scheduling by a PEV Aggregator Under Uncertainty
Momber, I., Siddiqui, A. S., Román, T. G. S., & Söder, L. (2014) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2330375 · Cited by 104 · see citing works
Scalable Real-Time Electric Vehicles Charging With Discrete Charging Rates
Binetti, G., Davoudi, A. A., Naso, D., Turchiano, B., & Lewis, F. L. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2396772 · Cited by 98 · see citing works
Cost-Constrained Dynamic Optimal Electric Vehicle Charging
Maigha, M., & Crow, M. L. (2016) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2016.2615865 · Cited by 95 · see citing works
Optimal Resource Allocation and Charging Prices for Benefit Maximization in Smart PEV-Parking Lots
Awad, A. S. A., Shaaban, M. F., EL-Fouly, T. H. M., El-Saadany, E. F., & Salama, M. M. A. (2016) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2016.2617679 · Cited by 92 · see citing works
The importance of experience replay database composition in deep reinforcement learning
de Bruin, T., Kober, J., Tuyls, K., & Babuška, R. (2015) · neural information processing systems · Cited by 77 · see citing works
PDE Modeling and Control of Electric Vehicle Fleets for Ancillary Services: A Discrete Charging Case
Floch, C. L., Kara, E. C., & Moura, S. J. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2556643 · Cited by 71 · see citing works
Smart Grid Agent: Plug-in Electric Vehicle
Dallinger, D., Link, J., & Büttner, M. (2014) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2014.2298887 · Cited by 62 · see citing works
ADP: Goals, Opportunities and Principles
Si, J., Barto, A. G., Powell, W. B., & Wunsch, D. (2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544785.ch1 · Cited by 43 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Das, H. S., Rahman, M. M., Li, S., & Tan, C. W. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2019.109618 · Cited by 1,140 · see citing works
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
“For load monitoring and electric vehicle charging scheduling, TL [200] and deep RL [215] methods are proposed.”
BackgroundMethodReinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review
Cao, D., Hu, W., Zhao, J., Zhang, G., Zhang, B., Liu, Z., Chen, Z., & Blaabjerg, F. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000552 · Cited by 460 · see citing works
“Reference [80] proposes a DQN-based approach for the optimization of EV charging while [81] extends this work by integrating the DQN algorithm with the long-short term memory (LSTM) NN.”
MethodConstrained EV Charging Scheduling Based on Safe Deep Reinforcement Learning
Li, H., Wan, Z., & He, H. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2955437 · Cited by 434 · see citing works
“If we maximize the lower bound to generate the policy π k + 1 when at the k th iteration, we are guaranteed that an improving policy can be obtained according to Wan et al. [16], i.e., Now, we can use the update rule in (38) to improve the policy π k to maximize the return J (π k ) .”
BackgroundMethodInfluentialSelf-citationA Survey of Recent Advances in Edge-Computing-Powered Artificial Intelligence of Things
Chang, Z., Liu, S., Xiong, X., Cai, Z., & Tu, G. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3088875 · Cited by 431 · see citing works
Chen, X., Qu, G., Tang, Y., Low, S. H., & Li, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3154718 · Cited by 440 · see citing works
“In [121]–[123], various RL techniques are studied to design EV charging policies to deal with the randomness in the arrival and departure time of an EV.”
BackgroundA Multi-Agent Reinforcement Learning-Based Data-Driven Method for Home Energy Management
Xu, X., Jia, Y., Xu, Y., Xu, Z., Chai, S., & Lai, C. S. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2971427 · Cited by 420 · see citing works
“[14], Wan et al. use deep RL algorithms to determine the optimal solutions of the EV charging/discharging scheduling problem.”
Du, Y., & Li, F. F. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2930299 · Cited by 396 · see citing works
“The authors in [23] proposed the application of deep RL to optimize the real-time electric vehicle charging schedule with the consideration of future electricity price.”
BackgroundCao, J. H., Harrold, D. J. B., Fan, Z., Morstyn, T., Healey, D., & Li, K. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2986333 · Cited by 340 · see citing works
“The authors of [14] proposed a deep reinforcement learning based approach to optimize the EV charging scheduling.”
BackgroundDeep Reinforcement Learning for Strategic Bidding in Electricity Markets
Ye, Y., Qiu, D., Sun, M., Papadaskalopoulos, D. A., & Štrbac, G. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2936142 · Cited by 298 · see citing works
Ganesh, A. H., & Xu, B. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111833 · Cited by 290 · see citing works
Machine Learning Approaches for EV Charging Behavior: A Review
Shahriar, S., Al-Ali, A. R., Osman, A. H., Dhou, S., & Nijim, M. (2020) · IEEE Access · doi:10.1109/access.2020.3023388 · Cited by 280 · see citing works
“There are a few recent works utilizing RL-based concepts for scheduling of EV charging [93]–[96] and therefore has great potential for further research.”
BackgroundQian, T., Shao, C., Wang, X., & Shahidehpour, M. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2942593 · Cited by 265 · see citing works
“Sampling in replay buffer breaks the temporal correlations between the selected tuples [26], making the training more stable.”
BackgroundDeep learning for edge computing applications: A State-of-the-Art survey
Wang, F., Zhang, M., Wang, X., Ma, X., & Liu, J. (2020) · IEEE Access · doi:10.1109/access.2020.2982411 · Cited by 259 · see citing works
“For instance, Wan et al. [104] jointly considered the electricity price and battery energy of electric vehicles and proposed a model-free DRL based model to automatically determine the charging policy.”
BackgroundDeep reinforcement learning for power system: An overview
Zhang, Z., Zhang, D., & Qiu, R. C. (2019) · CSEE Journal of Power and Energy Systems · doi:10.17775/cseejpes.2019.00920 · Cited by 276 · see citing works
Ye, Y., Qiu, D., Wu, X., Štrbac, G., & Ward, J. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2976771 · Cited by 246 · see citing works
“In this respect, the deep Q network (DQN) method is applied to optimize energy usage schedules of shiftable loads [36], [37], EV [33], [38], an integrated renewable energy and EES system [39]–[41], and a heating, ventilation and air conditioning system [42].”
MethodHu, H., Jia, X., He, Q., Fu, S., & Liu, K. (2020) · Computers & Industrial Engineering · doi:10.1016/j.cie.2020.106749 · Cited by 243 · see citing works
Reinforcement Learning Based EV Charging Management Systems–A Review
Abdullah, H. M., Gastli, A., & Ben‐Brahim, L. (2021) · IEEE Access · doi:10.1109/access.2021.3064354 · Cited by 249 · see citing works
“[75], [77], [79]–[84], [86], [88], [90]–[92], [94], [95], and [97]–[99], used the charging or discharging cost as a state parameter.”
BackgroundMethodInfluentialReinforcement learning in sustainable energy and electric systems: a survey
Yang, T., Zhao, L., Li, W., & Zomaya, A. Y. (2020) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2020.03.001 · Cited by 239 · see citing works
Yang, T., Zhao, L., Li, W., & Zomaya, A. Y. (2021) · Energy · doi:10.1016/j.energy.2021.121377 · Cited by 224 · see citing works
Hu, D., Ye, Z., Gao, Y., Ye, Z., Peng, Y., & Yu, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3185975 · Cited by 216 · see citing works
“A model-free approach is proposed to schedule the home charging of EVs considering the timevarying electricity price in distribution systems [9].”
MethodReal-Time Residential Demand Response
Li, H., Wan, Z., & He, H. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2978061 · Cited by 206 · see citing works
“…Anxiety: The range anxiety measures the resi-dent’s fear that the EV has insufficient energy to reach its destination, which is measured by, where E EV t − E EVmax represents the uncharged battery energy when the EV departs at t EV β , and the squared term measures the range anxiety in $/kWh 2 [22].”
BackgroundMethodSelf-citationQian, T., Shao, C., Li, X., Wang, X., & Shahidehpour, M. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2969650 · Cited by 205 · see citing works
“problem with uncertain electricity prices [20] and a robot motion planning problem aimed at moving through a bottleneck exit [21] are addressed by deep Q network (DQN) [15].”
BackgroundShin, M., Choi, D., & Kim, J. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2944183 · Cited by 201 · see citing works
Multi-agent deep reinforcement learning approach for EV charging scheduling in a smart grid
Park, K., & Moon, I. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.120111 · Cited by 193 · see citing works
“However, using the data generated from the above situations can more practically approach the situation that occurs in a more realistic environment [5].”
BackgroundMethodInfluentialYan, L., Chen, X., Zhou, J., Chen, Y., & Wen, J. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3098298 · Cited by 191 · see citing works
Lu, R., Li, Y., Li, Y., Jiang, J., & Ding, Y. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115473 · Cited by 185 · see citing works
CDDPG: A Deep-Reinforcement-Learning-Based Approach for Electric Vehicle Charging Control
Zhang, F., Yang, Q., & An, D. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3015204 · Cited by 179 · see citing works
“According to [33], we assume that the charging and discharging price at the same time are the same.”
BackgroundMethodInfluentialReinforcement learning for electric vehicle applications in power systems:A critical review
Qiu, D., Wang, Y., Hua, W., & Štrbac, G. (2022) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2022.113052 · Cited by 193 · see citing works
Smart and Resilient EV Charging in SDN-Enhanced Vehicular Edge Computing Networks
Liu, J., Guo, H., Xiong, J., Kato, N., Zhang, J., & Zhang, Y. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2951966 · Cited by 177 · see citing works
“[34] formulated the real-time EV charging/ discharging scheduling problem as a finite Markov decision process (MDP) with unknown transition probability, and proposed a model-free approach based on DRL to determine an optimal strategy for charging and discharging.”
MethodArtificial intelligence techniques for enabling Big Data services in distribution networks: A review
Barja-Martínez, S., Aragüés‐Peñalba, M., Munné‐Collado, Í., Lloret-Gallego, P., Bullich‐Massagué, E., & Villafáfila‐Robles, R. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111459 · Cited by 187 · see citing works
Effective Charging Planning Based on Deep Reinforcement Learning for Electric Vehicles
Zhang, C., Liu, Y., Wu, F., Tang, B., & Fan, W. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3002271 · Cited by 168 · see citing works
“Because the existence of randomness in traffic conditions, users’ commuting behaviors, and the pricing process of the utility, the literatures [19] proposes a model-free approach to determine an optimal charging strategy for EVs charging.”
BackgroundQiu, D., Ye, Y., Papadaskalopoulos, D. A., & Štrbac, G. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.116940 · Cited by 164 · see citing works
Liang, Y., Ding, Z., Ding, T., & Lee, W. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3025082 · Cited by 161 · see citing works
“1 2 ) [19].”
BackgroundAgent-Based Modeling in Electricity Market Using Deep Deterministic Policy Gradient Algorithm
Liang, Y., Guo, C., Ding, Z., & Hua, H. (2020) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2020.2999536 · Cited by 159 · see citing works
“…has been applied to model-free optimization and control in complex environments of power systems, such as electric vehicle charging scheduling [16], online building energy optimization [17], microgrid energy management [18], short-term voltage con-trol [19], and adaptive power system…”
MethodYe, Y., Tang, Y. P., Wang, H., Zhang, X., & Štrbac, G. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3103917 · Cited by 156 · see citing works
Learning to Operate an Electric Vehicle Charging Station Considering Vehicle-Grid Integration
Ye, Z., Gao, Y., & Yu, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3165479 · Cited by 144 · see citing works
Electric Vehicle Charging Management Based on Deep Reinforcement Learning
Li, S., Hu, W., Cao, D., Dragičević, T., Huang, Q. P., Chen, Z., & Blaabjerg, F. (2022) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000460 · Cited by 144 · see citing works
“In recent years, different neural network (NN) based meth‐ ods have been applied to the research of EV [12]-[15].”
MethodDorokhova, M., Martinson, Y., Ballif, C., & Wyrsch, N. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.117504 · Cited by 143 · see citing works
Deep Reinforcement Learning for Demand Response in Distribution Networks
Bahrami, S., Chen, Y. C., & Wong, V. W. S. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3037066 · Cited by 141 · see citing works
“Wan et al. [23] considered a long short-term memory (LSTM) network to extract informative features from the price rates and developed a DRL-based online charging scheduling algorithm for electric vehicles.”
BackgroundCyber-Attack Recovery Strategy for Smart Grid Based on Deep Reinforcement Learning
Wei, F., Wan, Z., & He, H. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2956161 · Cited by 137 · see citing works
“As to the power grid, [25], [26] present deep RL methods to determine the optimal strategy for real-time EV charging scheduling.”
MethodSelf-citationMulti-Agent Deep Reinforcement Learning Method for EV Charging Station Game
Qian, T., Shao, C., Li, X., Wang, X., Chen, Z., & Shahidehpour, M. (2021) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2021.3111014 · Cited by 135 · see citing works
Dynamic Pricing for EV Charging Stations: A Deep Reinforcement Learning Approach
Zhao, Z., & Lee, C. K. M. (2021) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2021.3139674 · Cited by 135 · see citing works
“Consequently, it is difficult to mathematically model the customers’ charging demand response in this context [35].”
BackgroundΤσαούσογλου, Γ., Giraldo, J. S., & Paterakis, N. G. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111890 · Cited by 133 · see citing works
“In [84], a penalty term is designed to teach a Neural Network to respect the local constraints of the DERs.”
MethodSafe reinforcement learning for real-time automatic control in a smart energy-hub
Qiu, D., Dong, Z., Zhang, X., Wang, Y., & Štrbac, G. (2022) · Applied Energy · doi:10.1016/j.apenergy.2021.118403 · Cited by 128 · see citing works
Zhang, Y., Yang, Q., An, D., Li, D., & Wu, Z. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3165074 · Cited by 129 · see citing works
“In recent years, the RL methods have been used to develop energy scheduling strategies [24]–[28].”
BackgroundMethodA Deep Reinforcement Learning Method for Pricing Electric Vehicles With Discrete Charging Levels
Qiu, D., Ye, Y., Papadaskalopoulos, D. A., & Štrbac, G. (2020) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2020.2984614 · Cited by 127 · see citing works
“Numerous recent papers have applied the DQN method to different electricity system problems [29]–[31], although its application in pricing design problems has yet to be explored.”
MethodDistributed Deep Reinforcement Learning for Intelligent Load Scheduling in Residential Smart Grids
Chung, H.-M., Maharjan, S., Zhang, Y., & Eliassen, F. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3007167 · Cited by 120 · see citing works
“To further address the discretization issue associated with Q-learning [18]–[23], DQN [24]–[26], and policy gradient [27], [28], the employed model-free approach is similar to model-based methods that provide a continuous value as the solution without discretizing the action space.”
MethodJahangir, H., Gougheri, S. S., Vatandoust, B., Golkar, M. A., Ahmadian, A., & Hajizadeh, A. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2998072 · Cited by 121 · see citing works
“Each aggre-gator needs to devise plans that are lucrative for PEV owners and at the same time ensure its profit margin in competition with other aggregators [2].”
BackgroundMethodJin, J., & Xu, Y. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3028470 · Cited by 118 · see citing works
“We further consider the rolling model predictive control (MPC) method [22], [24] as a benchmark.”
BackgroundMethodWang, Z., Ikechukwu, O. M., Huang, H., Qiu, C., Hisada, M., & Abdallah, A. B. (2020) · IEEE Access · doi:10.1109/access.2020.3044612 · Cited by 114 · see citing works
“Variants of intelligent energy management methods have also been proposed based on RL [35]–[37], [51], [52] and recurrent neural network (RNN) [31], [32].”
BackgroundMethodZhang, G., Hu, W., Cao, D., Huang, Q. P., Yi, J., Chen, Z., & Blaabjerg, F. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2997790 · Cited by 113 · see citing works
“Since equation (26) satisfies the recursive relationship, it can be transformed into the Bellman expectation equation [25]:”
BackgroundCoordinated Electric Vehicle Active and Reactive Power Control for Active Distribution Networks
Wang, Y., Qiu, D., Štrbac, G., & Gao, Z. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3169975 · Cited by 113 · see citing works
“In SARL, deep Q-network (DQN) has been widely applied to solve various EV problems, e.g., realtime charging/discharging strategies [14], optimal EV charging navigation [15], and fast frequency regulation services [16].”
BackgroundDesign possibilities and challenges of DNN models: a review on the perspective of end devices
Hussain, H., Tamizharasan, P. S., & Rahul, C. S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10138-z · Cited by 113 · see citing works
“Another application that considers the battery energy and power consumption of vehicles to perform an automated charging policy (Wan et al. 2019).”
BackgroundChen, Q., & Folly, K. A. (2022) · Energies · doi:10.3390/en16010146 · Cited by 116 · see citing works
Gao, Y., Yang, J., Yang, M. Y., & Li, Z. (2020) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2020.2986412 · Cited by 107 · see citing works
A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning
Zhou, Y., Zhang, B., Xu, C., Lan, T., Diao, R., Shi, D., Wang, Z., & Lee, W. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000522 · Cited by 106 · see citing works
“In [27], the DRL is used to solve the EV charging scheduling prob‐ lem for real-time control, which can adaptively learn the transition probability and does not require any system model information.”
MethodPrivacy Preserving Load Control of Residential Microgrid via Deep Reinforcement Learning
Qin, Z., Liu, D., Hua, H., & Cao, J. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3088290 · Cited by 105 · see citing works
“The deep Q-network (DQN) is used for EV charging scheduling in [21], while dueling DQN is applied to optimize the demand response management of interruptible load in [22].”
MethodYan, L., Chen, X., Chen, Y., & Wen, J. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3152218 · Cited by 105 · see citing works
“A deep-Q-network-based learning algorithm is proposed to schedule the charging process of the individual EV in [16].”
BackgroundMethodDu, Y., Li, F. F., Zandi, H., & Xue, Y. S. (2021) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000502 · Cited by 102 · see citing works
“In power systems, the potentials of implementing deep RL for demand-side energy management and electric vehicle charg ‐ ing/discharging scheduling are shown in [14], [15].”
BackgroundFang, X., Zhao, Q., Wang, J., Han, Y., & Li, Y. (2021) · Sustainable Cities and Society · doi:10.1016/j.scs.2021.103163 · Cited by 98 · see citing works
Hedayatnia, A., Ghafourian, J., Sepehrzad, R., Al-Durrad, A., & Anvari‐Moghaddam, A. (2024) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2024.110142 · Cited by 96 · see citing works
Xu, H., Chen, Y., Wang, H., Wang, H., Meng, Y., & Gu, Q. (2022) · Sustainable Cities and Society · doi:10.1016/j.scs.2022.104345 · Cited by 94 · see citing works
Guo, C., Wang, X., Zheng, Y., & Zhang, F. (2021) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107048 · Cited by 93 · see citing works
Deep Reinforcement Learning-Based Demand Response for Smart Facilities Energy Management
Lu, R., Bai, R., Luo, Z., Jiang, J., Sun, M., & Zhang, H. (2021) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2021.3104596 · Cited by 92 · see citing works
Fescioglu-Unver, N., & Aktaş, M. Y. (2023) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2023.113873 · Cited by 92 · see citing works
Wang, X., Wang, J., & Liu, J. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2993858 · Cited by 91 · see citing works
“Inrecent years, deep reinforcement learning (DRL) ap-proacheshavebeensuccessfullyadoptedinEVoptimalchargingscheduling[26]–[28]notonlybecauseitcanhandlethenon-convex relations between the EVs and the electricity market and always guarantee a feasible solution but also because of its real-time…”
BackgroundIntegrating Battery Aging in the Optimization for Bidirectional Charging of Electric Vehicles
Schwenk, K., Meisenbacher, S., Briegel, B., Harr, T., Hagenmeyer, V., & Mikut, R. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3099206 · Cited by 91 · see citing works
“, model-based optimization [16], [22] or model-free reinforcement learning [9].”
MethodGorostiza, F. S., & Gonzalez-longatt, F. M. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2996274 · Cited by 90 · see citing works
“In [17], a DQN agent was trained to find the optimal charging schedule for an EV considering battery SOC, range anxiety, and charging costs in the formulation of the rewards.”
Norouzi, M., Aghaei, J., Niknam, T., Alipour, M., Pirouzi, S., & Lehtonen, M. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121573 · Cited by 90 · see citing works
A Review of Denial of Service Attack and Mitigation in the Smart Grid Using Reinforcement Learning
Ortega-Fernandez, I., & Liberati, F. (2023) · Energies · doi:10.3390/en16020635 · Cited by 89 · see citing works
“For example, implementing a deep RL algorithm to learn the optimal strategy for real-time electric vehicle charging scheduling [46,47], or to use of a model-free RL method for load frequency control resistant to weather uncertainties [48].”
MethodEntropy-based Sampling Approaches for Multi-Class Imbalanced Problems
Li, L., He, H., & Li, J. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2913859 · Cited by 88 · see citing works
“Oversampling and undersampling techniques are commonly used to imbalanced learning [5].”
BackgroundSelf-citationGoh, H. H., Huang, Y., Lim, C. S., Zhang, D., Liu, H., Dai, W., Kurniawan, T. A., & Rahman, S. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3179567 · Cited by 87 · see citing works
“The BRF is commonly defined as follows [8], [22], [37]–[39]: is the power equation for the self-supply of the microgrid using distributed energy.”
BackgroundDolatabadi, A., Abdeltawab, H. H., & Mohamed, Y. A. I. (2022) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2022.3212938 · Cited by 87 · see citing works
“Intheproposed approach, since it is entirely data-dependent, all hyperparameters are optimally tailored using the training data following the recommendations of the deep learning community [55], while on the other hand, the scheme for setting the parameters presented in [34] is referred to.”
BackgroundCoordinated Topology Attacks in Smart Grid Using Deep Reinforcement Learning
Wang, Z., He, H., Wan, Z., & Sun, Y. L. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2994977 · Cited by 86 · see citing works
“There is no need to construct a model for the power system, and it can be easily applied to different power systems [17], [18].”
BackgroundMethodSelf-citationZhang, W., Chen, Q., Yan, J., Zhang, S., & Xu, J. (2021) · Energy · doi:10.1016/j.energy.2021.121492 · Cited by 86 · see citing works
Zhao, Z., Lee, C. K. M., & Ren, J. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.122278 · Cited by 86 · see citing works
Demand Response Management for Industrial Facilities: A Deep Reinforcement Learning Approach
Huang, X., Hong, S. H., Yu, M., Ding, Y., & Jiang, J. (2019) · IEEE Access · doi:10.1109/access.2019.2924030 · Cited by 84 · see citing works
“In [28], Wan et al.”
Xu, P., Zhang, J., Gao, T., Chen, S., Wang, X., Jiang, H., & Gao, W. (2022) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2022.108030 · Cited by 82 · see citing works
Kim, S. H., & Shin, Y. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.109144 · Cited by 82 · see citing works
Wu, T., Wang, J., Lu, X., & Du, Y. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.118189 · Cited by 79 · see citing works
Cao, D., Hu, W., Xu, X., Dragičević, T., Huang, Q. P., Liu, Z., Chen, Z., & Blaabjerg, F. (2019) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2019.105648 · Cited by 79 · see citing works
Hua, H., Wei, Z., Qin, Y., Wang, T., Li, L., & Cao, J. (2021) · IET Cyber-Physical Systems Theory & Applications · doi:10.1049/cps2.12007 · Cited by 77 · see citing works
“[109] describe the real‐time charging scheduling of electric vehicles as a Markov decision process with unknown transition probabilities, and propose a model‐free optimal scheduling method using DRL to obtain charge and discharge scheduling.”
MethodShariatzadeh, M., Lopes, M. A. R., & Antunes, C. H. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126167 · Cited by 74 · see citing works
Reinforcement learning for electric vehicle charging scheduling: A systematic review
Zhao, Z., Lee, C. K. M., Yan, X., & Wang, H. (2024) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2024.103698 · Cited by 81 · see citing works
Kim, H. J., & Kim, M.-K. (2024) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2024.3368419 · Cited by 75 · see citing works
“Statistical models primarily include Monte Carlo simulations [3], fuzzy linear regression [4], quantile linear regression [5], autoregressive integral moving averaging (ARIMA) [6], and the Kalman filter method [7].”
MethodZheng, Y., Yan, Z., Chen, K., Sun, J., Xu, Y. H., & Liu, Y. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3062700 · Cited by 70 · see citing works
“DRL has been shown as an effective solution for many problems in power systems [11]–[20], and the performance of DRL gives credit to the ability of DNNs.”
BackgroundLee, J., Lee, E., & Kim, J. (2020) · Energies · doi:10.3390/en13081950 · Cited by 69 · see citing works
Liu, D., Zeng, P., Cui, S., & Song, C. (2023) · Sensors · doi:10.3390/s23031618 · Cited by 68 · see citing works
Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling
Fang, X., Wang, J., Song, G., Han, Y., Zhao, Q., & Cao, Z. (2019) · Energies · doi:10.3390/en13010123 · Cited by 67 · see citing works
“Considering the integration of EVs and RGs, residential MES becomes more complicated due to high uncertainties from EVs and RGs, such as RGs output and EVs’ usage attributes [8,9].”
BackgroundJin, J., Mao, S., & Xu, Y. (2023) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2023.3250505 · Cited by 68 · see citing works
“…based techniques to solve the formulated MDP. Recently, deep reinforcement learning (DRL) based approaches have been adopted for the scheduling of EV charging and cloud computing demands under random task arrivals, renewable generation, and electricity prices [18], [19], [20], [21], [22].”
BackgroundMethodA Critical Review on Charging Technologies of Electric Vehicles
Shahjalal, M., Shams, T., Tasnim, M. N., Ahmed, M. R., Ahsan, M., & Haider, J. (2022) · Energies · doi:10.3390/en15218239 · Cited by 66 · see citing works
On data-driven modeling and control in modern power grids stability: Survey and perspective
Gong, X., Wang, X., & Cao, B. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121740 · Cited by 67 · see citing works
A Deep RL-Based Algorithm for Coordinated Charging of Electric Vehicles
Zhang, Z., Wan, Y., Qin, J., Fu, W., & Kang, Y. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3170000 · Cited by 63 · see citing works
“…input of our Q network contains the characteristics of the future price trend extracted from the long and short-term memory (LSTM) network, not only improving the fitting performance of the linear function approximation method in [13], [18], but also being applicable to various pricing mechanisms.”
BackgroundMethodInfluentialFeng, B., Liu, Z., Huang, G., & Guo, C. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121615 · Cited by 63 · see citing works
Deep Reinforcement Learning Based Optimal Route and Charging Station Selection
Lee, K.-B., Ahmed, M. A., Kang, D.-K., & Kim, Y.-C. (2020) · Energies · doi:10.3390/en13236255 · Cited by 62 · see citing works
Federated Reinforcement Learning for Electric Vehicles Charging Control on Distribution Networks
Qian, J., Jiang, Y., Liu, X., Wang, Q., Wang, T., Shi, Y., & Chen, W. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3306826 · Cited by 65 · see citing works
“In [29], discriminative features of electricity prices are extracted through a representation network, and Q -learning is developed to obtain the optimal discrete EV charging control strategy.”
MethodDevelopment of improved reinforcement learning smart charging strategy for electric vehicle fleet
Sultanuddin, S. J., Vibin, R., Kumar, A., Behera, N. R., Pasha, M. J., & Baseer, K. K. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.106987 · Cited by 60 · see citing works
Motlagh, S. G., Oladigbolu, J. O., & Li, L. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126058 · Cited by 68 · see citing works
Mansour, S. H., Azzam, S. M., Hasanien, H. M., Tostado‐Véliz, M., Alkuhayli, A. A., & Jurado, F. (2025) · Energy · doi:10.1016/j.energy.2025.134420 · Cited by 59 · see citing works
- Authors
- Wan, Z., Li, H., He, H., & Prokhorov, D.
- Affiliations
- University of Rhode Island (US)
- Shenyang Institute of Automation (CN)
- Toyota Motor North America Research & Development (United States) (US)
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2019-09
- Volume
- 10
- Issue
- 5
- Pages
- 5246-5257
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W2899639849
- Cited by
- 548
- References
- 46
- Citations by year
- 2026: 61 · 2025: 71 · 2024: 83 · 2023: 95 · 2022: 87 · 2021: 74 · 2020: 57 · 2019: 20
- FWCI
- 14.43
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- IEEE Transactions on Smart Grid (publishedVersion)
- Journal of Media Literacy Education (submittedVersion, cc-by)
- SIA OpenIR (Chinese Academy of Sciences) (submittedVersion)
- Funders
- Office of Naval Research (N00014-18-1-2396)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Electric Vehicles and Infrastructure
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Smart Grid Energy Management
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Electric Power System Optimization
- Keywords
- electric vehicle charging scheduling, real-time pricing, demand response, deep reinforcement learning, model-free learning, unknown transition probabilities, Markov decision process, Q-network, electricity price forecasting, smart grid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
