The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning
Open accessAbstract
Driven by the recent advances in electric vehicle (EV) technologies, EVs have become important for smart grid economy. When EVs participate in demand response program which has real-time pricing signals, the charging cost can be greatly reduced by taking full advantage of these pricing signals. However, it is challenging to determine an optimal charging strategy due to the existence of randomness in traffic conditions, user's commuting behavior, and the pricing process of the utility. Conventional model-based approaches require a model of forecast on the uncertainty and optimization for the scheduling process. In this paper, we formulate this scheduling problem as a Markov Decision Process (MDP) with unknown transition probability. A model-free approach based on deep reinforcement learning is proposed to determine the optimal strategy for this problem. The proposed approach can adaptively learn the transition probability and does not require any system model information. The architecture of the proposed approach contains two networks: a representation network to extract discriminative features from the electricity prices and a Q network to approximate the optimal action-value function. Numerous experimental results demonstrate the effectiveness of the proposed approach.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For load monitoring and electric vehicle charging scheduling, transfer learning [200] and deep reinforcement learning [215] methods are proposed.”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [215]
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
A Critical Review of Recurrent Neural Networks for Sequence LearningPreprint
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 3,159
Short-Term Residential Load Forecasting Based on LSTM Recurrent Neural Network
Kong, W., Dong, Z. Y., Jia, Y., Hill, D. J., Xu, Y. H., & Zhang, Y. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2753802 · Cited by 2,806
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
Research works printed but not linked: 5
- bellman, 1958, Dynamic Programming
- 2018, Grid-Connected Renewable Energy Systems
- goodfellow, 2016, Deep Learning, 1
- 2017, The Egallon How Much Cheaper is it to Drive on Electricity?
- 2017, California Independent System Operator
Cited by
Sorted by citations
Das, H. S., Rahman, M. M., Li, S., & Tan, C. W. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2019.109618 · Cited by 1,140
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
“For load monitoring and electric vehicle charging scheduling, TL [200] and deep RL [215] methods are proposed.”
BackgroundMethodReinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review
Cao, D., Hu, W., Zhao, J., Zhang, G., Zhang, B., Liu, Z., Chen, Z., & Blaabjerg, F. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000552 · Cited by 460
“Reference [80] proposes a DQN-based approach for the optimization of EV charging while [81] extends this work by integrating the DQN algorithm with the long-short term memory (LSTM) NN.”
MethodConstrained EV Charging Scheduling Based on Safe Deep Reinforcement Learning
Li, H., Wan, Z., & He, H. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2955437 · Cited by 442
“If we maximize the lower bound to generate the policy π k + 1 when at the k th iteration, we are guaranteed that an improving policy can be obtained according to Wan et al. [16], i.e., Now, we can use the update rule in (38) to improve the policy π k to maximize the return J (π k ) .”
BackgroundMethodInfluentialSelf-citationA Survey of Recent Advances in Edge-Computing-Powered Artificial Intelligence of Things
Chang, Z., Liu, S., Xiong, X., Cai, Z., & Tu, G. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3088875 · Cited by 431
Chen, X., Qu, G., Tang, Y., Low, S. H., & Li, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3154718 · Cited by 465
“In [121]–[123], various RL techniques are studied to design EV charging policies to deal with the randomness in the arrival and departure time of an EV.”
BackgroundA Multi-Agent Reinforcement Learning-Based Data-Driven Method for Home Energy Management
Xu, X., Jia, Y., Xu, Y., Xu, Z., Chai, S., & Lai, C. S. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2971427 · Cited by 420
“[14], Wan et al. use deep RL algorithms to determine the optimal solutions of the EV charging/discharging scheduling problem.”
Du, Y., & Li, F. F. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2930299 · Cited by 396
“The authors in [23] proposed the application of deep RL to optimize the real-time electric vehicle charging schedule with the consideration of future electricity price.”
BackgroundCao, J. H., Harrold, D. J. B., Fan, Z., Morstyn, T., Healey, D., & Li, K. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2986333 · Cited by 357
“The authors of [14] proposed a deep reinforcement learning based approach to optimize the EV charging scheduling.”
BackgroundDeep Reinforcement Learning for Strategic Bidding in Electricity Markets
Ye, Y., Qiu, D., Sun, M., Papadaskalopoulos, D. A., & Štrbac, G. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2936142 · Cited by 310
Ganesh, A. H., & Xu, B. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111833 · Cited by 295
Machine Learning Approaches for EV Charging Behavior: A Review
Shahriar, S., Al-Ali, A. R., Osman, A. H., Dhou, S., & Nijim, M. (2020) · IEEE Access · doi:10.1109/access.2020.3023388 · Cited by 301
“There are a few recent works utilizing RL-based concepts for scheduling of EV charging [93]–[96] and therefore has great potential for further research.”
BackgroundQian, T., Shao, C., Wang, X., & Shahidehpour, M. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2942593 · Cited by 271
“Sampling in replay buffer breaks the temporal correlations between the selected tuples [26], making the training more stable.”
BackgroundDeep learning for edge computing applications: A State-of-the-Art survey
Wang, F., Zhang, M., Wang, X., Ma, X., & Liu, J. (2020) · IEEE Access · doi:10.1109/access.2020.2982411 · Cited by 259
“For instance, Wan et al. [104] jointly considered the electricity price and battery energy of electric vehicles and proposed a model-free DRL based model to automatically determine the charging policy.”
BackgroundDeep reinforcement learning for power system: An overview
Zhang, Z., Zhang, D., & Qiu, R. C. (2019) · CSEE Journal of Power and Energy Systems · doi:10.17775/cseejpes.2019.00920 · Cited by 287
Ye, Y., Qiu, D., Wu, X., Štrbac, G., & Ward, J. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2976771 · Cited by 246
“In this respect, the deep Q network (DQN) method is applied to optimize energy usage schedules of shiftable loads [36], [37], EV [33], [38], an integrated renewable energy and EES system [39]–[41], and a heating, ventilation and air conditioning system [42].”
MethodHu, H., Jia, X., He, Q., Fu, S., & Liu, K. (2020) · Computers & Industrial Engineering · doi:10.1016/j.cie.2020.106749 · Cited by 254
Reinforcement Learning Based EV Charging Management Systems–A Review
Abdullah, H. M., Gastli, A., & Ben‐Brahim, L. (2021) · IEEE Access · doi:10.1109/access.2021.3064354 · Cited by 249
“[75], [77], [79]–[84], [86], [88], [90]–[92], [94], [95], and [97]–[99], used the charging or discharging cost as a state parameter.”
BackgroundMethodInfluentialReinforcement learning in sustainable energy and electric systems: a survey
Yang, T., Zhao, L., Li, W., & Zomaya, A. Y. (2020) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2020.03.001 · Cited by 248
Yang, T., Zhao, L., Li, W., & Zomaya, A. Y. (2021) · Energy · doi:10.1016/j.energy.2021.121377 · Cited by 224
Hu, D., Ye, Z., Gao, Y., Ye, Z., Peng, Y., & Yu, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3185975 · Cited by 232
“A model-free approach is proposed to schedule the home charging of EVs considering the timevarying electricity price in distribution systems [9].”
MethodReal-Time Residential Demand Response
Li, H., Wan, Z., & He, H. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2978061 · Cited by 208
“…Anxiety: The range anxiety measures the resi-dent’s fear that the EV has insufficient energy to reach its destination, which is measured by, where E EV t − E EVmax represents the uncharged battery energy when the EV departs at t EV β , and the squared term measures the range anxiety in $/kWh 2 [22].”
BackgroundMethodSelf-citationQian, T., Shao, C., Li, X., Wang, X., & Shahidehpour, M. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2969650 · Cited by 210
“problem with uncertain electricity prices [20] and a robot motion planning problem aimed at moving through a bottleneck exit [21] are addressed by deep Q network (DQN) [15].”
BackgroundShin, M., Choi, D., & Kim, J. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2944183 · Cited by 207
Multi-agent deep reinforcement learning approach for EV charging scheduling in a smart grid
Park, K., & Moon, I. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.120111 · Cited by 193
“However, using the data generated from the above situations can more practically approach the situation that occurs in a more realistic environment [5].”
BackgroundMethodInfluentialYan, L., Chen, X., Zhou, J., Chen, Y., & Wen, J. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3098298 · Cited by 197
Lu, R., Li, Y., Li, Y., Jiang, J., & Ding, Y. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115473 · Cited by 185
CDDPG: A Deep-Reinforcement-Learning-Based Approach for Electric Vehicle Charging Control
Zhang, F., Yang, Q., & An, D. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3015204 · Cited by 179
“According to [33], we assume that the charging and discharging price at the same time are the same.”
BackgroundMethodInfluentialReinforcement learning for electric vehicle applications in power systems:A critical review
Qiu, D., Wang, Y., Hua, W., & Štrbac, G. (2022) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2022.113052 · Cited by 202
Smart and Resilient EV Charging in SDN-Enhanced Vehicular Edge Computing Networks
Liu, J., Guo, H., Xiong, J., Kato, N., Zhang, J., & Zhang, Y. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2951966 · Cited by 177
“[34] formulated the real-time EV charging/ discharging scheduling problem as a finite Markov decision process (MDP) with unknown transition probability, and proposed a model-free approach based on DRL to determine an optimal strategy for charging and discharging.”
MethodArtificial intelligence techniques for enabling Big Data services in distribution networks: A review
Barja-Martínez, S., Aragüés‐Peñalba, M., Munné‐Collado, Í., Lloret-Gallego, P., Bullich‐Massagué, E., & Villafáfila‐Robles, R. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111459 · Cited by 187
Effective Charging Planning Based on Deep Reinforcement Learning for Electric Vehicles
Zhang, C., Liu, Y., Wu, F., Tang, B., & Fan, W. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3002271 · Cited by 168
“Because the existence of randomness in traffic conditions, users’ commuting behaviors, and the pricing process of the utility, the literatures [19] proposes a model-free approach to determine an optimal charging strategy for EVs charging.”
BackgroundQiu, D., Ye, Y., Papadaskalopoulos, D. A., & Štrbac, G. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.116940 · Cited by 164
Liang, Y., Ding, Z., Ding, T., & Lee, W. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3025082 · Cited by 166
“1 2 ) [19].”
BackgroundAgent-Based Modeling in Electricity Market Using Deep Deterministic Policy Gradient Algorithm
Liang, Y., Guo, C., Ding, Z., & Hua, H. (2020) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2020.2999536 · Cited by 168
“…has been applied to model-free optimization and control in complex environments of power systems, such as electric vehicle charging scheduling [16], online building energy optimization [17], microgrid energy management [18], short-term voltage con-trol [19], and adaptive power system…”
MethodYe, Y., Tang, Y. P., Wang, H., Zhang, X., & Štrbac, G. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3103917 · Cited by 156
Learning to Operate an Electric Vehicle Charging Station Considering Vehicle-Grid Integration
Ye, Z., Gao, Y., & Yu, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3165479 · Cited by 144
Electric Vehicle Charging Management Based on Deep Reinforcement Learning
Li, S., Hu, W., Cao, D., Dragičević, T., Huang, Q., Chen, Z., & Blaabjerg, F. (2022) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000460 · Cited by 151
“In recent years, different neural network (NN) based meth‐ ods have been applied to the research of EV [12]-[15].”
MethodDorokhova, M., Martinson, Y., Ballif, C., & Wyrsch, N. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.117504 · Cited by 143
Deep Reinforcement Learning for Demand Response in Distribution Networks
Bahrami, S., Chen, Y. C., & Wong, V. W. S. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3037066 · Cited by 141
“Wan et al. [23] considered a long short-term memory (LSTM) network to extract informative features from the price rates and developed a DRL-based online charging scheduling algorithm for electric vehicles.”
BackgroundCyber-Attack Recovery Strategy for Smart Grid Based on Deep Reinforcement Learning
Wei, F., Wan, Z., & He, H. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2956161 · Cited by 137
“As to the power grid, [25], [26] present deep RL methods to determine the optimal strategy for real-time EV charging scheduling.”
MethodSelf-citationMulti-Agent Deep Reinforcement Learning Method for EV Charging Station Game
Qian, T., Shao, C., Li, X., Wang, X., Chen, Z., & Shahidehpour, M. (2021) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2021.3111014 · Cited by 139
Dynamic Pricing for EV Charging Stations: A Deep Reinforcement Learning Approach
Zhao, Z., & Lee, C. K. M. (2021) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2021.3139674 · Cited by 135
“Consequently, it is difficult to mathematically model the customers’ charging demand response in this context [35].”
BackgroundΤσαούσογλου, Γ., Giraldo, J. S., & Paterakis, N. G. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111890 · Cited by 133
“In [84], a penalty term is designed to teach a Neural Network to respect the local constraints of the DERs.”
MethodSafe reinforcement learning for real-time automatic control in a smart energy-hub
Qiu, D., Dong, Z., Zhang, X., Wang, Y., & Štrbac, G. (2022) · Applied Energy · doi:10.1016/j.apenergy.2021.118403 · Cited by 130
Zhang, Y., Yang, Q., An, D., Li, D., & Wu, Z. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3165074 · Cited by 134
“In recent years, the RL methods have been used to develop energy scheduling strategies [24]–[28].”
BackgroundMethodA Deep Reinforcement Learning Method for Pricing Electric Vehicles With Discrete Charging Levels
Qiu, D., Ye, Y., Papadaskalopoulos, D. A., & Štrbac, G. (2020) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2020.2984614 · Cited by 127
“Numerous recent papers have applied the DQN method to different electricity system problems [29]–[31], although its application in pricing design problems has yet to be explored.”
MethodDistributed Deep Reinforcement Learning for Intelligent Load Scheduling in Residential Smart Grids
Chung, H.-M., Maharjan, S., Zhang, Y., & Eliassen, F. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3007167 · Cited by 120
“To further address the discretization issue associated with Q-learning [18]–[23], DQN [24]–[26], and policy gradient [27], [28], the employed model-free approach is similar to model-based methods that provide a continuous value as the solution without discretizing the action space.”
MethodJahangir, H., Gougheri, S. S., Vatandoust, B., Golkar, M. A., Ahmadian, A., & Hajizadeh, A. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2998072 · Cited by 121
“Each aggre-gator needs to devise plans that are lucrative for PEV owners and at the same time ensure its profit margin in competition with other aggregators [2].”
BackgroundMethodJin, J., & Xu, Y. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3028470 · Cited by 118
“We further consider the rolling model predictive control (MPC) method [22], [24] as a benchmark.”
BackgroundMethodWang, Z., Ikechukwu, O. M., Huang, H., Qiu, C., Hisada, M., & Abdallah, A. B. (2020) · IEEE Access · doi:10.1109/access.2020.3044612 · Cited by 114
“Variants of intelligent energy management methods have also been proposed based on RL [35]–[37], [51], [52] and recurrent neural network (RNN) [31], [32].”
BackgroundMethodZhang, G., Hu, W., Cao, D., Huang, Q. P., Yi, J., Chen, Z., & Blaabjerg, F. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2997790 · Cited by 113
“Since equation (26) satisfies the recursive relationship, it can be transformed into the Bellman expectation equation [25]:”
BackgroundCoordinated Electric Vehicle Active and Reactive Power Control for Active Distribution Networks
Wang, Y., Qiu, D., Štrbac, G., & Gao, Z. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3169975 · Cited by 120
“In SARL, deep Q-network (DQN) has been widely applied to solve various EV problems, e.g., realtime charging/discharging strategies [14], optimal EV charging navigation [15], and fast frequency regulation services [16].”
BackgroundDesign possibilities and challenges of DNN models: a review on the perspective of end devices
Hussain, H., Tamizharasan, P. S., & Rahul, C. S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10138-z · Cited by 118
“Another application that considers the battery energy and power consumption of vehicles to perform an automated charging policy (Wan et al. 2019).”
BackgroundChen, Q., & Folly, K. A. (2022) · Energies · doi:10.3390/en16010146 · Cited by 121
Gao, Y., Yang, J., Yang, M. Y., & Li, Z. (2020) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2020.2986412 · Cited by 107
A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning
Zhou, Y., Zhang, B., Xu, C., Lan, T., Diao, R., Shi, D., Wang, Z., & Lee, W. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000522 · Cited by 106
“In [27], the DRL is used to solve the EV charging scheduling prob‐ lem for real-time control, which can adaptively learn the transition probability and does not require any system model information.”
MethodPrivacy Preserving Load Control of Residential Microgrid via Deep Reinforcement Learning
Qin, Z., Liu, D., Hua, H., & Cao, J. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3088290 · Cited by 105
“The deep Q-network (DQN) is used for EV charging scheduling in [21], while dueling DQN is applied to optimize the demand response management of interruptible load in [22].”
MethodYan, L., Chen, X., Chen, Y., & Wen, J. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3152218 · Cited by 105
“A deep-Q-network-based learning algorithm is proposed to schedule the charging process of the individual EV in [16].”
BackgroundMethodDu, Y., Li, F. F., Zandi, H., & Xue, Y. S. (2021) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000502 · Cited by 102
“In power systems, the potentials of implementing deep RL for demand-side energy management and electric vehicle charg ‐ ing/discharging scheduling are shown in [14], [15].”
BackgroundFang, X., Zhao, Q., Wang, J., Han, Y., & Li, Y. (2021) · Sustainable Cities and Society · doi:10.1016/j.scs.2021.103163 · Cited by 98
Hedayatnia, A., Ghafourian, J., Sepehrzad, R., Al-Durrad, A., & Anvari‐Moghaddam, A. (2024) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2024.110142 · Cited by 96
Xu, H., Chen, Y., Wang, H., Wang, H., Meng, Y., & Gu, Q. (2022) · Sustainable Cities and Society · doi:10.1016/j.scs.2022.104345 · Cited by 94
Guo, C., Wang, X., Zheng, Y., & Zhang, F. (2021) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107048 · Cited by 93
Deep Reinforcement Learning-Based Demand Response for Smart Facilities Energy Management
Lu, R., Bai, R., Luo, Z., Jiang, J., Sun, M., & Zhang, H. (2021) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2021.3104596 · Cited by 92
Fescioglu-Unver, N., & Aktaş, M. Y. (2023) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2023.113873 · Cited by 92
Wang, X., Wang, J., & Liu, J. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2993858 · Cited by 91
“Inrecent years, deep reinforcement learning (DRL) ap-proacheshavebeensuccessfullyadoptedinEVoptimalchargingscheduling[26]–[28]notonlybecauseitcanhandlethenon-convex relations between the EVs and the electricity market and always guarantee a feasible solution but also because of its real-time…”
BackgroundIntegrating Battery Aging in the Optimization for Bidirectional Charging of Electric Vehicles
Schwenk, K., Meisenbacher, S., Briegel, B., Harr, T., Hagenmeyer, V., & Mikut, R. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3099206 · Cited by 95
“, model-based optimization [16], [22] or model-free reinforcement learning [9].”
MethodGorostiza, F. S., & Gonzalez-longatt, F. M. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2996274 · Cited by 90
“In [17], a DQN agent was trained to find the optimal charging schedule for an EV considering battery SOC, range anxiety, and charging costs in the formulation of the rewards.”
Norouzi, M., Aghaei, J., Niknam, T., Alipour, M., Pirouzi, S., & Lehtonen, M. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121573 · Cited by 92
A Review of Denial of Service Attack and Mitigation in the Smart Grid Using Reinforcement Learning
Ortega-Fernandez, I., & Liberati, F. (2023) · Energies · doi:10.3390/en16020635 · Cited by 89
“For example, implementing a deep RL algorithm to learn the optimal strategy for real-time electric vehicle charging scheduling [46,47], or to use of a model-free RL method for load frequency control resistant to weather uncertainties [48].”
MethodEntropy-based Sampling Approaches for Multi-Class Imbalanced Problems
Li, L., He, H., & Li, J. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2913859 · Cited by 88
“Oversampling and undersampling techniques are commonly used to imbalanced learning [5].”
BackgroundSelf-citationGoh, H. H., Huang, Y., Lim, C. S., Zhang, D., Liu, H., Dai, W., Kurniawan, T. A., & Rahman, S. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3179567 · Cited by 93
“The BRF is commonly defined as follows [8], [22], [37]–[39]: is the power equation for the self-supply of the microgrid using distributed energy.”
BackgroundDolatabadi, A., Abdeltawab, H. H., & Mohamed, Y. A. I. (2022) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2022.3212938 · Cited by 87
“Intheproposed approach, since it is entirely data-dependent, all hyperparameters are optimally tailored using the training data following the recommendations of the deep learning community [55], while on the other hand, the scheme for setting the parameters presented in [34] is referred to.”
BackgroundCoordinated Topology Attacks in Smart Grid Using Deep Reinforcement Learning
Wang, Z., He, H., Wan, Z., & Sun, Y. L. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2994977 · Cited by 86
“There is no need to construct a model for the power system, and it can be easily applied to different power systems [17], [18].”
BackgroundMethodSelf-citationZhang, W., Chen, Q., Yan, J., Zhang, S., & Xu, J. (2021) · Energy · doi:10.1016/j.energy.2021.121492 · Cited by 89
Zhao, Z., Lee, C. K. M., & Ren, J. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.122278 · Cited by 86
Demand Response Management for Industrial Facilities: A Deep Reinforcement Learning Approach
Huang, X., Hong, S. H., Yu, M., Ding, Y., & Jiang, J. (2019) · IEEE Access · doi:10.1109/access.2019.2924030 · Cited by 85
“In [28], Wan et al.”
Xu, P., Zhang, J., Gao, T., Chen, S., Wang, X., Jiang, H., & Gao, W. (2022) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2022.108030 · Cited by 84
Kim, S. H., & Shin, Y. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.109144 · Cited by 82
Wu, T., Wang, J., Lu, X., & Du, Y. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.118189 · Cited by 79
Cao, D., Hu, W., Xu, X., Dragičević, T., Huang, Q. P., Liu, Z., Chen, Z., & Blaabjerg, F. (2019) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2019.105648 · Cited by 79
Hua, H., Wei, Z., Qin, Y., Wang, T., Li, L., & Cao, J. (2021) · IET Cyber-Physical Systems Theory & Applications · doi:10.1049/cps2.12007 · Cited by 77
“[109] describe the real‐time charging scheduling of electric vehicles as a Markov decision process with unknown transition probabilities, and propose a model‐free optimal scheduling method using DRL to obtain charge and discharge scheduling.”
MethodShariatzadeh, M., Lopes, M. A. R., & Antunes, C. H. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126167 · Cited by 74
Reinforcement learning for electric vehicle charging scheduling: A systematic review
Zhao, Z., Lee, C. K. M., Yan, X., & Wang, H. (2024) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2024.103698 · Cited by 81
Kim, H. J., & Kim, M.-K. (2024) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2024.3368419 · Cited by 80
“Statistical models primarily include Monte Carlo simulations [3], fuzzy linear regression [4], quantile linear regression [5], autoregressive integral moving averaging (ARIMA) [6], and the Kalman filter method [7].”
MethodZheng, Y., Yan, Z., Chen, K., Sun, J., Xu, Y. H., & Liu, Y. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3062700 · Cited by 70
“DRL has been shown as an effective solution for many problems in power systems [11]–[20], and the performance of DRL gives credit to the ability of DNNs.”
BackgroundLee, J., Lee, E., & Kim, J. (2020) · Energies · doi:10.3390/en13081950 · Cited by 69
Liu, D., Zeng, P., Cui, S., & Song, C. (2023) · Sensors · doi:10.3390/s23031618 · Cited by 68
Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling
Fang, X., Wang, J., Song, G., Han, Y., Zhao, Q., & Cao, Z. (2019) · Energies · doi:10.3390/en13010123 · Cited by 67
“Considering the integration of EVs and RGs, residential MES becomes more complicated due to high uncertainties from EVs and RGs, such as RGs output and EVs’ usage attributes [8,9].”
BackgroundJin, J., Mao, S., & Xu, Y. (2023) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2023.3250505 · Cited by 68
“…based techniques to solve the formulated MDP. Recently, deep reinforcement learning (DRL) based approaches have been adopted for the scheduling of EV charging and cloud computing demands under random task arrivals, renewable generation, and electricity prices [18], [19], [20], [21], [22].”
BackgroundMethodA Critical Review on Charging Technologies of Electric Vehicles
Shahjalal, M., Shams, T., Tasnim, M. N., Ahmed, M. R., Ahsan, M., & Haider, J. (2022) · Energies · doi:10.3390/en15218239 · Cited by 66
On data-driven modeling and control in modern power grids stability: Survey and perspective
Gong, X., Wang, X., & Cao, B. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121740 · Cited by 67
A Deep RL-Based Algorithm for Coordinated Charging of Electric Vehicles
Zhang, Z., Wan, Y., Qin, J., Fu, W., & Kang, Y. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3170000 · Cited by 63
“…input of our Q network contains the characteristics of the future price trend extracted from the long and short-term memory (LSTM) network, not only improving the fitting performance of the linear function approximation method in [13], [18], but also being applicable to various pricing mechanisms.”
BackgroundMethodInfluentialFeng, B., Liu, Z., Huang, G., & Guo, C. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121615 · Cited by 63
Deep Reinforcement Learning Based Optimal Route and Charging Station Selection
Lee, K.-B., Ahmed, M. A., Kang, D.-K., & Kim, Y.-C. (2020) · Energies · doi:10.3390/en13236255 · Cited by 62
Federated Reinforcement Learning for Electric Vehicles Charging Control on Distribution Networks
Qian, J., Jiang, Y., Liu, X., Wang, Q., Wang, T., Shi, Y., & Chen, W. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3306826 · Cited by 65
“In [29], discriminative features of electricity prices are extracted through a representation network, and Q -learning is developed to obtain the optimal discrete EV charging control strategy.”
MethodDevelopment of improved reinforcement learning smart charging strategy for electric vehicle fleet
Sultanuddin, S. J., Vibin, R., Kumar, A., Behera, N. R., Pasha, M. J., & Baseer, K. K. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.106987 · Cited by 60
Motlagh, S. G., Oladigbolu, J. O., & Li, L. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126058 · Cited by 68
Mansour, S. H., Azzam, S. M., Hasanien, H. M., Tostado‐Véliz, M., Alkuhayli, A. A., & Jurado, F. (2025) · Energy · doi:10.1016/j.energy.2025.134420 · Cited by 59
Metadata
- Authors
- Wan, Z., Li, H., He, H., & Prokhorov, D.
- Affiliations
- University of Rhode Island (US)
- Shenyang Institute of Automation (CN)
- Toyota Motor North America Research & Development (United States) (US)
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2019-09
- Volume
- 10
- Issue
- 5
- Pages
- 5246-5257
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W2899639849
- Cited by
- 548
- References
- 46
- Citations by year
- 2026: 61 · 2025: 71 · 2024: 83 · 2023: 95 · 2022: 87 · 2021: 74 · 2020: 57 · 2019: 20
- FWCI
- 14.43
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- IEEE Transactions on Smart Grid (publishedVersion)
- Journal of Media Literacy Education (submittedVersion, cc-by)
- SIA OpenIR (Chinese Academy of Sciences) (submittedVersion)
- Funders
- Office of Naval Research (N00014-18-1-2396)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Electric Vehicles and Infrastructure
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Smart Grid Energy Management
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Electric Power System Optimization
- Keywords
- electric vehicle charging scheduling, real-time pricing, demand response, deep reinforcement learning, model-free learning, unknown transition probabilities, Markov decision process, Q-network, electricity price forecasting, smart grid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
