A Generic Markov Decision Process Model and Reinforcement Learning Method for Scheduling Agile Earth Observation Satellites
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
We investigate a general solution based on reinforcement learning for the agile satellite scheduling problem. The core idea of this method is to determine a value function for evaluating the long-term benefit under a certain state by training from experiences, and then apply this value function to guide decisions in unknown situations. First, the process of agile satellite scheduling is modeled as a finite Markov decision process with continuous state space and discrete action space. Two subproblems of the agile Earth observation satellite scheduling problem, i.e., the sequencing problem and the timing problem are solved by the part of the agent and the environment in the model, respectively. A satisfactory solution of the timing problem can be quickly produced by a constructive heuristic algorithm. The objective function of this problem is to maximize the total reward of the entire scheduling process. Based on the above design, we demonstrate that Q-network has advantages in fitting the long-term benefit of such problems. After that, we train the Q-network by Q-learning. The experimental results show that the trained Q-network performs efficiently to cope with unknown data, and can generate high total profit in a short time. The method has good scalability and can be applied to different types of satellite scheduling problems by customizing only the constraints checking process and reward signals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Self-citationA general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562 · cited 2×
“However, this approach requires quantities of labeled data, and the results are strongly coupled with the characteristics of the data set [5], [6].”
BackgroundEvolutionary computation: Comments on the history and current state
Back, T., Hammel, U., & Schwefel, H.-P. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585888 · Cited by 1,536 · cited 1×
“The idea of using meta-heuristics to solve decision problems originated in the 1950s [25], while it has developed rapidly in the past decade.”
BackgroundSelecting and scheduling observations of agile satellites
Lemaı̂tre, M., Verfaillie, G., Jouhaud, F., Lachiver, J.-M., & Bataille, N. (2002) · Aerospace Science and Technology · doi:10.1016/s1270-9638(02)01173-2 · Cited by 472 · cited 4×
“Unfortunately, we cannot guarantee the value function reaches its optimal value under the limited iterations of training, because the AEOS scheduling problem is an NP-hard problem, and the optimality of HADRT in the environment of the MDP cannot be guaranteed under all circumstances.”
BackgroundInfluentialLiu, X., Laporte, G., Chen, Y., & He, R. (2017) · Computers & Operations Research · doi:10.1016/j.cor.2017.04.006 · Cited by 250
Self-citationAn anytime branch and bound algorithm for agile earth observation satellite onboard scheduling
Chu, X., Chen, Y., & Tan, Y. (2017) · Advances in Space Research · doi:10.1016/j.asr.2017.07.026 · Cited by 110
Self-citationAFSCN scheduling: How the problem and solution have evolved
Barbulescu, L., Howe, A., & Whitley, D. (2006) · Mathematical and Computer Modelling · doi:10.1016/j.mcm.2005.12.004 · Cited by 69
Cited by
Deep Reinforcement Learning Method for Satellite Range Scheduling Problem
Ou, J., Xing, L., Yao, F., Li, M., Lv, J., He, Y., Song, Y., Wu, J., & Zhang, G. (2023) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2023.101233 · Cited by 91
Self-citationSong, Y., Ou, J., Pedrycz, W., Suganthan, P. N., Wang, X., Xing, L., & Zhang, Y. (2024) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2023.3345928 · Cited by 63
Self-citationLi, J., Wu, G., Liao, T., Fan, M., Mao, X., & Pedrycz, W. (2023) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3233358 · Cited by 61
Self-citationWen, Z., Li, L., Song, J., Zhang, S., & Hu, H. (2023) · Advances in Space Research · doi:10.1016/j.asr.2022.10.024 · Cited by 46
DRL-Based Dynamic Destroy Approaches for Agile-Satellite Mission Planning
Huang, W., Li, Z., He, X., Xiang, J., Du, X., & Liang, X. (2023) · Remote Sensing · doi:10.3390/rs15184503 · Cited by 11
Metadata
- Authors
- He, Y., Xing, L., Chen, Y., Pedrycz, W., Wang, L., & Wu, G.
- Venue
- IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Published
- 2022-03
- Volume
- 52
- Issue
- 3
- Pages
- 1463-1474
- ISSN
- 2168-2216 (ISSN-L)
- 2168-2232
- OpenAlex
- W3089255138
- Cited by
- 159
- References
- 8
- FWCI
- 6.61
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.