Fuel- and noise-minimal departure trajectory using deep reinforcement learning with aircraft dynamics and topography constraints
Open accessAbstract
Designing an optimal departure trajectory for an airport can minimize fuel emissions within the surrounding airspace and noise perceived by nearby populations, which brings positive sociological and economic implications in addition to environmental benefits. Yet, designing a trajectory that considers realistic operational constraints could be complex and, consequently, computationally expensive. Traditional trajectory optimization methods often simplify the problem to manage computational costs, which leads to compromised accuracy. To overcome this challenge, we propose a reinforcement learning (RL) approach that can satisfy multidisciplinary constraints by leveraging accurately modeled flight dynamics, high-fidelity population data, and topological data. This is achieved by establishing a comprehensive, physically-consistent simulated environment for the learning algorithm, while keeping the computational cost low. Instead of directly designing the trajectory itself, we train an RL agent to control the aircraft, whose trajectory is then considered as optimal. We model the RL problem as a continuous Markov decision process and employ the soft actor-critic architecture. By changing the relative importance of fuel consumption and noise in the optimization objective, we can obtain different optimum trajectories that are well-suited to the specific region of interest. Not surprisingly, a trade-off between fuel consumption and noise impact is observed in our results. This develop…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · see citing works
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · see citing works
(1999) · Cited by 5,147 · see citing works
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · see citing works
Markov Games as a Framework for Multi-Agent Reinforcement Learning
Littman, M. L. (1994) · International Conference on Machine Learning · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 3,406 · see citing works
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275 · see citing works
Bellman, R. (1957) · Indiana University Mathematics Journal · doi:10.1512/iumj.1957.6.56038 · Cited by 2,408 · see citing works
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342 · see citing works
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Baxter, L. A., & Puterman, M. L. (1995) · Technometrics · doi:10.2307/1269932 · Cited by 1,889 · see citing works
Competitive Markov Decision Processes
Filar, J. A., & Vrieze, K. (1996) · Springer eBooks · doi:10.1007/978-1-4612-4054-9 · Cited by 1,202 · see citing works
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · see citing works
Reinforcement Learning for UAV Attitude Control
Koch, W., Mancuso, R., West, R., & Azer Bestavros, C. (2019) · ACM Transactions on Cyber-Physical Systems · doi:10.1145/3301273 · Cited by 512 · see citing works
Markov Decision Processes with Applications to Finance
Bäuerle, N., & Rieder, U. (2011) · Universitext · doi:10.1007/978-3-642-18324-9 · Cited by 395 · see citing works
Machine Learning Proceedings 1993
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-27798-1 · Cited by 359 · see citing works
Autonomous navigation of stratospheric balloons using reinforcement learning
Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., & Wang, Z. (2020) · Nature · doi:10.1038/s41586-020-2939-8 · Cited by 295 · see citing works
BADA: An advanced aircraft performance model for present and future ATM systems
Nuic, A., Poles, D., & Mouillet, V. (2010) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.1176 · Cited by 209 · see citing works
Assessing the environmental impacts of aircraft noise and emissions
Mahashabde, A., Wolfe, P. J., Ashok, A., Dorbian, C. S., He, Q., Fan, A., Lukachko, S. P., Mozdzanowska, A., Wollersheim, C., Barrett, S. R. H., et al. (2010) · Progress in Aerospace Sciences · doi:10.1016/j.paerosci.2010.04.003 · Cited by 169 · see citing works
Xavier: a robot navigation architecture based on partially observable Markov decision process models
Koenig, S., & Simmons, R. G. (1998) · Cited by 147 · see citing works
Chai, R., Savvaris, A., Tsourdos, A., Xia, Y., & Chai, S. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2881190 · Cited by 141 · see citing works
On solving constrained optimization problems with neural networks: a penalty method approach
Lillo, W. E., Loh, M. H., Hui, S., & Żak, S. H. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.286888 · Cited by 134 · see citing works
Aircraft noise-abatement and mitigation strategies
Girvin, R. (2008) · Journal of Air Transport Management · doi:10.1016/j.jairtraman.2008.09.012 · Cited by 122 · see citing works
Optimization of Noise Abatement Departure Trajectories
Visser, H. G., & Wijnen, R. A. A. (2001) · Journal of Aircraft · doi:10.2514/2.2838 · Cited by 111 · see citing works
Multi-objective optimisation for aircraft departure trajectories minimising noise annoyance
Prats, X., Puig, V., Quevedo, J., & Nejjari, F. (2010) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2010.03.001 · Cited by 86 · see citing works
Constrained Optimization Using a Nondifferentiable Penalty Function
Conn, A. R. (1973) · SIAM Journal on Numerical Analysis · doi:10.1137/0710063 · Cited by 83 · see citing works
Sun, Z., & Li, L. (2013) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2013.08.033 · Cited by 73 · see citing works
Optimal departure trajectories with respect to sleep disturbance
Wijnen, R. A. A., & Visser, H. G. (2003) · Aerospace Science and Technology · doi:10.1016/s1270-9638(02)01183-5 · Cited by 58 · see citing works
Optimal, Environmentally Friendly Departure Procedures for Civil Aircraft
Torres, R., Chaptal, J., Bès, C., & Hiriart‐Urruty, J. (2011) · Journal of Aircraft · doi:10.2514/1.c031012 · Cited by 56 · see citing works
Vertical Navigation Trajectory Optimization Algorithm For A Commercial Aircraft
Mendoza, A. M., & Botez, R. M. (2014) · AIAA/3AF Aircraft Noise and Emissions Reduction Symposium (ANERS) · doi:10.2514/6.2014-3019 · Cited by 47 · see citing works
Reinforcement Learning for Two-Aircraft Conflict Resolution in the Presence of Uncertainty
Pham, D.-T., Tran, N. P., Goh, S. K., Alam, S., & Dương, V. (2019) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf.2019.8713624 · Cited by 41 · see citing works
Multi-impact optimization to reduce aviation noise and emissions
Zachary, D. S., Gervais, J., & Leopold, U. (2009) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2009.09.005 · Cited by 36 · see citing works
Hartjes, S., & Visser, H. G. (2016) · Proceedings of the Institution of Mechanical Engineers Part G Journal of Aerospace Engineering · doi:10.1177/0954410016648980 · Cited by 36 · see citing works
Trajectory Planning With Deep Reinforcement Learning in High-Level Action Spaces
Williams, K., Schlossman, R., Whitten, D., Ingram, J., Musuvathy, S., Pagan, J. E. G., Williams, K. A., Green, S., Patel, A., Mazumdar, A., et al. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3218496 · Cited by 31 · see citing works
Review of Technologies to Achieve Sustainable (Green) Aviation
Ramesh, K. (2012) · InTech eBooks · doi:10.5772/38899 · Cited by 24 · see citing works
Integrated design and allocation of optimal aircraft departure routes
Ho-Huu, V., Hartjes, S., Visser, H. G., & Curran, R. (2018) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2018.07.006 · Cited by 23 · see citing works
Ho-Huu, V., Hartjes, S., Pérez–Castán, J. A., Visser, H. G., & Curran, R. (2020) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2020.102684 · Cited by 14 · see citing works
Utilizing Reinforcement Learning to Continuously Improve a Primitive-Based Motion Planner
Goddard, Z. C., Wardlaw, K., Williams, K., Parish, J. J., & Mazumdar, A. (2022) · Journal of Aerospace Information Systems · doi:10.2514/1.i011044 · Cited by 11 · see citing works
Review of Technologies to Achieve Sustainable (Green) Aviation
Agarwal, R. K. (2012) · Contact-less Assessment of In-vivo Body Signals Using Microwave Doppler Radar (InTech) · Cited by 10 · see citing works
AirTrafficSim: An open-source web-based air traffic simulation platform.
Hui, K. Y., Nguyen, C. H., Lui, G. N., & Liem, R. P. (2023) · The Journal of Open Source Software · doi:10.21105/joss.04916 · Cited by 9 · see citing works
Self-citationKim, D., & Liem, R. P. (2022) · AIAA Journal · doi:10.2514/1.j061603 · Cited by 8 · see citing works
Self-citationAIAA/3AF Aircraft Noise and Emissions Reduction Symposium
(2014) · doi:10.2514/maners14 · Cited by 8 · see citing works
Noise Abatement and Control: An Overview of Federal Standards and Regulations
Bearden, D. M. (2002) · University of North Texas Digital Library (University of North Texas) · Cited by 6 · see citing works
Utilizing Reinforcement Learning to Continuously Improve a Primitive-Based Motion Planner
Goddard, Z. C., Wardlaw, K., Krishnan, R., Panagiotis Tsiotras, B., Smith, M. R., Sena, M., Parish, J. J., & Mazumdar, A. (2020) · AIAA SciTech Forum · doi:10.2172/1836182 · Cited by 6 · see citing works
Kim, D., Lyu, Y., & Liem, R. P. (2019) · AIAA AVIATION Forum · doi:10.2514/6.2019-3622 · Cited by 4 · see citing works
Self-citationNoise- and Fuel-Minimal Departure Trajectory Optimization with Reinforcement Learning
Nguyen, C. H., Shihua, J. M., Hui, K. Y., & Liem, R. P. (2023) · AIAA AVIATION Forum · doi:10.2514/6.2023-4215 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
1 of the 53 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A review of data science and artificial intelligence applications in air transportation systems
Li, L. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100023 · Cited by 6 · see citing works
Low-Noise Path Planning for Urban Drone Missions: Acoustic Ray Tracing and DDPG Algorithm
Rinaldi, M., Primatesta, S., & Guglieri, G. (2025) · The Aeronautical Journal · doi:10.1017/aer.2025.10054 · Cited by 5 · see citing works
Eti, S., Karaman, F., Yüksel, S., Dinçer, H., & Yaylalı, İ. (2025) · Scientific Reports · doi:10.1038/s41598-025-32988-7 · Cited by 3 · see citing works
“Nguyen et al. 9 used deep reinforcement learning methods to examine takeoff routes that minimize both fuel consumption and noise impacts.”
Louie, R., Tai, T. S., & Liem, R. P. (2025) · Journal of the Air Transport Research Society · doi:10.1016/j.jatrs.2025.100072 · Cited by 3 · see citing works
Self-citationReal-time bi-level aircraft trajectory optimisation under the presence of unsteady wind
Shihua, J. M., Nguyen, C. H., & Liem, R. P. (2025) · Optimization and Engineering · doi:10.1007/s11081-025-10005-0 · Cited by 1 · see citing works
Self-citationIbili, F., Adams, C. A., Adebanji, A., & Andam‐Akorful, S. A. (2025) · African Transport Studies · doi:10.1016/j.aftran.2025.100066 · Cited by 1 · see citing works
C, A. (2026) · International Journal of Scientific Research in Science Engineering and Technology · doi:10.32628/ijsrset2513885
Semantic Web-Guided Bi-Context Scheduling for Airport Ground Vehicle Operations
Wang, S., & Chi, T. (2026) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.405431
Trajectory optimization learning for fixed-wing aircraft
Yaoming, Z. H. O. U., Zhang, C., Wang, Y., Guangtong, X. U., Qihan, S. U. N., Qin, Q., & Lingyun, G. U. (2026) · Chinese Journal of Aeronautics · doi:10.1016/j.cja.2026.104388
LLM-enhanced Dynamic Fleet Planning with Hierarchical Multi-agent Reinforcement Learning Framework
Lijing, L., Shihua, J. M., Yan, Q., & Liem, R. P. (2026) · MATEC Web of Conferences · doi:10.1051/matecconf/202642204003
Self-citationLi, M., Wang, L., & Chen, S. (2026) · Cleaner Logistics and Supply Chain · doi:10.1016/j.clscn.2026.100378
Arrival Route Network Design for Reducing Noise Impact in the Terminal Maneuvering Area
Chen, W., Yu, F., Shen, D., Peng, Y., & Zheng, T. (2026) · Aerospace · doi:10.3390/aerospace13100936
- Authors
- Nguyen, C. H., Shihua, J. M., & Liem, R. P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Corresponding author
- Liem, Rhea P.
- Venue
- Communications in Transportation Research
- Publisher
- Elsevier BV
- Published
- 2025-12
- Volume
- 5
- Pages
- 100165
- ISSN
- 2772-4247 (ISSN-L)
- OpenAlex
- W4408361925
- Cited by
- 12
- References
- 53
- Citations by year
- 2026: 6 · 2025: 6
- FWCI
- 4.53
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Communications in Transportation Research (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Innovation and Technology Commission - Hong Kong
- University Grants Committee
- Topics
- Physical Sciences › Engineering › Aerospace Engineering › Air Traffic Management and Optimization
- Physical Sciences › Environmental Science › Global and Planetary Change › Advanced Aircraft Design and Technologies
- Health Sciences › Health Professions › Speech and Hearing › Noise Effects and Management
- Keywords
- aircraft noise, fuel consumption, deep reinforcement learning, soft actor-critic, flight dynamics, Markov decision process, sustainable aviation, airspace design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
