The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Deep Multi-Agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic
Open accessAbstract
On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 3×
“4, where the actor network and the critic network share the same low-level representations, and the policy loss and the value function error loss are thus combined into a single loss function [40].”
BackgroundMethodPlaying Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Congested traffic states in empirical observations and microscopic simulations
Treiber, M., Hennecke, A., & Helbing, D. (2000) · Physical Review E · doi:10.1103/physreve.62.1805 · Cited by 4,762
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
High-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103
Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents
Tan, M. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50049-6 · Cited by 1,810
General Lane-Changing Model MOBIL for Car-Following Models
Kesting, A., Treiber, M., & Helbing, D. (2007) · Transportation Research Record: Journal of the Transportation Research Board · doi:10.3141/1999-10 · Cited by 1,345
Review of road traffic control strategies
Papageorgiou, M., Kiakaki, C., Dinopoulou, V., Kotsialos, A., & Wang, Y. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.819610 · Cited by 1,251
Self-citation(2015) · Journal of Machine Learning Research · Cited by 1,119
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · cited 1×
“MARL has found great success across a wide range of multi-agent systems, including traffic light control [32], games [33], resource management in wireless networks [34], and powergrid control [35], only to name a few.”
BackgroundMulti-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control
Chu, T., Wang, J., Codeca, L., & Li, Z. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2901791 · Cited by 1,046
Self-citationThe Surprising Effectiveness of PPO in Cooperative Multi-Agent Games
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A. M., & Wu, Y. (2022) · neural information processing systems · doi:10.52202/068431-1787 · Cited by 1,027
Self-citationSim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941 · cited 1×
“Interested readers are referred to an extensive survey on sim-to-real paradigms for reinforcement learning [62].”
BackgroundRios-Torres, J., & Malikopoulos, A. A. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2016.2600504 · Cited by 794
Freeway ramp metering: an overview
Papageorgiou, M., & Kotsialos, A. (2002) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2002.806803 · Cited by 701
Deep Recurrent Q-Learning for Partially Observable MDPs
Hausknecht, M., & Stone, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06527 · Cited by 667
A Closer Look at Invalid Action Masking in Policy Gradient Algorithms
Huang, S., & Ontañón, S. (2022) · The International FLAIRS Conference Proceedings · doi:10.32473/flairs.v35i.130584 · Cited by 435 · cited 1×
“Second, sampling invalid/unsafe actions also impedes policy training as invalid policy updates [59] are executed for invalid actions, since the collected experiences associated with the invalid actions are not meaningful and misleading.”
BackgroundScalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
Wu, Y., Mansimov, E., Liao, S., Grosse, R., & Ba, J. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05144 · Cited by 432
Polack, P., Altche, F., d'Andrea-Novel, B., & de La Fortelle, A. (2017) · 2017 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2017.7995816 · Cited by 408
Examining accident reports involving autonomous vehicles in California
Favarò, F. M., Nader, N., Eurich, S. O., Tripp, M., & Varadaraju, N. (2017) · PLOS ONE · doi:10.1371/journal.pone.0184952 · Cited by 407
Autonomous Vehicles: Disengagements, Accidents and Reaction Times
Dixit, V. V., Chand, S., & Nair, D. J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0168054 · Cited by 399 · cited 1×
“Despite the great advances over the past decade that have made this possible, the number of traffic accidents involving AVs are increasing in recent years [3], [4].”
BackgroundEstimating Acceleration and Lane-Changing Dynamics from Next Generation Simulation Trajectory Data
Thiemann, C., Treiber, M., & Kesting, A. (2008) · Transportation Research Record Journal of the Transportation Research Board · doi:10.3141/2088-10 · Cited by 389
Efficient Large-Scale Fleet Management via Multi-Agent Deep Reinforcement Learning
Lin, K., Zhao, R., Xu, Z., & Zhou, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219993 · Cited by 379
Lane Change and Merge Maneuvers for Connected and Automated Vehicles: A Survey
Bevly, D., Cao, X., Gordon, M., Ozbilgin, G., Kari, D., Nelson, B., Woodruff, J., Barth, M., Murray, C., Kurt, A., et al. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2015.2503342 · Cited by 282
Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control
Lin, Y., McPhee, J., & Azad, N. L. (2021) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2020.3012947 · Cited by 236
Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning
Palanisamy, P. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207663 · Cited by 211
Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
Naderializadeh, N., Sydir, J. J., Simsek, M., & Nikopour, H. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3051163 · Cited by 187
Cao, W., Mukai, M., Kawabe, T., Nishira, H., & Fujiki, N. (2015) · Control Engineering Practice · doi:10.1016/j.conengprac.2014.10.005 · Cited by 182
Yu, C., Wang, X., Xu, X., Zhang, M., Ge, H., Ren, J., Sun, L., Chen, B., & Tan, G. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2893683 · Cited by 170
Cooperation-Aware Reinforcement Learning for Merging in Dense Traffic
Bouton, M., Nakhaei, A., Fujimura, K., & Kochenderfer, M. J. (2019) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2019.8916924 · Cited by 140
Preferred time-headway of highway drivers
Ayres, T. J., Li, L., Schleuning, D., & Young, D. (2002) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2001.948767 · Cited by 137
Driving in Dense Traffic with Model-Free Reinforcement Learning
Saxena, D. M., Bae, S., Nakhaei, A., Fujimura, K., & Likhachev, M. A. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197132 · Cited by 136
Novel Decision-Making Strategy for Connected and Autonomous Vehicles in Highway On-Ramp Merging
Kherroubi, Z. E. A., Aknine, S., & Bacha, R. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3114983 · Cited by 125
Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games
Wang, J., Zhang, Y., Kim, T.-K., & Gu, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6220 · Cited by 117
PowerNet: Multi-Agent Deep Reinforcement Learning for Scalable Powergrid Control
Chen, D., Chen, K., Li, Z., Chu, T., Yao, R., Qiu, F., & Lin, K. (2021) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2021.3100898 · Cited by 107
Self-citationOvertaking Maneuvers in Simulated Highway Driving using Deep Reinforcement Learning
Kaushik, M., Prasad, V., Krishna, K. M., & Ravindran, B. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500718 · Cited by 87
A Feasibility Study on Vehicle-to-Infrastructure Communication: WiFi vs. WiMAX
Chou, C., Li, C., Chien, W.-M., & Lan, K.-C. (2009) · IEEE International Conference on Mobile Data Management (MDM) · doi:10.1109/mdm.2009.127 · Cited by 84
Deep Multi Agent Reinforcement Learning for Autonomous Driving
Bhalla, S., Subramanian, S. G., & Crowley, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-47358-7_7 · Cited by 84
Anti-Jerk On-Ramp Merging Using Deep Reinforcement Learning
Lin, Y. S., McPhee, J., & Azad, N. L. (2020) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv47402.2020.9304647 · Cited by 81
Chen, D., Jiang, L., Wang, Y., & Li, Z. (2020) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc45564.2020.9147626 · Cited by 77
Self-citationShapley Q-value:: A Local Reward Approach to Solve Global Reward Games
Wang, J., Zhang, Y., Kim, T., & Gu, Y. (2020) · Research Explorer (The University of Manchester) · Cited by 69
Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging
Lubars, J., Gupta, H., Chinchali, S., Li, L., Raja, A., Srikant, R., & Wu, X. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9564954 · Cited by 61
Ha, P. Y. J., Chen, S., Dong, J., Du, R., Li, Y., & Labi, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05436 · Cited by 58
Dong, J., Chen, S., Ha, P. Y. J., Li, Y., & Labi, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05437 · Cited by 49
Safe Reinforcement Learning via Shielding
Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., & Topcu, U. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11797 · Cited by 37
Safe Multi-Agent Reinforcement Learning through Decentralized Multiple Control Barrier FunctionsPreprint
Cai, Z., Cao, H., Lu, W., Zhang, L., & Xiong, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12553 · Cited by 34
An Explicit Decision Tree Approach for Automated Driving
Li, N., Chen, H., Kolmanovsky, I. V., & Girard, A. (2017) · ASME Dynamic Systems and Control Conference (DSCC) · doi:10.1115/dscc2017-5099 · Cited by 29
SKFAC: Training Neural Networks with Faster Kronecker-Factored Approximate Curvature
Tang, Z., Jiang, F., Gong, M., Li, H., Wu, Y., Yu, F., Wang, Z., & Wang, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01327 · Cited by 12
Parameter Sharing Reinforcement Learning Architecture for Multi Agent Driving BehaviorsPreprint
Kaushik, M., Phaniteja, S., & Krishna, K. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07214 · Cited by 10
Cited by
Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives
Teng, S., Hu, X., Deng, P., Li, B., Li, Y., Ai, Y., Yang, D., Li, L., Xuanyuan, Z., Zhu, F., et al. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3274536 · Cited by 590
Hua, M., Qi, X., Chen, D., Jiang, K., Liu, Z. E., Sun, H., Zhou, Q., & Xu, H. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3574280 · Cited by 88
“In the context of CAV control, MARL has been increasingly adopted to optimize collective behaviors, o ff ering a scalable and cooperative solution to real-world deployment challenges [30].”
InfluentialSelf-citationPrivacy-Preserving Data-Enabled Predictive Leading Cruise Control in Mixed Traffic
Zhang, K., Chen, K., Li, Z., Chen, J., & Zheng, Y. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3329484 · Cited by 24
Self-citation
Metadata
- Authors
- Chen, D., Hajidavalloo, M. R., Li, Z., Chen, K., Wang, Y., Jiang, L., & Wang, Y.
- Venue
- IEEE Transactions on Intelligent Transportation Systems
- Published
- 2023-11
- Volume
- 24
- Issue
- 11
- Pages
- 11623-11638
- ISSN
- 1524-9050 (ISSN-L)
- 1558-0016
- OpenAlex
- W4381733136
- Cited by
- 257
- References
- 52
- FWCI
- 17.72
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.