CoChat
Sign Up
Cited by 21Traffic control and managementTraffic Prediction and Management TechniquesTransportation Planning and Optimization

The DOI matches its Crossref record.

A critical review of traffic signal control and a novel unified view of reinforcement learning and model predictive control approaches for adaptive traffic signal control

Open access
Wang, Xiaoyu, Abdulhai, Baher, Sanner, Scott·2023·Handbook on Artificial Intelligence and Transport
DOI 10.4337/9781803929545.00029
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent years have witnessed substantial growth in adaptive traffic signal control (ATSC) methodologies that improve transportation network efficiency, especially in branches leveraging artificial intelligence based optimization and control algorithms such as reinforcement learning as well as conventional model predictive control. However, lack of cross-domain analysis and comparison of the effectiveness of applied methods in ATSC research limits our understanding of existing challenges and research directions. This chapter proposes a novel unified view of modern ATSCs to identify common ground as well as differences and shortcomings of existing methodologies with the ultimate goal to facilitate cross-fertilization and advance the state-of-the-art. The unified view applies the mathematical language of the Markov decision process, describes the process of controller design from both the world (problem) and solution modeling perspectives. The unified view also analyses systematic issues commonly ignored in existing studies and suggests future potential directions to resolve these issues.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.4337/9781803929545.00029

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 84

  1. Abdulhai, B., & Kattan, L. (2003). "Reinforcement learning: Introduction to theory and potential for transport applications", Canadian Journal of Civil Engineering, 30(6), 981-991.
  2. Abdulhai, B., Pringle, R., & Karakoulas, G. J. (2003). "Reinforcement learning for true adaptive traffic signal control", Journal of Transportation Engineering, 129(3), 278-285.
  3. Amos, B., Jimenez, I., Sacks, J., Boots, B., & Kolter, J. Z. (2018). Differentiable mpc for end-to-end planning and control. Advances in neural information processing systems, 31.
  4. Anderson, L., Pumir, T., Triantafyllos, D., & Bayen, A. M. (2018). Stability and implementation of a cycle-based max pressure controller for signalized traffic networks. Networks & Heterogeneous Media, 13(2), 241.
  5. Araya-López, M. et al. (2010) "A closer look at MOMDPs", in 2010 22nd IEEE International Conference on Tools with Artificial Intelligence. IEEE, pp. 197-204.
  6. Auer, P. (2002) "Using confidence bounds for exploitation-exploration trade-offs", Journal of Machine Learning Research, 3(Nov), pp. 397-422.
  7. Baird, L. (1995) "Residual algorithms: Reinforcement learning with function approximation", in Machine Learning Proceedings 1995. Elsevier, pp. 30-37.
  8. Ding, B. (2018) "Open-loop optimization and closed-loop optimization in synthesis approaches", in Modern Predictive Control. CRC Press, pp. 221-245.
  9. Barth-Maron, G. et al. (2018) "Distributed Distributional Deterministic Policy Gradients", in International Conference on Learning Representations.
  10. Bernstein, D. S. et al. (2002) "The complexity of decentralized control of Markov decision processes", Mathematics of operations research. INFORMS, 27(4), pp. 819-840.
  11. Boutilier, C., Dean, T. en Hanks, S. (1999) "Decision-theoretic planning: Structural assumptions and computational leverage", Journal of Artificial Intelligence Research, 11, pp. 1-94.
  12. Bretherton, R. D. (1990) "SCOOT Urban Traffic Control System-Philosophy and Evaluation", IFAC Proceedings Volumes, 23(2), pp. 237-239.
  13. Buntine, W. L. (1994) "Operations for learning with graphical models", Journal of artificial intelligence research, 2, pp. 159-225.
  14. Cai, C., Wong, C. K. en Heydecker, B. G. (2009) "Adaptive traffic signal control using approximate dynamic programming", Transportation Research Part C: Emerging Technologies. Elsevier, 17(5), pp. 456-474.
  15. Chen, S. en Sun, D. J. (2016) "An improved adaptive signal control method for isolated signalized intersection based on dynamic programming", IEEE Intelligent Transportation Systems Magazine. IEEE, 8(4), pp. 4-14.
  16. Chiou, S.-W. (2019) "A two-stage model for period-dependent traffic signal control in a road networked system with stochastic travel demand", Information Sciences. Elsevier, 476, pp. 256-273.
  17. Christen, S. et al. (2021) "Learning functionally decomposed hierarchies for continuous control tasks with path planning", IEEE Robotics and Automation Letters. IEEE, 6(2), pp. 3623-3630.
  18. Devailly, F.-X., Larocque, D. en Charlin, L. (2021) "IG-RL: Inductive graph reinforcement learning for massive-scale traffic signal control", IEEE Transactions on Intelligent Transportation Systems. IEEE.
  19. Dey, K. C. et al. (2016) "Vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication in a heterogeneous wireless network--Performance evaluation", Transportation Research Part C: Emerging Technologies. Elsevier, 68, pp. 168-184.
  20. Diakaki, C., Papageorgiou, M. en Aboudolas, K. (2002) "A multivariable regulator approach to traffic-responsive network-wide signal control", Control Engineering Practice. Elsevier, 10(2), pp. 183-195.
  21. El-Tantawy, S., Abdulhai, B. en Abdelgawad, H. (2013) "Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): methodology and large-scale application on downtown Toronto", IEEE Transactions on Intelligent Transportation Systems. IEEE, 14(3), pp. 1140-1150.
  22. Farges, J.-L., Khoudour, I. en Lesort, J. B. (1994) "PRODYN: on site evaluation", in Third International Conference on Road Traffic Control, 1990. IET, pp. 62-66.
  23. Foerster, J. et al. (2018) "Counterfactual multi-agent policy gradients", in Proceedings of the AAAI Conference on Artificial Intelligence.
  24. Gartner, N. H. et al. (1991) "A multi-band approach to arterial traffic signal optimization", Transportation Research Part B: Methodological. Elsevier, 25(1), pp. 55-74.
  25. Genders, W. en Razavi, S. (2016) "Using a deep reinforcement learning agent for traffic signal control", arXiv preprint arXiv:1611. 01142.
  26. Guestrin, C., Koller, D. en Parr, R. (2001) "Multiagent Planning with Factored MDPs", in NIPS, pp. 1523-1530.
  27. Guestrin, C. et al. (2003) "Efficient solution algorithms for factored MDPs", Journal of Artificial Intelligence Research, 19, pp. 399- 468.
  28. Hao, B. et al. (2019) "Bootstrapping Upper Confidence Bound", Advances in Neural Information Processing Systems, 32, pp. 12123- 12133.
  29. Henry, J.-J., Farges, J. L. en Tuffal, J. (1984) "The PRODYN real time traffic algorithm", in Control in Transportation Systems. Elsevier, pp. 305-310.
  30. Hessel, M. et al. (2018) "Rainbow: Combining improvements in deep reinforcement learning", in Thirty-second AAAI conference on artificial intelligence.
  31. Horgan, D. et al. (2018) "Distributed prioritized experience replay", arXiv preprint arXiv:1803. 00933.
  32. Jaggi, P. et al. (2021). "Microscopic Model-Based RL Approaches for Traffic Signal Control Generalize Better than Model-Free RL Approaches", In 2021 IEEE International Intelligent Transportation Systems Conference, ITSC, 2021. Indianapolis, IN, USA, 19-22 September. IEEE, pp. 2525-2532.
  33. Kaelbling, L. P., Littman, M. L. en Cassandra, A. R. (1998) "Planning and acting in partially observable stochastic domains", Artificial intelligence. Elsevier, 101(1-2), pp. 99-134.
  34. Kong, X. et al. (2011) "Urban arterial traffic two-direction green wave intelligent coordination control technique and its application", International Journal of Control, Automation and Systems. Springer, 9(1), pp. 60-68.
  35. Kotsialos, A. et al. (2002) "Coordinated and integrated control of motorway networks via non-linear optimal control", Transportation Research Part C: Emerging Technologies. Elsevier, 10(1), pp. 65-84.
  36. Kouvelas, A., Triantafyllos, D. en Geroliminis, N. (2018) "Two-layer hierarchical control for large-scale urban traffic networks", in 2018 European Control Conference (ECC). IEEE, pp. 1295-1300.
  37. Lillicrap, T. P. et al. (2016) "Continuous control with deep reinforcement learning", in ICLR (Poster).
  38. Little, J. D. C., Kelson, M. D. en Gartner, N. H. (1981) "MAXBAND: A program for setting signals on arteries and triangular networks", Transportation Research Record 795, pp. 40-46.
  39. MacGowan, J., Fullerton, I. J. (1979) "Development and Testing of Advanced Control Strategies in the Urban Traffic Control Systems", Public Roads, 43(3), pp. 97-105.
  40. Mauro, V. en Di Taranto, C. (1990) "UTOPIA", IFAC Proceedings Volumes. Elsevier, 23(2), pp. 245-252.
  41. Meckl, P. H. en Kinceler, R. (1994) "Robust motion control of flexible systems using feedforward forcing functions", IEEE Transactions on Control Systems Technology. IEEE, 2(3), pp. 245-254.
  42. Melo, F. S. (2001) "Convergence of Q-learning: A simple proof", Institute of Systems and Robotics, Tech. Rep, pp. 1-4.
  43. Mirchandani, P. en Head, L. (2001) "A real-time traffic signal control system: architecture, algorithms, and analysis", Transportation Research Part C: Emerging Technologies. Elsevier, 9(6), pp. 415-432.
  44. Mnih, V. et al. (2016) "Asynchronous methods for deep reinforcement learning", in International conference on machine learning. PMLR, pp. 1928-1937.
  45. Negenborn, R. R. en Maestre, J. M. (2014) "Distributed model predictive control: An overview and roadmap of future research opportunities", IEEE Control Systems Magazine. IEEE, 34(4), pp. 87-97.
  46. Nishi, T. et al. (2018) "Traffic Signal Control Based on Reinforcement Learning with Graph Convolutional Neural Nets", in 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp. 877-883.
  47. Ong, S. C. W. et al. (2010) "Planning under uncertainty for robotic tasks with mixed observability", The International Journal of Robotics Research. SAGE Publications Sage UK: London, England, 29(8), pp. 1053-1068.
  48. Papageorgiou, M. (1998) "Automatic control methods in traffic and transportation", in Operations research and decision aid methodologies in traffic and transportation management. Springer, pp. 46-83.
  49. Papageorgiou, M. et al. (1991) "ALINEA: A local feedback control law for on-ramp metering", Transportation research record, 1320(1), pp. 58-67.
  50. Raghavan, A. et al. (2012) "Planning in factored action spaces with symbolic dynamic programming", in Twenty-Sixth AAAI Conference on Artificial Intelligence.
  51. Rashid, T. et al. (2018) "QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning", in International Conference on Machine Learning. PMLR, pp. 4295-4304.
  52. Robson, J. M. (1983) "The complexity of Go", in Proc. 9th World Computer Congress on Information Processing, 1983, pp. 413- 417.
  53. Roess, R. P., Prassas, E. S. en McShane, W. R. (2004) Traffic engineering. Pearson/Prentice Hall.
  54. Roy, N., Gordon, G. en Thrun, S. (2005) "Finding approximate POMDP solutions through belief compression", Journal of artificial intelligence research, 23, pp. 1-40.
  55. Saunders, J. (2014) "Transportation Planning as Infrastructural Fix: Regulating Traffic Congestion in the Greater Toronto and Hamilton Area", in The Organization of Transport. Routledge, pp. 207-227.
  56. Schulman, J. et al. (2015) "Trust region policy optimization", in International Conference on Machine Learning, pp. 1889-1897.
  57. Schulman, J. et al. (2017) "Proximal policy optimization algorithms", arXiv preprint arXiv:1707. 06347.
  58. Shabestary, S. M. A., and Abdulhai, B. (2018). "Deep learning vs. discrete reinforcement learning for adaptive traffic signal control", In 2018 21st International Conference on Intelligent Transportation Systems, ITSC, 2014, Hawaii, USA, November. IEEE, pp. 286- 293.
  59. Silver, D. et al. (2016) "Mastering the game of Go with deep neural networks and tree search", Nature. Nature Publishing Group, 529(7587), pp. 484-489.
  60. Silver, D. et al. (2017) "Mastering the game of go without human knowledge", Nature. Nature Publishing Group, 550(7676), pp. 354-359.
  61. Sims, A. G. en Dobinson, K. W. (1979) "SCAT the Sydney coordinated adaptive traffic system: philosophy and benefits", in International Symposium on Traffic Control Systems, 1979, Berkeley, California, USA.
  62. Smith, S. F. et al. (2013) "Smart urban signal networks: Initial application of the surtrac adaptive traffic signal control system", in Twenty-third international conference on automated planning and scheduling.
  63. Son, K. et al. (2019) "QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning", in International Conference on Machine Learning. PMLR, pp. 5887-5896.
  64. Sunehag, P. et al. (2018) "Value-Decomposition Networks for Cooperative Multi-Agent Learning Based On Team Reward", in AAMAS, pp. 2085-2087.
  65. Sutton, R. S. (1991) "Dyna, an integrated architecture for learning, planning, and reacting", ACM Sigart Bulletin. ACM New York, NY, USA, 2(4), pp. 160-163.
  66. Tan, T. et al. (2019a) "Cooperative deep reinforcement learning for large-scale traffic grid signal control", IEEE transactions on cybernetics. IEEE, 50(6), pp. 2687-2700.
  67. Tang, Z. et al. (2019) "Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification", in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8797-8806.
  68. Varaiya, P. (2013) "The max-pressure controller for arbitrary networks of signalized intersections", in Advances in Dynamic Network Modeling in Complex Transportation Systems. Springer, pp. 27-66.
  69. Vinyals, O. et al. (2019) "Grandmaster level in StarCraft II using multi-agent reinforcement learning", Nature. Nature Publishing Group, 575(7782), pp. 350-354.
  70. Wang, T. et al. (2020) "ROMA: Multi-Agent Reinforcement Learning with Emergent Roles", in Proceedings of the 37th International Conference on Machine Learning.
  71. Wei, H., Chen, C., et al. (2019) "PressLight: Learning Max Pressure Control to Coordinate Traffic Signals in Arterial Network", in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, pp. 1290-1298.
  72. Wei, H., Xu, N., et al. (2019) "CoLight: Learning network-level cooperation for traffic signal control", in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1913-1922.
  73. Wei, H., Zheng, G., et al. (2019) "A survey on traffic signal control methods", arXiv preprint arXiv:1904. 08117.
  74. Wei, H. et al. (2018) "IntelliLight: A reinforcement learning approach for intelligent traffic light control", in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, pp. 2496-2505.
  75. Williams, R. J. (1992) "Simple statistical gradient-following algorithms for connectionist reinforcement learning", Machine learning. Springer, 8(3), pp. 229-256.
  76. Xi, Y. en Li, D. (2019) Predictive Control: Fundamentals and Developments. John Wiley & Sons.
  77. Xie, X.-F. et al. (2012) "Schedule-driven intersection control", Transportation Research Part C: Emerging Technologies. Elsevier, 24, pp. 168-189.
  78. Xu, B. et al. (2021) "Hierarchically and Cooperatively Learning Traffic Signal Control", in Proceedings of the AAAI Conference on Artificial Intelligence, pp. 669-677.
  79. Xu, M. et al. (2020) "Network-wide traffic signal control based on the discovery of critical nodes and deep reinforcement learning", Journal of Intelligent Transportation Systems. Taylor & Francis, 24(1), pp. 1-10.
  80. Yao, Z. et al. (2019) "A Dynamic Predictive Traffic Signal Control Framework in a Cross-Sectional Vehicle Infrastructure Integration Environment", IEEE Transactions on Intelligent Transportation Systems. IEEE.
  81. Ye, B.-L. et al. (2019) "A survey of model predictive control methods for traffic signal control", IEEE/CAA Journal of Automatica Sinica. IEEE, 6(3), pp. 623-640.
  82. Yu, C. et al. (2020) "The Power of Predictions in Online Control", Advances in Neural Information Processing Systems, 33.
  83. Zhang, Z., Yang, J. en Zha, H. (2020) "Integrating Independent and Centralized Multi-agent Reinforcement Learning for Traffic Signal Network Optimization", in Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2083-2085.
  84. Zhao, D., Dai, Y. en Zhang, Z. (2011) "Computational intelligence in urban traffic signal control: A survey", IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). IEEE, 42(4), pp. 485-494.

Cited by

Sorted by citations

See all 21

Related

See all

Metadata

Authors
Wang, X., Abdulhai, B., & Sanner, S.
ORCID
Venue
Handbook on Artificial Intelligence and Transport
Publisher
Edward Elgar Publishing
Published
2023-10-13
Pages
482-532
DOI
10.4337/9781803929545.00029
OpenAlex
W4388271065
Cited by
20
References
0
Citations by year
2026: 4 · 2025: 11 · 2024: 5
FWCI
19.27
Type
book chapter
Language
en
Open access
Locations
  • Edward Elgar Publishing eBooks (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Traffic control and management
  • Social Sciences › Transportation › Traffic Prediction and Management Techniques
  • Social Sciences › Transportation › Transportation Planning and Optimization
Keywords
adaptive traffic signal control, reinforcement learning, model predictive control, Markov decision process, controller design, cross-domain comparison, traffic signal control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A critical review of traffic signal control and a novel unified view of reinforcement learning and model predictive control approaches for adaptive traffic signal control. https://app.cochat.ai/papers/10.4337/9781803929545.00029
The paper
Wang, X., Abdulhai, B., & Sanner, S. (2023). A critical review of traffic signal control and a novel unified view of reinforcement learning and model predictive control approaches for adaptive traffic signal control. Handbook on Artificial Intelligence and Transport. https://doi.org/10.4337/9781803929545.00029