AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
Abstract
End-to-end models for self-driving hold the promise of learning complex behaviors directly from sensor data, but face critical challenges in safety and handling long-tail events, and Reinforcement Learning (RL) offers a promising path to overcome these limitations. We identify a fundamental flaw: a deep-seated “optimistic bias” in the world models used for RL. Trained exclusively on safe expert data, these models fail to predict the consequences of erroneous actions. To address this, we introduce a framework for posttraining policy refinement built around an Impartial World Model, which teaches this model to be honest about danger. We achieve this with a novel data synthesis pipeline, Counterfactual Synthesis, which systematically generates a rich curriculum of plausible collisions and off-road events. This transforms the model from a passive scene completer into a veridical forecaster that remains faithful to the causal link between actions and outcomes. We then integrate this Impartial World Model into our closed-loop RL framework, where it serves as an internal critic. During refinement, the agent queries the critic to “dream” of the outcomes for candidate actions. We demonstrate through extensive experiments, including on a new Risk Foreseeing Benchmark, that our model significantly outperforms baselines in predicting failures. Consequently, when used as a critic, it enables a substantial reduction in safety violations in challenging simulations, proving that teaching a model to dream of danger is a critical step towards building truly safe and intelligent autonomous agents.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · see citing works
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00252 · Cited by 3,437 · see citing works
Self-citationTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · see citing works
Schubert, E., Sander, J., Ester, M., Kriegel, H., & Xu, X. (2017) · ACM Transactions on Database Systems · doi:10.1145/3068335 · Cited by 2,789 · see citing works
Planning-oriented Autonomous Driving
Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01712 · Cited by 969 · see citing works
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving
Chitta, K., Prakash, A., Jaeger, B., Yu, Z., Renz, K., & Geiger, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3200245 · Cited by 528 · see citing works
VAD: Vectorized Scene Representation for Efficient Autonomous Driving
Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q., Liu, W., Huang, C., & Wang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00766 · Cited by 353 · see citing works
MagicDrive: Street View Generation with Diverse 3D Geometry Control
Gao, R., Chen, K., Xie, E.-Z., Hong, L.-Q., Li, Z.-G., Yeung, D.-Y., & Xu, Q. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.02601 · Cited by 323
DriveDreamer: Towards Real-World-Drive World Models for Autonomous Driving
Wang, X., Zhu, Z. H., Huang, G., Chen, X., Zhu, J., & Lu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73195-2_4 · Cited by 184 · see citing works
ADriver-I: A General World Model for Autonomous DrivingPreprint
Jia, F., Mao, W.-X., Liu, Y.-F., Zhao, Y.-C., Wen, Y.-Q., Zhang, C., Zhang, X.-Y., & Wang, T.-C. (2023) · arXiv.org · doi:10.48550/arxiv.2311.13549 · Cited by 128 · see citing works
Self-citationDriveArena: A Closed-Loop Generative Simulation Platform for Autonomous Driving
Yang, X.-M., Wen, L.-C., Wei, T., Ma, Y.-K., Mei, J.-B., Li, X., Lei, W., Fu, D., Cai, P.-L., Dou, M., et al. (2024) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.02500 · Cited by 115
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving
Liao, B., Chen, S., Yin, H., Jiang, B., Wang, C., Yan, S., Zhang, X., Li, X., Zhang, Y., Zhang, Q., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01124 · Cited by 112 · see citing works
Self-citationOcc3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
Tian, X., Jiang, T., Yun, L., Mao, Y., Yang, H., Wang, Y., Wang, Y., & Zhao, H. (2023) · neural information processing systems · doi:10.52202/075280-2809 · Cited by 102 · see citing works
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72624-8_4 · Cited by 93 · see citing works
3D and 4D World Modeling: A Survey
Kong, L.-D., Yang, W., Mei, J.-B., Liu, Y.-Q., Liang, A., Zhu, D.-K., Lu, D.-Y., Yin, W., Hu, X., Jia, M.-K., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.07996 · Cited by 74
Self-citationDriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X., & Wang, X. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i10.33130 · Cited by 71 · see citing works
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
Dauner, D., Hallgarten, M., Li, T., Weng, X., Huang, Z., Huang, Z., Li, H., Gilitschenski, I., Ivanovic, B., Pavone, M., et al. (2024) · neural information processing systems · doi:10.52202/079017-0902 · Cited by 62 · see citing works
What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models
Deng, T., Pan, Y., Yuan, S., Li, D., Wang, C., Li, M., Chen, L., Xie, L.-H., Wang, D.-W., Wang, J.-C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2512.03422 · Cited by 51
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., & Li, H. (2024) · neural information processing systems · doi:10.52202/079017-2906 · Cited by 49 · see citing works
Li, Q., Jia, X., Wang, S., & Yan, J. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72995-9_9 · Cited by 42 · see citing works
MCN-SLAM: Multi-Agent Collaborative Neural SLAM with Hybrid Implicit Neural Scene Representation
Deng, T., Shen, G.-L., Chen, X., Yuan, S., Shen, H., Peng, G., Wu, Z.-Y., Wang, J.-C., Xie, L.-H., Wang, D.-W., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.18678 · Cited by 40
Finetuning Generative Trajectory Model with Reinforcement Learning from Human Feedback
Li, D., Ren, J., Wang, Y., Wen, X., Li, P.-X., Xu, L.-M., Zhan, K., Xia, Z.-P., Jia, P., Lang, X.-P., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2503.10434 · Cited by 34
Self-citationXing, Z., Zhang, X., Hu, Y., et al. (2025) · doi:10.1109/cvpr52734.2025.00157 · Cited by 30 · see citing works
Self-citationEnd-to-End Driving with Online Trajectory Evaluation via BEV World Model
Li, Y., Wang, Y., Liu, Y., He, J., Fan, L., & Zhang, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02519 · Cited by 27 · see citing works
Self-citationWorld4Drive: End-to-End Autonomous Driving via Intention-Aware Physical Latent World Model
Zheng, Y., Yang, P., Xing, Z., Zhang, Q., Zheng, Y., Gao, Y., Li, P., Zhang, T. L., Xia, Z., Jia, P., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02659 · Cited by 21 · see citing works
Epona: Autoregressive Diffusion World Model for Autonomous Driving
Zhang, K., Tang, Z., Hu, X., Pan, X., Guo, X., Liu, Y., Huang, J., Yuan, L., Zhang, Q., Long, X., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02527 · Cited by 20 · see citing works
MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control
Gao, R., Chen, K., Xiao, B., Hong, L., Li, Z., & Xu, Q. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02612 · Cited by 19 · see citing works
MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control
Gao, R., Chen, K., Xiao, B., Hong, L., Li, Z., & Xu, Q. (2024) · arXiv · doi:10.48550/arxiv.2411.13807 · Cited by 16
RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
Gao, H., Chen, S., Jiang, B., Liao, B., Shi, Y., Guo, X., Pu, Y., yin, H., Li, X., Zhang, X., et al. (2025) · neural information processing systems · doi:10.52202/085713-1092 · Cited by 12 · see citing works
Self-citationChen, Y., Wang, Y., & Zhang, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02496 · Cited by 11 · see citing works
DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes
Bian, H., Kong, L., Xie, H., Pan, L., Qiao, Y., & Liu, Z.-W. (2024) · arXiv.org · doi:10.48550/arxiv.2410.18084 · Cited by 10
Self-citationOccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving
Wang, L., Zheng, W., Ren, Y., Jiang, H., Cui, Z., Yu, H., & Lu, J. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3687468 · Cited by 6 · see citing works
DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
Shang, S., Chen, Y., Wang, Y., Li, Y., & Zhang, Z. (2025) · neural information processing systems · doi:10.52202/085713-2724 · Cited by 6 · see citing works
Self-citationReinforced Refinement With Self-Aware Expansion for End-to-End Autonomous Driving
Liu, H., Li, T., Yang, H., Chen, L., Wang, C., Guo, K., Tian, H., Li, H. Y., Li, H., & Lv, C. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653866 · Cited by 6 · see citing works
Yang, Z., Jia, X., Li, Q., Yang, X. K., Yao, M., & Yan, J. (2025) · neural information processing systems · doi:10.52202/085713-4471 · Cited by 5 · see citing works
OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving
Yan, T., Yin, J., Lang, X., Yang, R., Xu, C., & Shen, J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i9.32987 · Cited by 4 · see citing works
Self-citationRLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation
Yan, T., Han, W., Zhou, X., Zhang, X., Zhan, K., Xu, C., & Shen, J. (2025) · neural information processing systems · doi:10.52202/085713-4286 · Cited by 3 · see citing works
Self-citationCOME: Adding Scene-Centric Forecasting Control to Occupancy World Model
Shi, Y., Jiang, K., Meng, Q., Wang, K., Wang, J., Sun, W., Wen, T., Yang, M., & Yang, D. (2025) · neural information processing systems · doi:10.52202/085713-0004 · Cited by 3 · see citing works
OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving
Tang, T., Ma, E., Zhou, X., Wang, L., Yan, T., Zhang, X., Zhan, K., Jia, P., Lang, X., Bian, J.-W., et al. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754772 · Cited by 3 · see citing works
Self-citationVADv2: End-to-End Vectorized Autonomous Driving via Probabilistic PlanningPreprint
Jiang, B., Chen, S., Gao, H., Liao, B., Zhang, Q., Liu, W., & Wang, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.13243 · Cited by 3 · see citing works
$I^{\mathbf{2}}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
Liao, Z., Wei, P., Zhang, R., Chen, S., Wang, H., & Ren, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02394 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- Gu, 2024, Dome: Taming diffusion model into high-fidelity controllable occupancy world model
- Guo, 2025, Deepseek-r1: Incentivizing reasoning capability in 1lms via reinforcement learning
- Li, 2024, Enhancing end-to-end autonomous driving with latent world model
- Li, 2025, Recogdrive: A reinforced cognitive framework for end-to-end autonomous driving
- Li, 2024, Hydra-mdp: End-to-end multimodal planning with multitarget hydra-distillation
- Liang, 2025, Worldlens: Full-spectrum evaluations of driving world models in real world
- Liao, 2025, I2-world: Intra-inter tokenization for efficient dynamic 4d scene forecasting, 25810
- Lu, 2025, Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning
- Renz, 2024, Carllava: Vision language models for camera-only closed-loop driving
- Yan, 2022, Plug-and-play pseudo label correction network for unsupervised person re-identification
- Yan, 2024, Drivingsphere: Building a high-fidelity 4d world for closedloop simulation
Cited by
Still being checked.
- Authors
- Yan, T., Tang, T., Gui, X., Li, Y., Zheng, J., Huang, W., Kong, L., Han, W., Zhou, X., Zhang, X., Zhan, Y., Zhan, K., Xu, C.-Z., & Shen, J.
- Affiliations
- University of Macau (MO)
- Sun Yat-sen University (CN)
- Huazhong University of Science and Technology (CN)
- Northwestern University (US)
- National University of Singapore (SG)
- Li Auto Inc.
- Venue
- 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF)
- Publisher
- IEEE
- Published
- 2026-06
- Pages
- 1085-1095
- OpenAlex
- W7220919905
- Semantic Scholar
- 0fb561492690b2ab0cdf0ff90e2ad745e5010f6c
- Cited by
- 0
- References
- 42
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- end-to-end autonomous driving, world models, autonomous driving
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
