The DOI matches its Crossref record; 57 of the 61 references it cites were found in a registry or an index.
MP3: A Unified Model to Map, Perceive, Predict and Plan
Open accessAbstract
High-definition maps (HD maps) are a key component of most modern self-driving systems due to their valuable semantic and geometric information. Unfortunately, building HD maps has proven hard to scale due to their cost as well as the requirements they impose in the localization system that has to work everywhere with centimeter-level accuracy. Being able to drive without an HD map would be very beneficial to scale self-driving solutions as well as to increase the failure tolerance of existing ones (e.g., if localization fails or the map is not up-to-date). Towards this goal, we propose MP3, an end-to-end approach to mapless1driving where the input is raw sensor data and a high-level command (e.g., turn left at the intersection). MP3 predicts intermediate representations in the form of an online map and the current and future state of dynamic agents, and exploits them in a novel neural motion planner to make interpretable decisions taking into account uncertainty. We show that our approach is significantly safer, more comfortable, and can follow commands better than the baselines in challenging long-term closed-loop simulations, as well as when compared to an expert driver in a large-scale real-world dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These works have shown that MTL achieves better data utilization at lower computation cost.”
From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [58]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Self-citationMulti-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
Congested traffic states in empirical observations and microscopic simulations
Treiber, M., Hennecke, A., & Helbing, D. (2000) · Physical Review E · doi:10.1103/physreve.62.1805 · Cited by 4,762
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Using occupancy grids for mobile robot perception and navigation
Elfes, A. (1989) · Computer · doi:10.1109/2.30720 · Cited by 2,287
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
ALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. (2018) · doi:10.1184/r1/6603146 · Cited by 1,369
Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
Philion, J., & Fidler, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_12 · Cited by 1,271
PIXOR: Real-time 3D Object Detection from Point Clouds
Yang, B., Luo, W., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00798 · Cited by 1,249
Self-citationOptimal trajectory generation for dynamic street scenarios in a Frenét Frame
Werling, M., Ziegler, J., Kammel, S., & Thrun, S. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509799 · Cited by 806
An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
Liu, R., Lehman, J., Molino, P., Such, F. P., Frank, E., Sergeev, A., & Yosinski, J. (2018) · arXiv · doi:10.48550/arxiv.1807.03247 · Cited by 784
Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., & Shah, A. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793742 · Cited by 694 · cited 1×
“Subsequent to the success of deep learning, direct control based methods have advanced with deeper networks, richer sensors, and scalable learning methods [5, 11, 24, 35].”
BackgroundLuo, W., Yang, B., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00376 · Cited by 644
Self-citationRatliff, N., Bagnell, J. A., & Zinkevich, M. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143936 · Cited by 636
Exploring the Limitations of Behavior Cloning for Autonomous Driving
Codevilla, F., Santana, E., López, A. M., & Gaidon, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00942 · Cited by 608
DeepRoadMapper: Extracting Road Topology from Aerial Images
Máttyus, G., Luo, W., & Urtasun, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.372 · Cited by 568
Self-citationMulti-Agent Tensor Fusion for Contextual Trajectory Prediction
Zhao, T., Xu, Y., Monfort, M., Choi, W., Baker, C. L., Zhao, Y., Wang, Y., & Wu, Y. N. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01240 · Cited by 494
Learning Occupancy Grid Maps with Forward Sensor Models
Thrun, S. (2003) · Autonomous Robots · doi:10.1023/a:1025584807625 · Cited by 483
MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior PredictionPreprint
Chai, Y., Sapp, B., Bansal, M., & Anguelov, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05449 · Cited by 482
CoverNet: Multimodal Behavior Prediction Using Trajectory Sets
Phan-Minh, T., Grigore, E. C., Boulton, F. A., Beijbom, O., & Wolff, E. M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01408 · Cited by 456
End-To-End Interpretable Neural Motion Planner
Zeng, W., Luo, W., Suo, S., Sadat, A., Yang, B., Casas, S., & Urtasun, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00886 · Cited by 423
Self-citationPolack, P., Altché, F., d’Andréa-Novel, B., & Fortelle, A. D. L. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995816 · Cited by 423
RoadTracer: Automatic Extraction of Road Networks from Aerial Images
Bastani, F., He, S., Abbar, S., Alizadeh, M., Balakrishnan, H., Chawla, S., Madden, S. R., & DeWitt, D. J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00496 · Cited by 383
LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving
Meyer, G. P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., & Wellington, C. K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01296 · Cited by 374
IntentNet: Learning to Predict Intention from Raw Sensor DataPreprint
Casas, S., Luo, W., & Urtasun, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.07907 · Cited by 316
Self-citationMultiple Futures PredictionPreprint
Tang, Y., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00997 · Cited by 265
LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World
Manivasagam, S., Wang, S., Wong, K., Zeng, W., Sazanovich, M., Tan, S., Yang, B., Ma, W.-C., & Urtasun, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01118 · Cited by 257
Self-citationEnd-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point CloudsPreprint
Zhou, Y., Sun, P., Zhang, Y., Anguelov, D., Gao, J., Ouyang, T. Y., Guo, J. C. Y., Ngiam, J., & Vasudevan, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06528 · Cited by 241
3D-LaneNet: End-to-End 3D Multiple Lane Detection
Garnett, N., Cohen, R., Pe'er, T., Lahav, R., & Levi, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00301 · Cited by 218
r2p2: A ReparameteRized Pushforward Policy for Diverse, Precise Generative Path Forecasting
Rhinehart, N., Kitani, K., & Vernaza, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_47 · Cited by 212
PnPNet: End-to-End Perception and Prediction With Tracking in the Loop
Liang, M., Yang, B., Zeng, W., Chen, Y., Hu, R., Casas, S., & Urtasun, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01157 · Cited by 209
Self-citationWu, P., Chen, S., & Metaxas, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01140 · Cited by 201
Driving Policy Transfer via Modularity and AbstractionPreprint
Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.09364 · Cited by 200
The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
Liang, J., Jiang, L., Murphy, K. P., Yu, T., & Hauptmann, A. G. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01052 · Cited by 189
Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations
Sadat, A., Casas, S., Ren, M., Wu, X., Dhawan, P., & Urtasun, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58592-1_25 · Cited by 183
Self-citationSpAGNN: Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data
Casas, S., Gulino, C., Liao, R., & Urtasun, R. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196697 · Cited by 181
Self-citationHoermann, S., Bach, M., & Dietmayer, K. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460874 · Cited by 171
Urban Driving with Conditional Imitation Learning
Hawke, J., Shen, R., Gurău, C., Sharma, S., Reda, D., Nikolov, N., Mazur, P., Micklethwaite, S., Griffiths, N., Shah, A., et al. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197408 · Cited by 158
Implicit Latent Variable Model for Scene-Consistent Motion Forecasting
Casas, S., Gulino, C., Suo, S., Luo, K. Z., Liao, R., & Urtasun, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58592-1_37 · Cited by 152
Self-citationEnd-to-end Contextual Perception and Prediction with Interaction Transformer
Li, L. L., Yang, B., Liang, M., Zeng, W., Ren, M., Segal, S., & Urtasun, R. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341392 · Cited by 133
Self-citationEnd-to-End Driving Via Conditional Imitation Learning
Codevilla, F., Muller, M., Lopez, A., Koltun, V., & Dosovitskiy, A. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460487 · Cited by 129
Gen-LaneNet: A Generalized and Scalable Approach for 3D Lane Detection
Guo, Y., Chen, G., Zhao, P., Zhang, W., Miao, J., Wang, J., & Choe, T. E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_40 · Cited by 127
DAGMapper: Learning to Map by Discovering Lane Topology
Homayounfar, N., Liang, J., Ma, W.-C., Fan, J., Wu, X., & Urtasun, R. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00300 · Cited by 109
Self-citationSearch-Based Optimal Motion Planning for Automated Driving
Ajanović, Z., Lačević, B., Shyrokau, B. N., Stolz, M., & Horn, M. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593813 · Cited by 108
Deep Multi-Sensor Lane Detection
Bai, M., Máttyus, G., Homayounfar, N., Wang, S., Lakshmikanth, S. K., & Urtasun, R. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594388 · Cited by 102
Self-citationJointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles
Sadat, A., Ren, M., Pokrovsky, A., Lin, Y.-C., Yumer, E., & Urtasun, R. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967615 · Cited by 87
Self-citationHierarchical Recurrent Attention Networks for Structured Online Maps
Homayounfar, N., Ma, W.-C., Lakshmikanth, S. K., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00360 · Cited by 71
Self-citationDiscrete Residual Flow for Probabilistic Pedestrian Behavior PredictionPreprint
Jain, A., Casas, S., Liao, R., Xiong, Y., Feng, S., Segal, S., & Urtasun, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08041 · Cited by 64
Self-citationDeep Imitative Models for Flexible Inference, Planning, and ControlPreprint
Rhinehart, N., McAllister, R., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06544 · Cited by 56
Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data
Casas, S., Gulino, C., Liao, R., & Urtasun, R. (2019) · arXiv · doi:10.48550/arxiv.1910.08233 · Cited by 44
Self-citationThe Importance of Prior Knowledge in Precise Multimodal Prediction
Casas, S., Gulino, C., Suo, S., & Urtasun, R. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341199 · Cited by 43
Self-citationLearning to Predict Ego-Vehicle Poses for Sampling-Based Nonholonomic Motion Planning
Banzhaf, H., Sanzenbacher, P., Baumann, U. A., & Zöllner, J. M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2893975 · Cited by 31
StrObe: Streaming Object Detection from LiDAR PacketsPreprint
Frossard, D., Suo, S., Casas, S., Tu, J., Hu, R., & Urtasun, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06425 · Cited by 15
Self-citationScene Compliant Trajectory Forecast With Agent-Centric Spatio-Temporal Grids
Ridel, D. A., Deo, N., Wolf, D. F., & Trivedi, M. M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974393 · Cited by 15
An Auto-tuning Framework for Autonomous VehiclesPreprint
Fan, H., Xia, Z., Liu, C., Chen, Y., & Kong, Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04913 · Cited by 14
Ma, H., Wang, Y., Tang, L., Kodagoda, S., & Xiong, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02468 · Cited by 11
Research works printed but not linked: 1
- zhao, 2019, Multiagent tensor fusion for contextual trajectory prediction, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 12126
Cited by
Sorted by citations
Planning-oriented Autonomous Driving
Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01712 · Cited by 1,139
End-to-End Autonomous Driving: Challenges and Frontiers
Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., & Li, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3435937 · Cited by 843
3D Object Detection for Autonomous Driving: A Comprehensive Survey
Mao, J., Shi, S., Wang, X., & Li, H. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01790-1 · Cited by 529
VAD: Vectorized Scene Representation for Efficient Autonomous Driving
Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q., Liu, W., Huang, C., & Wang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00766 · Cited by 471
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. G. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 459
Metadata
- Authors
- Casas, S., Sadat, A., & Urtasun, R.
- Affiliations
- University of Toronto (CA)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 14398-14407
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3125926538
- Cited by
- 228
- References
- 61
- Citations by year
- 2026: 17 · 2025: 49 · 2024: 59 · 2023: 59 · 2022: 34 · 2021: 10
- FWCI
- 40.11
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- autonomous vehicles, end-to-end driving, online maps, decision making under uncertainty, high-definition maps, closed-loop simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
