CoChat
Sign Up
Cited by 46Autonomous Vehicle Technology and SafetyTraffic control and managementReinforcement Learning in Robotics

Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving

Open access
Karnchanachari, Napat, Geromichalos, Dimitris, Tan, Kok Seang, Li, Nanxiang, Eriksen, Christopher, Yaghoubi, Shakiba, Mehdipour, Noushin, Bernasconi, Gianmarco +3 more·2024·2024 IEEE International Conference on Robotics and Automation (ICRA)
DOI 10.1109/icra57147.2024.10610077
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine Learning (ML) has replaced handcrafted methods for perception and prediction in autonomous vehicles. Yet for the equally important planning task, the adoption of ML-based techniques is slow. We present nuPlan, the world’s first real-world autonomous driving dataset and benchmark. The benchmark is designed to test the ability of ML-based planners to handle diverse driving situations and to make safe and efficient decisions. We introduce a new large-scale dataset that consists of 1282 hours of diverse driving scenarios from 4 cities (Las Vegas, Boston, Pittsburgh, and Singapore) and includes high-quality auto-labeled object tracks and traffic light data. We mine and taxonomize common & rare driving scenarios which are used during evaluation to get fine-grained insights into the performance and characteristics of a planner. Beyond the dataset, we provide a simulation and evaluation framework that enables a planner’s actions to be simulated in closed-loop to account for interactions with other traffic participants. We present a detailed analysis of numerous baselines and investigate gaps between ML-based and traditional methods. Find the nuPlan dataset and code at nuplan.org.

AI snapshot

Written from the paper’s full text.

TL;DR

nuPlan is presented as the first real-world autonomous-driving dataset and benchmark for planning, with 1282 hours from 4 cities, 128 hours of sensor data, scenario mining, and a closed-loop simulation/evaluation framework. Experiments show supervised ML planners excel in open-loop but rule-based and hybrid planners remain stronger in closed-loop, with hybrid methods most promising in the nuPlan challenge.

Key findings

  • The dataset contains 1282 hours of driving from Las Vegas, Boston, Pittsburgh, and Singapore, plus 128 hours of released sensor data, 73 unique scenario types, and auto-labeled object tracks and traffic-light statuses.
  • In baseline evaluations, UrbanDriver achieved the best open-loop score (0.90), while IDM achieved the best closed-loop non-reactive (0.73) and closed-loop reactive (0.76) scores; Simple Planner scored 0.22/0.32/0.37 and Raster Planner 0.52/0.47/0.46 across open-loop, closed-loop non-reactive, and closed-loop reactive.
  • Generalization was weak: UrbanDriver trained only on Las Vegas dropped by 53.8% in open-loop, 35.1% in closed-loop non-reactive, and 41.5% in closed-loop reactive when tested on other cities; Singapore was worst, attributed to left-hand traffic.
  • The first nuPlan challenge’s top overall planner was CS Tu (rule-based + ML refinement) at 0.895, followed by AutoHorizon (ML + optimizer) at 0.875, Pegasus (ML + collision checking) at 0.848, and AID (ML + hierarchical game theory) at 0.829.

Methods

  • Data were collected manually by vehicle operators in Chrysler Pacifica PHEVs across 4 cities, with lidar, camera, GNSS, and IMU sensors; heavy rain and night data were excluded.
  • Auto-labeling used an offline perception pipeline with 3D object detection, offline tracking, and global track refinement, plus motion-based traffic-light status inference and scenario mining from low-level attributes.
  • Evaluation used open-loop metrics (ADE, FDE, AHE, FHE, MR) and closed-loop metrics (collisions, drivable area, progress, driving direction, TTC, speed limit, comfort), with closed-loop simulation using log-replay or IDM reactive agents and an LQR plus kinematic bicycle controller.

Limitations

  • The paper states that heavy rain and night data were excluded because they would impact perception quality; reactive agents are only vehicles, while vulnerable road users are replayed from logs because their behavior is hard to model; and the IDM reactive agents may give the IDM planner an unfair advantage in closed-loop evaluation.
  • The paper also states that traffic-light transition inference is difficult due to lag in agent movement, that motion forecasting models alone cannot achieve realistic scene-coherent traffic simulation, and that the scoring heuristic is an initial hand-tuned proposal for the dataset and simulation framework.
  • Evident from the text: scenario mining prioritizes high precision while recall is less relevant, and the introduction notes that learning-based planning lacks formal safety guarantees, making it challenging to certify.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/icra57147.2024.10610077

Cites

92 of 92 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

15 of 32 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured

  • Findingcitation

    The nuPlan dataset is less long-tailed than other datasets with more fine-grained classes.

    “While there are about 2 orders of magnitude between the most (generic object) and least (construction zone sign) common class, the dataset is nevertheless less long-tailed than other datasets with more fine-grained classes [69].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
  • Interpretationnone given

    Including points from four side lidars in addition to the top lidar prevents objects closer to the ego vehicle from being missed.

    “First, different from other works [16], [89], we propose to include the points from four side lidars in addition to the top lidar, so the objects closer to the ego vehicle will not be missed.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [16] source

      The Introduction discusses combining viewpoints across point-cloud sequences, but does not say whether the method uses only a top lidar or also four side lidars. The Dataset and detection-results sections likewise do not address whether adding side-lidar points prevents nearby objects from being missed (searched: four side lidars, top lidar, nearby objects). (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [89]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingcitation

    In the first public nuPlan challenge, the highest-scoring planner in closed-loop was mostly rule-based.

    “The highest-scoring planner in closed-loop was mostly rule-based [87].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [87] Dauner, 2023, Parting with misconceptions about learning-based vehicle motion planning source
  • Findingreasoning

    The proposed single network for global track refinement achieves position, heading, size and velocity refinement in one shot.

    “Notably, instead of using two networks for static and dynamic objects [16], or for different purposes (size, position or heading) [91], we design a single network, which achieves the above goals in one shot.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [16] source
    • cites [91]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Interpretationreasoning

    The proposed global track refinement method reduces deployment overhead compared to other works.

    “In practice, the proposed method reduces the deployment overhead compared to other works [16], [91], because the order of the two networks is not clear and the classification of dynamic/static is arbitrary for some slowmoving objects.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [16] source

      Section 4.4 describes the cited method as using motion-state classification followed by separate static and dynamic auto-labeling models, supporting the description of the work being compared. It does not assess the citing authors’ proposed method or establish that it reduces deployment overhead. It also specifies the order of the stages and reports over 99% vehicle motion-state classification accuracy, rather than describing that classification as arbitrary. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [91]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims. 29
  • Classified as probably not a substantial claim. 25
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Traffic light statuses are crucial for realistic traffic simulation, but only Waymo and Lyft contain these and only from an online perception system, rather than developing offline traffic light status inference as in nuPlan.

    “Likewise, the availability of traffic light statuses is crucial for realistic traffic simulation, but only Waymo [17] and Lyft [19] contain these and only from an online perception system, rather than developing offline traffic light status inference as in nuPlan.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [17] source

      Section 3 (Dataset; page unavailable) explicitly says the Waymo Open Motion Dataset includes traffic signal states, supporting the part attached to the Waymo citation. Section 3.3 discusses offboard perception for object tracks, but does not establish that the traffic signal states came only from an online perception system. The copy also does not establish the broader 'only Waymo and Lyft' comparison or address nuPlan's inference method. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [19] Houston, 2020, One thousand and one hours: Self-driving motion prediction dataset, CoRL source

  • May combine two claims.

    Premisecitation

    Offline perception is crucial to train and evaluate planners using high-quality object tracks, but only present in Waymo, MONA and nuPlan.

    “Offline perception [16] is crucial to train and evaluate planners using high-quality object tracks, but only present in Waymo [17], MONA [18] and nuPlan.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [16] source

      The Introduction (page unavailable) says high-quality offboard perception can help supervise or evaluate downstream modules, such as behavior prediction. The Design space section describes generating refined object tracks. The paper does not establish that offline perception is crucial for training and evaluating planners specifically; the dataset-list claim is not attributed to this citation. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [17] source
    • cites [18] source

  • May combine two claims.

    Interpretationreasoning

    The offline perception system used for auto-labeling is not constrained by latency and causality, so the fidelity of generated tracks is drastically higher than online perception systems, enabling evaluation of planning under very limited perception noise.

    “Compared with the online perception systems used in many other datasets [19], [13], the offline version is not constrained by latency and causality.Therefore, the fidelity of the generated tracks is drastically higher, enabling us to evaluate planning performance under very limited perception noise.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [19] Houston, 2020, One thousand and one hours: Self-driving motion prediction dataset, CoRL source
    • cites [13] source

      Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Only the Lyft, Shifts and nuPlan datasets provide more than 1000h of driving data, and nuPlan was the first such dataset that provides lidar and camera sensor data (128h), although Waymo later also released compressed lidar data.

    “Only the Lyft [19], Shifts [15] and nuPlan datasets provide more than 1000h of driving data and nuPlan was the first such dataset that provides lidar and camera sensor data (128h), although Waymo [17] later also released compressed lidar data.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [19] Houston, 2020, One thousand and one hours: Self-driving motion prediction dataset, CoRL source
    • cites [15] Malinin, 2021, Shifts: A dataset of real distributional shift across multiple large-scale tasks, NeurIPS Track on Datasets and Benchmarks source
    • cites [17] source

      The Dataset section describes releasing object tracks and map features, while the Offboard perception system section describes using LiDAR frames to generate tracks. Neither passage says Waymo later released compressed LiDAR data. Searches for “compressed LiDAR data,” “sensor data,” and “compress” found no passage establishing the part attributed to Waymo [17]. (LLM, conf=0.75)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Multi-task learning jointly training perception, prediction and planning achieves better data utilization at lower computation cost.

    “These works have shown that MTL achieves better data utilization at lower computation cost.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [62] source
    • cites [63] Ngiam, 2022, Scene transformer: A unified architecture for predicting future trajectories of multiple agents, ICLR source
    • cites [58] source

      The Training details (p. 13) describe multi-task training of mapping, perception and prediction, while Learning (p. 6) says the planner is trained separately after those components are frozen. Neither passage shows that this approach improves data utilization or lowers computation cost, so the claimed benefits are unsupported. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Most other datasets use a combination of manual and automated driving, which may lead the planner to imitate less desirable driving behavior.

    “Since our focus is on planning, it is crucial that we drive manually, while most other datasets [13], [14], [17], [19] use a combination of manual and automated driving, which may lead the planner to imitate less desirable driving behavior.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [13] source

      The Related Work section compares autonomous-driving datasets, and §3.3 describes trajectories mined from Argoverse driving logs. Neither passage says whether those logs combine manual and automated driving or whether such a combination could teach a planner less desirable behavior. Searches for those driving modes and planner imitation found no passage addressing the claim. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [14] Wilson, 2021, Argoverse 2: Next generation datasets for self-driving perception and forecasting, NeurIPS Track on Datasets and Benchmarks

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [17] source

      The Related Work section compares datasets by their coverage, interactions, and labeling, but does not say that most combine manual and automated driving. Searches for driving mode and imitation found no passage linking such a combination to less desirable behavior for a planner. (LLM, conf=0.73)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingnone given

    The nuPlan dataset includes high-quality auto-labeled object tracks and traffic light data.

    “includes high-quality auto-labeled object tracks and traffic light data”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Due to the grid-like road layout, most vehicles in nuPlan have orientations that are multiples of ±90°.

    “Due to the grid-like road layout, most vehicles have orientations that are multiples of ± 90 • .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The nuPlan vehicles are equipped with 5 lidars, 8 cameras, as well as GNSS & IMU.

    “The vehicles are equipped with 5 lidars, 8 cameras, as well as GNSS & IMU.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The lidars can generate lidar point clouds with up to 360k and 720k points for 20-channel and 40-channel lidars, respectively.

    “The lidars can generate lidar point clouds with up to 360k and 720k points for 20-channel and 40-channel lidars, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Reactive agents simulated with IDM can react to the ego's actions and other reactive agents, reducing false collisions and letting scenarios play out for longer.

    “This allows the agents to react to the ego's actions, as well as other reactive agents.In turn, this reduces false collisions and lets scenarios play out for longer.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In Las Vegas, most routes start and end in PUDOs, which leads to these areas being visited more often.

    “In Las Vegas, most routes start and end in PUDOs, which leads to these areas being visited more often.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Log-replay agents provide a near-perfect recreation of the recorded data in open-loop and closed-loop simulation.

    “Log-replay agents are used to simulate scenarios both in open-loop and closed-loop, a nearperfect recreation of the recorded data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most objects in nuPlan are slow-moving, which represents primarily urban scenarios.

    “We observe that most of the objects are slow-moving, which represents primarily urban scenarios, while a small number of pedestrians and bicycles are moving at unrealistically high speeds, possibly due to noisy annotations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    UrbanDriver trained purely on Las Vegas data showed an open-loop performance drop of 53.8%, a closed-loop non-reactive drop of 35.1%, and a closed-loop reactive drop of 41.5% when tested on other cities.

    “The open-loop performance dropped by 53.8%, while closed-loop non-reactive and closed-loop reactive performance dropped by 35.1% and 41.5% respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The difference between open and closed-loop scores across Singapore, Boston, and Pittsburgh is only 16.3%, while for Las Vegas it is more than double at 37.3%.

    “The difference between open and closed-loop scores across Singapore, Boston, and Pittsburgh is only 16.3%, while for Las Vegas it is more than double at 37.3%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In other cities, the spatial distribution of ego vehicles is more uniform, with key intersections being visited the most.

    “In other cities, the distribution is more uniform, with key intersections being visited the most.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    A graph can be constructed from the network of lane and lane connectors in nuPlan.

    “A graph can be constructed from the network of lane and lane connectors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There are a total of 73 unique scenario types in the nuPlan dataset.

    “We have a total of 73 unique scenario types in the dataset, and their distributions per city are shown in Fig. 11.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    LaneGCN independently predicts agents, resulting in a lack of scene cohesion, for example, predicting colliding trajectories.

    “The model independently predicts agents, resulting in a lack of scene cohesion, for example, predicting colliding trajectories.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The details and parameters of the 14 scenarios that were used to grade the nuPlan planning challenge can be found in Tab.XII.

    “The details and parameters of the 14 scenarios that were used to grade the nuPlan planning challenge can be found in Tab.XII.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    nuPlan's semantic map has 10 semantic layers and includes traffic light statuses encoded into the lane connectors.

    “Semantic map of nuPlan with 10 semantic layers in different colors, polygons and lines.It also includes traffic light statuses encoded into the lane connectors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The bounding boxes from the detection network are smoothed by a Kalman Filter based tracker, but the smoothing does not happen at a global scale.

    “Moreover, although the bounding boxes from the detection network have been smoothed to some extent by a Kalman Filter based tracker, the smoothing does not happen at a global scale.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The scenario mining approach results in high-precision scenario labels, while recall is less relevant.

    “This approach results in high-precision scenario labels, while recall is less relevant for us.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    nuPlan covers four cities.

    “Spatial data coverage for four cities in nuPlan.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    By taking the mean across the selected horizons, the errors at the beginning of the horizon will have more impact on the averaged value.

    “Additionally, by taking the mean across the selected horizons, the errors at the beginning of the horizon will have more impact on the averaged value.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The number of simulated agents had to be limited to maintain acceptable simulation runtime.

    “Furthermore, the number of simulated agents had to be limited to maintain acceptable simulation runtime.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    nuPlan sensor data includes lidar data, a map, and 6 camera views.

    “Examples of nuPlan sensor data.We can see the lidar data and map in the center and 6 camera views around it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    nuPlan is the world's first real-world autonomous driving dataset and benchmark.

    “We present nuPlan, the world's first real-world autonomous driving dataset and benchmark.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    CommonRoad does not provide a real-world dataset as the basis for simulation, instead using a small number of manually crafted scenarios and tools to import other datasets without sensor data.

    “However, CommonRoad [12] does not provide a real world dataset as the basis for the simulation, instead resorting to a small number of manually crafted scenarios and tools to import other datasets, albeit without any sensor data.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [12] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All nuPlan metric scores are normalized to [0-1] using thresholds selected based on legal requirements and natural human driving.

    “All metrics' scores are normalized to the range [0 -1] using thresholds that are selected based on legal requirements and natural human driving.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Both lidars and cameras cover the full 360-degree environment and minimize blindspots.

    “Thus both lidars and cameras cover the full 360-degree environment and minimize blindspots (Fig. 5).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Imitation learning often suffers from poor generalization where compounding error leads to driving scenarios outside the training data, known as covariate shift.

    “IL often suffers from a poor generalization where the compounding error leads to driving scenarios that are outside of the training data, known as “covariate shift” [34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] Pomerleau, 1988, Alvinn: An autonomous land vehicle in a neural network, NeurIPS source

  • May combine two claims.

    Interpretationreasoning

    The poor performance in Singapore can be explained by left-hand traffic, while the model was trained on right-hand traffic.

    “This can be explained by the lefthand traffic, while the model was trained on right-hand traffic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The dataset is stratified across days, cities, and driving scenarios to ensure all splits have similar balances across these dimensions.

    “In addition, the dataset is stratified across days, cities, and driving scenarios to ensure all splits have similar balances across these dimensions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The nuPlan controller requirement asserts kinematic feasibility and prevents users from cheating.

    “To assert kinematic feasibility and prevent users from cheating, we require the use of a controller.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Global track refinement is efficacious and necessary for producing high-quality bounding boxes.

    “In particular, when a stricter matching criterion is applied (BEV IOU = 0.9), we see a relative boost of 56.8%, which demonstrates the efficacy and necessity of global track refinement in producing high-quality bounding boxes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    In closed-loop simulation, identifying the correct lead object is not trivial.

    “However, in closed-loop simulation identifying the correct lead object is not trivial.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The autolabeling system was evaluated on an internal dataset that follows a similar data distribution to nuPlan and contains 3098 human-labeled scenes collected from Singapore, Boston, Pittsburgh and Las Vegas.

    “To evaluate the proposed autolabeling system we use an internal dataset that follows a similar data distribution to nuPlan and contains 3098 human-labeled scenes collected from Singapore, Boston, Pittsburgh and Las Vegas.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The fewer number of ID switches indicates that the tracks in nuPlan dataset are less fragmented and thus reflect the agents' movement in the real world.

    “The fewer number of ID switches indicates that the tracks in nuPlan dataset are less fragmented and thus reflect the agents' movement in the real world.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    The lead agent search will identify the oncoming agent, and the agent will know to slow down sooner, giving way to the oncoming objects.

    “The lead agent search will identify the oncoming agent.The agent will know to slow down sooner, giving way to the oncoming objects.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    If the directed Hausdorff distance is small, there are agents moving along the lane connector, and thus the traffic light controlling that lane connector is likely green.

    “If the directed Hausdorff distance is small, we surmise that there are agents moving along the lane connector, and thus the traffic light controlling that lane connector is likely green.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Vehicles are allowed to move under both green and amber light statuses.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Interpretationown data

    Extending all other objects' footprints in the scene proportional to their speed can help in lane-merge situations.

    “Fig. 13 shows an example of a situation where an extended footprint can help in lanemerge situations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: IDM does not inherently account for multi-lane interactions.

  • May combine two claims.

    Interpretationreasoning

    The detection network perceives the scene in a short time window, and the resulting points are limited to a certain perspective, posing challenges to estimating size and heading accurately.

    “Even though the detection network takes n frames of lidar point clouds as input, it still perceives the scene in a short time window.The resulting points are limited to a certain perspective, posing challenges to estimating the size and heading accurately.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    If the minimum speed of agents on a lane is low, the agents are stopped.

    “If the minimum speed of the agents on that lane is low, we surmise that the agents are stopped.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The semantic map layers in nuPlan are designed with planning in mind.

    “The map layers are designed with planning in mind.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Within each traffic light intersection, the individual traffic lights control the flow of vehicles from a lane on one side of the intersection to another lane.

    “Within each traffic light intersection, the individual traffic lights control the flow of vehicles from a lane on one side of the intersection to another lane.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Among the 3098 scenes, 2958 are used for training, 49 for validation and 91 for testing.

    “Among them, 2958 scenes are used for training, 49 for validation and 91 for testing.”
    Quote in the paperNot a substantial claim
48 of 48

  • The paper relies on this without support.

    Driving scenarios typically follow a long-tail distribution.

    “Furthermore, driving scenarios typically follow a long-tail distribution, which further exacerbates the generalization issue.”

  • The paper relies on this without support.

    Learning-based planning lacks formal safety guarantees.

    “Finally, learning-based planning lacks formal safety guarantees, thus making it potentially unsafe and challenging to certify.”

  • The paper relies on this without support.

    Traditional approaches to evaluate planning performance are dominated by monotonous lane following scenarios.

    “Traditional approaches to evaluate planning performance are dominated by monotonous lane following scenarios.”

  • The paper relies on this without support.

    VRU behavior is often uncooperative and thus hard to model.

    “We choose not to model VRUs as reactive agents as their behavior is often uncooperative and thus hard to model.”

  • The paper relies on this without support.

    A large-scale dataset is needed to generalize well.

    “Finally, we need a large-scale dataset to generalize well.”

  • The paper relies on this without support.

    IDM does not inherently account for multi-lane interactions.

    “Merging situations can be tricky because IDM does not inherently account for multi-lane interactions.”

  • The paper relies on this without support.

    Vehicles are allowed to move under both green and amber light statuses.

    “We consider both green and amber statues of traffic lights to be the same, i.e. green, since vehicles are allowed to move under both light statuses.”

  • The paper relies on this without support.

    When a traffic light is red, it is common for drivers to begin decelerating as they approach the intersection.

    “When a traffic light is red, it is common for drivers to begin decelerating as they approach the intersection.”

  • The paper relies on this without support.

    mCenterPoint is lightweight and meets the real-time requirement.

    “We choose mCenterPoint as a baseline because it is lightweight and meets the real-time requirement.”

  • The paper relies on this without support.

    The planners submitted to the nuPlan challenge are representative of purely learned models.

    “This further supports the argument that most purely learned models fail to generalize to closed-loop scenarios.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Seven object classes are listed as six

    Minor

    The sentence says there are six object classes but enumerates seven distinct classes.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Scheel, 2022, Urban driver: Learning to drive from real-world demonstrations using policy gradients, CoRL
  2. Daniels, Scenarionet: An interpretable data-driven model for scene understanding
  3. 10.15607/RSS.2019.XV.031 · DOI
  4. Ngiam, 2022, Scene transformer: A unified architecture for predicting future trajectories of multiple agents, ICLR
  5. 10.1109/CVPR46437.2021.01417 · DOI · Paper page
  6. Houston, 2021, One thousand and one hours: Self-driving motion prediction dataset, CoRL
  7. Jiang, 2022, An efficient framework for reliable and personalized motion planner in autonomous driving, RA-L

Cited by

Sorted by citations

Related

Authors
Karnchanachari, N., Geromichalos, D., Tan, K. S., Li, N., Eriksen, C., Yaghoubi, S., Mehdipour, N., Bernasconi, G., Fong, W. K., Guo, Y., & Caesar, H.
ORCID
Venue
2024 IEEE International Conference on Robotics and Automation (ICRA)
Publisher
IEEE
Published
2024-05-13
Pages
629-636
ISSN
1050-4729 (ISSN-L)
DOI
10.1109/icra57147.2024.10610077
OpenAlex
W4401416944
Semantic Scholar
e99ca2b482c8d020882bc9d83286f04d1b2c1846
Cited by
46
References
83
Citations by year
2026: 24 · 2025: 18 · 2024: 4
FWCI
25.75
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Engineering › Control and Systems Engineering › Traffic control and management
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
autonomous driving, learning-based planning, planning benchmarks, driving scenarios, object tracking, closed-loop simulation, multi-agent interaction, real-world driving data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-17.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-17.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review