CoChat
Sign Up
Cited by 334Advanced Neural Network ApplicationsAdvanced Vision and ImagingAutonomous Vehicle Technology and Safety

Is Pseudo-Lidar needed for Monocular 3D Object detection?

Open access
Park, Dennis, Ambrus, Rares, Guizilini, Vitor, Li, Jie, Gaidon, Adrien·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv48922.2021.00313
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent progress in 3D object detection from single images leverages monocular depth estimation as a way to produce 3D pointclouds, turning cameras into pseudo-lidar sensors. These two-stage detectors improve with the accuracy of the intermediate depth estimation network, which can itself be improved without manual labels via large-scale self-supervised learning. However, they tend to suffer from overfitting more than end-to-end methods, are more complex, and the gap with similar lidar-based detectors remains significant. In this work, we propose an end-to-end, single stage, monocular 3D object detector, DD3D, that can benefit from depth pre-training like pseudo-lidar methods, but without their limitations. Our architecture is designed for effective information transfer between depth estimation and 3D detection, allowing us to scale with the amount of unlabeled pre-training data. Our method achieves state-of-the-art results on two challenging benchmarks, with 16.34% and 9.28% AP for Cars and Pedestrians (respectively) on the KITTI-3D benchmark, and 41.5% mAP on NuScenes.

AI snapshot

Written from the paper’s full text.

TL;DR

DD3D is an end-to-end single-stage monocular 3D detector that uses large-scale dense depth pre-training to gain pseudo-lidar-like scalability without pseudo-lidar’s two-stage complexity and generalization issues. It reports state-of-the-art results on KITTI-3D and nuScenes.

Key findings

  • DD3D achieves 16.34% 3D AP on KITTI-3D Moderate Cars, a 23% improvement over the previous best 13.25%, and 9.30% 3D AP on Moderate Pedestrians (Table 4; abstract reports 9.28%), an 80.5% improvement over 5.14%. On Cyclist it is second best, behind a method using ground-truth pointclouds.
  • On nuScenes, DD3D reports 41.8% mAP and 0.57 NDS (Table 2; abstract reports 41.5% mAP), a 17% mAP improvement over the previously best published method (41.8% vs. 35.8%), and surpasses the lidar-based PointPillars detector.
  • Removing depth pre-training costs 5.3% Car Moderate BEV AP with DLA-34 and 10.7% with V2-99; removing COCO pre-training costs 2.0%. In a controlled pre-training comparison on 136,571 nuScenes images, dense depth pre-training gives 21.8% vs. 20.5% Car Moderate BEV AP over 2D detection pre-training.
  • Pseudo-lidar (PL) requires in-domain depth fine-tuning: without KITTI fine-tuning, PL drops from 30.1% to 19.1% BEV AP. PL also generalizes worse from KITTI-3D val to test (30.1% → 18.6% BEV AP) than DD3D (29.4% → 22.56% BEV AP).

Methods

  • DD3D is a fully convolutional single-stage network extending FCOS with an FPN backbone (V2-99 or DLA-34) and three heads: classification, 2D box, and a 3D detection head predicting quaternion orientation, per-pixel depth, 3D box center offsets, size deviations, and 3D confidence. It is camera-aware via focal lengths and trained with FCOS 2D losses, a disentangled L1 3D box loss, and a 3D confidence loss.
  • Pre-training uses dense depth prediction with an L1 loss on sparse Lidar depth across all FPN levels. The main pre-training dataset is DDAD15M (25,000 multi-camera videos, ~15M images). Evaluation uses KITTI-3D (7481 train / 7518 test images; 3712/3769 train/val split) and nuScenes (1000 videos; 700/150/150 train/val/test).
  • The PL baseline uses PackNet for depth, fine-tuned on a new KITTI Eigen-clean split, followed by a 3D detector based on prior PL work with a ResNet-18 backbone and 3D confidence. Analysis includes ablations, controlled pre-training on nuScenes images, and scaling with 0.6M, 3M, 6M, and 15M pre-training images.

Limitations

  • The paper states PL limitations: it requires in-domain depth fine-tuning (PL drops 30.1% → 19.1% BEV AP without it), tends to overfit, and generalizes worse from KITTI-3D val to test than DD3D.
  • The paper does not explicitly state limitations for DD3D itself. The text makes evident that DD3D is not best on Cyclist (second best, behind a method using ground-truth pointclouds) and relies on the in-house DDAD15M dataset for large-scale pre-training.
  • The text also notes test-time augmentation gave a 2.3% gain on KITTI val but no improvement on nuScenes validation, and DD3D metrics are averaged over 4 runs with 0.5–1.2% BEV AP variance.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.00313

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 24 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Pseudo-lidar detectors tend to overfit more than end-to-end methods.

    “However, they tend to suffer from overfitting more than end-to-end methods”
    Quote in the paperNot a substantial claim
  • Findingown data

    DD3D achieves 16.34% AP for Cars on the KITTI-3D benchmark.

    “with 16.34% and 9.28% AP for Cars and Pedestrians (respectively) on the KITTI-3D benchmark”
    Quote in the paperSubstantial claim
  • Findingown data

    DD3D achieves 9.28% AP for Pedestrians on the KITTI-3D benchmark.

    “with 16.34% and 9.28% AP for Cars and Pedestrians (respectively) on the KITTI-3D benchmark”
    Quote in the paperSubstantial claim
  • Findingown data

    Using a large corpus of unlabeled raw data, DD3D scales similarly to pseudo-lidar methods.

    “Using a large corpus of unlabeled raw data, we show that DD3D scales similarly to pseudo-lidar methods”
    Quote in the paperSubstantial claim
  • Findingown data

    DD3D achieves 41.5% mAP on NuScenes.

    “and 41.5% mAP on NuScenes.”
    Quote in the paperSubstantial claim
  • May combine two claims. 17
  • Classified as probably not a substantial claim. 8
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingnone given

    DD3D establishes a new best result for monocular 3D detection on KITTI-3D and nuScenes, improving significantly over earlier methods.

    “Our method sets a new state of the art on the task of monocular 3D detection on KITTI-3D [14] and nuScenes [5] with significant improvements compared to previous state-of-the-art methods.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Pseudo-lidar two-stage detectors improve as the intermediate depth estimation network becomes more accurate, and that depth network can be improved without manual labels through large-scale self-supervised learning.

    “These two-stage detectors improve with the accuracy of the intermediate depth estimation network, which can itself be improved without manual labels via large-scale self-supervised learning.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    Resizing and flipping inputs at test time improves Car BEV AP on KITTI val by 2.3%, but does not improve nuScenes validation.

    “We observed 2.3% gain in”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    PL's BEV AP on KITTI-3D drops from 30.1% on val to 18.6% on test, a larger drop than other methods including DD3D (29.4% to 22.56%).

    “PL suffers particularly more (30.1% → 18.6% BEV AP), when compared to other methods including DD3D (29.4% → 22.56%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Averaged over the three object classes, relative gains over the prior best method [65] are 103.7% at the 0.5m threshold and 35.5% at the 1.0m threshold.

    “comparing to the previous best method [65], the relative improvements averaged across the 3 object classes are 103.7% and 35.5% for 0.5m and 1.0m thresholds, respectively.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [65] source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    On KITTI-3D Moderate Cars, DD3D reaches 16.34% 3D AP, which is 23% better than the prior best of 13.25%.

    “DD3D achieves a significant improvement over all methods with 16.34% 3D AP on Moderate Cars, which amounts to a 23% improvement from the previous best method (13.25%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The PL detector matches state-of-the-art methods but is much worse than DD3D (13.05% vs 16.34%).

    “The accuracy of our PL detector is on par with stateof-the-art methods, however it performs significantly worse than DD3D (13.05% vs. 16.34%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At every checkpoint, DD3D and PL are similar, and both keep improving with more depth pre-training data up to 15M images.

    “We note that DD3D and PL perform similarly at each checkpoint, and continue to improve as more depth data is used in pretraining, at least up to 15M images.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On nuScenes, DD3D beats all other methods, improving mAP by 17% over the prior best published result (41.8% vs 35.8% mAP) and by 13% over the best unpublished result.

    “DD3D outperforms all other methods with a 17% improvement in mAP compared to the previously best published method [65] (41.8% vs. 35.8%mAP) as well as a 13% improvement compared to the best (unpublished) method.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [65] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Variance over four training runs is small, between 0.5 and 1.2% BEV AP.

    “We observed the variance over the runs to be small, i.e. 0.5 ∼ 1.2% BEV AP.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Pedestrians, DD3D beats all other methods, improving by 80.5% over the previous best (9.30% vs 5.14% 3D AP).

    “DD3D outperforms all other approaches on the Pedestrian category, with an 80.5% improvement from the previous best method (9.30% vs 5.14% 3D AP).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For Cyclists, DD3D is second best and narrows the gap to [27].

    “On Cyclist DD3D achieves the second best result, reducing the gap to [27] that uses ground-truth pointclouds to train a per-instance pointcloud reconstruction module and a two-stage regression network to refine 3D object proposals.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Findingown data

    DD3D improves significantly across every category and threshold on nuScenes.

    “In general, DD3D offers significant improvements across all categories and thresholds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    PackNet recovers fine structures in the depth map with high precision by avoiding feature sub-sampling.

    “By avoiding feature sub-sampling, Pack-Net recovers fine structures in the depth map with high precision”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A disentangled loss for 3D box regression helps stabilize training.

    “In [59], the authors proposed a disentangled loss for 3D box regression that helps stabilize training.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    PackNet generalizes better due to its increased capacity.

    “moreover, PackNet has been shown to generalize better thanks to its increased capacity [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    DD3D is an end-to-end, single-stage monocular 3D object detector that can benefit from depth pre-training like pseudo-lidar methods without their limitations.

    “In this work, we propose an end-to-end, single stage, monocular 3D object detector, DD3D, that can benefit from depth pre-training like pseudo-lidar methods, but without their limitations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Using camera focal lengths in DD3D endows it with camera-awareness, allowing depth inference from both the input image and the pixel size.

    “We note that this design of using camera focal lengths endows DD3D with camera-awareness, by allowing us to infer the depth not only from the input image, but also from the pixel size.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    DD3D's straightforward training and end-to-end optimization let it exploit large-scale depth data effectively, resulting in strong multi-class detection accuracy.

    “The simplicity of its training procedure and its end-to-end optimization allows for effective use of large-scale depth data, leading to impressive multi-class detection accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Incorporating uncertainty in estimating the 3D box or its depth greatly improves detection accuracy.

    “Incorporating uncertainty in estimating the 3D box (or its depth) has also been shown to greatly improve detection accuracy [59,9,58].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [59] source

      Not checked in the free run.

    • cites [9]

      Not checked in the free run.

    • cites [58]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There is a bias in pseudo-lidar results on the KITTI-3D benchmark.

    “Recently, [58] has shown a bias in the PL results on the representative KITTI-3D [14] benchmark”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [58]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    About one third of the Eigen split's training images overlap with KITTI-3D images, which biases pseudo-lidar model results.

    “However, as described in [58], up to a third of its training images overlap with KITTI-3D images, leading to biased results for PL models.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [58]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    DD3D gains the advantage of Pseudo Lidar approaches-accuracy scaling from large-scale depth data-while avoiding their drawbacks, such as impractical training and generalization problems.

    “DD3D, an end-to-end single-stage 3D object detector that enjoys the benefit of Pseudo Lidar meth- ods (i.e.scaling accuracy using large-scale depth data), but without its limitation (i.e.impractical training, issues in generalization).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    In DD3D, the route from the input image to the 3D box and the route to dense depth prediction share almost all parameters, which enables effective transfer from pre-training to the target task.

    “We note that the two paths in DD3D from the input image to the 3D bounding box and to the dense depth prediction differ only in the last 3×3 convolutional layer, and thus share nearly all parameters.This allows for effective transfer from the pre-trained representation to the target task.”
    Quote in the paperSubstantial claim
24 of 24

  • The paper relies on this without support.

    Estimating depth from a single image is an ill-posed inverse problem by nature.

    “However, regressing depth from single images is inherently an ill-posed inverse problem.”

  • The paper relies on this without support.

    Earlier end-to-end monocular 3D detectors lack the scalability from unsupervised pre-training that their single-stage design prevents.

    “Simpler end-to-end monocular 3D detectors [3,39] seem like a promising alternative, although they do not enjoy the same scalability benefits of unsupervised pre-training due to their single stage nature.”

  • The paper relies on this without support.

    A substantial gap still separates pseudo-lidar detectors from lidar-based detectors.

    “the gap with similar lidar-based detectors remains significant.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LiDAR-based detection processes 3D point cloud data, offering precise depth estimation and spatial structure representation.

    “In contrast, LiDAR-based detection [73,74,75,76,77] processes 3D point cloud data, offering precise depth estimation and spatial structure representation.”

    From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [76]

    Section 4 (page unavailable) says that a LiDAR-based 3D detector processes a point cloud and that ground-truth depth measurements come from LiDAR point clouds. It also describes point-cloud data as containing 3D coordinates, supporting the spatial-representation point. The copy does not establish the claimed precision of LiDAR-based depth estimation; its statement about high precision refers to a monocular depth network. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • BEV-based approaches have not shown significant advantages over other paradigms in 3D object detections.

    “BEV-based approaches have not shown significant advantages over other paradigm in 3D object detections [47, 31, 34].”

    From BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022) · cites this paper as [31]

    The KITTI-3D results report that the paper’s pseudo-Lidar detector performs significantly worse than DD3D, and Section 6.2.2 reports a larger validation-to-test performance drop for pseudo-Lidar. This supports a narrower comparison, not the claim that BEV-based approaches generally lack significant advantages over other 3D-detection paradigms. The Related Work section also describes potential benefits of BEV transformation alongside its limitations. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • DETR3D uses the same backbone as DD3D for a fair comparison.

    “our method uses the same backbone as DD3D [37] for a fair comparison.”

    From DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries (2021) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Box-size parameter lists depth incorrectly

    Minor

    The parameter tuple lists δ H twice, whereas the immediately following size formula uses δ D for depth.

  • Conflicting nuScenes mAP results

    Minor

    The abstract reports 41.5% mAP, while the benchmark-results discussion and table report 41.8% for DD3D.

  • Conflicting KITTI Pedestrian AP results

    Minor

    The abstract gives 9.28% AP for Pedestrians, but the reported Moderate Pedestrian 3D AP is 9.30%.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. wang, 2019, Pseudolidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  2. simonelli, 2020, Disentangling monocular 3d object detection: From single to multi-class recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence
  3. 10.1109/CVPR42600.2020.00256 · DOI · Paper page
  4. guizilini, 2019, Robust semi-supervised monocular depth estimation with reprojected distances, Conference on Robot Learning (CoRL)

Cited by

Sorted by citations

Related

Authors
Park, D., Ambrus, R., Guizilini, V., Li, J., & Gaidon, A.
Affiliations
  • University of British Columbia (CA)
  • Toyota Research Institute
  • Toyota Motor Corporation (Switzerland) (CH)
ORCID
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
3122-3132
DOI
10.1109/iccv48922.2021.00313
OpenAlex
W3194903941
Semantic Scholar
8620b942d3e31261971ce7b5d50f652a09e0aa62
Cited by
334
References
74
Citations by year
2026: 32 · 2025: 74 · 2024: 96 · 2023: 94 · 2022: 38
FWCI
18.61
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • IEEE/CVF International Conference on Computer Vision (ICCV)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
Keywords
monocular 3D object detection, pseudo-LiDAR, monocular depth estimation, single-stage detectors, self-supervised pre-training, unlabeled data, point cloud generation, nuScenes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review