CoChat
Sign Up
Cited by 741Advanced Vision and ImagingAdvanced Neural Network ApplicationsRobotics and Sensor-Based Localization

3D Packing for Self-Supervised Monocular Depth Estimation

Open access
Guizilini, Vitor, Ambrus, Rares, Pillai, Sudeep, Raventos, Allan, Gaidon, Adrien·2020·2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr42600.2020.00256
Open this paper in CoChat — free accountOpen PDF

Abstract

Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception. In this work, we propose a novel self-supervised monocular depth estimation method combining geometry with a new deep network, PackNet, learned only from unlabeled monocular videos. Our architecture leverages novel symmetrical packing and unpacking blocks to jointly learn to compress and decompress detail-preserving representations using 3D convolutions. Although self-supervised, our method outperforms other self, semi, and fully supervised methods on the KITTI benchmark. The 3D inductive bias in PackNet enables it to scale with input resolution and number of parameters without overfitting, generalizing better on out-of-domain data such as the NuScenes dataset. Furthermore, it does not require large-scale supervised pretraining on ImageNet and can run in real-time. Finally, we release DDAD (Dense Depth for Automated Driving), a new urban driving dataset with more challenging and accurate depth evaluation, thanks to longer-range and denser ground-truth depth generated from high-density LiDARs mounted on a fleet of self-driving cars operating world-wide.

AI snapshot

Written from the paper’s full text.

TL;DR

PackNet is a self-supervised monocular depth architecture using symmetrical 3D packing/unpacking blocks, trained on unlabeled monocular videos, optionally with velocity supervision for metric scale. It reports state-of-the-art KITTI results, better generalization on NuScenes, real-time inference, and releases the DDAD dataset for long-range dense depth evaluation.

Key findings

  • On DDAD, PackNet-SfM achieves Abs Rel 0.162, Sq Rel 3.917, RMSE 13.452, RMSE log 0.269, and δ<1.25 0.823, outperforming Monodepth2 (R18) at Abs Rel 0.381 and RMSE 21.277; the gap increases at longer ranges up to 200m.
  • On NuScenes generalization without fine-tuning, PackNet reaches Abs Rel 0.187, Sq Rel 1.852, RMSE 7.636, RMSE log 0.289, and δ<1.25 0.742, better than ResNet50 (Abs Rel 0.216, RMSE 8.477) and ResNet50‡ (Abs Rel 0.210, RMSE 8.111).
  • Ablation on KITTI at 640×192 shows PackNet with D=8 3D filters reaches Abs Rel 0.111, Sq Rel 0.785, RMSE 4.601, RMSE log 0.189, and δ<1.25 0.878, improving over no pack/unpack (Abs Rel 0.121, RMSE 4.831).
  • PackNet reaches about 128M parameters with 60ms inference on a Titan V100, improvable to <30ms with TensorRT; a packing/unpacking reconstruction test achieves loss 0.0079 versus 0.063 for max-pooling/bilinear upsampling.

Methods

  • Self-supervised monocular SfM jointly trains a depth network and pose network on unlabeled monocular videos using a photometric loss (SSIM + L1), minimum over source images, auto-masking, and edge-aware depth smoothness; an optional velocity supervision loss aligns pose-translation magnitude with measured velocity to resolve scale ambiguity.
  • PackNet is a symmetrical encoder-decoder with packing blocks (Space2Depth, 3D convolution expansion, 2D contraction) and unpacking blocks (2D convolution, 3D convolution, Depth2Space), plus skip connections and multiple inverse-depth outputs.
  • Training uses KITTI (39,810 train, 4,424 validation, 697 evaluation; improved ground truth 652 maps), DDAD (17,050 train and 4,150 evaluation frames, Luminar-H2, up to 200m), NuScenes (6,019 front-facing validation images), and CityScapes (88,250 training images) for pretraining; PyTorch on 8 Titan V100 GPUs, Adam, 100 epochs, batch size 4, depth LR 2e-4, pose LR 5e-4, α=0.85, λ1=0.001, λ2=0.05.

Limitations

  • The paper states monocular SfM is inherently scale-ambiguous; metric scale requires velocity supervision at training time, otherwise test-time LiDAR ground-truth scaling is needed.
  • The paper states DDAD experiments use only the front-facing camera, and that packing/unpacking do not ensure invertibility.
  • No explicit limitations section is provided; evident constraints include KITTI evaluation up to 80m and DDAD up to 200m, and pose results are deferred to supplementary material due to space.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/cvpr42600.2020.00256

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The proposed approach generalizes better on unseen data.

    “iii) generalizes better on unseen data;”
    Quote in the paperSubstantial claim
  • Findingown data

    The proposed approach is competitive with fully supervised methods.

    “ii) is competitive with fully supervised methods;”
    Quote in the paperSubstantial claim
  • Findingcitation

    In self-supervised monocular depth estimation, model architecture is a critical determinant of performance.

    “While recent works in self-supervised monocular depth estimation have mostly focused on engineering the loss function [6,34,48,54], we show that performance critically depends on the model architecture, in line with the observations of [28] for other self-supervised tasks.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [28] source

      Not checked in the free run.

  • Findingnone given

    The proposed approach can run in real time at high resolution.

    “v) can run in real time at high resolution;”
    Quote in the paperSubstantial claim
  • Findingown data

    The proposed self-supervised monocular approach improves on the state of the art, especially at longer ranges.

    “Our experiments on the standard KITTI benchmark [17], the recent NuScenes dataset [5], and our new proposed DDAD benchmark show that our self-supervised monocular approach i) improves on the state of the art, especially at longer ranges;”
    Quote in the paperSubstantial claim
  • May combine two claims. 17
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 7

  • May combine two claims.

    Findingnone given

    The proposed approach scales better with number of parameters, input resolution, and more unlabeled training data.

    “iv) scales better with number of parameters, input resolution, and more unlabeled training data;”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Using weak velocity information during training makes PackNet scale-aware, so it produces metrically accurate depth maps from a single image.

    “Additionally, by leveraging during training only weak velocity information, we are able to make our model scale-aware, i.e. producing metrically accurate depth maps from a single image.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Although trained only on unlabeled monocular videos, PackNet surpasses existing self- and semi-supervised methods, is competitive with fully supervised methods, and can run in real time.

    “Although purely trained on unlabeled monocular videos, our approach outperforms other existing selfand semi-supervised methods and is even competitive with fully-supervised methods while able to run in real-time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Performance improves at higher image resolutions, which the authors attribute to the network's ability to preserve and process spatial information end-to-end.

    “As indicated by Pillai et al. [39], we also observe an improvement in performance at higher image resolutions, which we attribute to the proposed network's ability to properly preserve and process spatial information end-to-end.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39]

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    DDAD is a more realistic and challenging benchmark for depth estimation, diverse, and captures precise structure at longer ranges (30k points per frame, up to 200m vs 80m for previous datasets).

    “This new dataset is a more realistic and challenging benchmark for depth estimation, as it is diverse and captures precise structure across images (30k points per frame) at longer ranges (up to 200m vs 80m for previous datasets).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    PackNet's performance keeps improving as more parameters are added, without overfitting.

    “and further improves as more complexity is added”
    • Also stated in: § Input image · boosted
      “improves significantly with more parameters in both cases without overfitting.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    PackNet-SfM achieves competitive ATE results on KITTI odometry relative to related methods, even without ground-truth scaling when trained with velocity supervision.

    “The ATE results are summarized in Table 6, with our proposed framework achieving competitive results relative to other related methods.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The pose network architecture from [53] without the explainability mask did not improve results.

    “We use the architecture proposed by [53] without the explainability mask, which we found not to improve results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    DDAD uses a high-resolution, long-range Luminar-H2 LiDAR with a maximum range of 250 m and sub-1 cm range precision.

    “It includes a highresolution, long-range Luminar-H2foot_0 as the LiDAR sensor used to generate pointclouds, with a maximum range of 250m and sub-1cm range precision.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    DDAD contains 17,050 training and 4,150 evaluation frames with ground-truth depth maps generated from dense LiDAR measurements using the Luminar-H2 sensor.

    “It contains 17,050 training and 4,150 evaluation frames with ground-truth depth maps generated from dense LiDAR measurements using the Luminar-H2 sensor.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Self-supervised pretraining on other datasets can further improve PackNet.

    “and can be further improved by self-supervised pretraining on other datasets”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Best results are achieved by injecting more unlabeled data at training time and processing higher resolution input images, achieving performance comparable to semi-supervised [29] and fully supervised [15] methods.

    “Our best results are achieved when injecting both more unlabeled data at training time and processing higher resolution input images, achieving performance comparable to semisupervised [29] and fully supervised [15] methods.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [29]

      Not checked in the free run.

    • cites [15]

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The proposed PackNet architecture has roughly 128 million parameters and takes about 60 milliseconds per inference on a Titan V100 GPU.

    “the proposed architecture (Table 1) reaches around 128M parameters with an inference time of 60ms on a Titan V100 GPU”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On DDAD, PackNet-SfM significantly outperforms the state-of-the-art ResNet-based method [19], with the performance gap increasing at larger distances.

    “From these results we can see that our PackNet-SfM approach significantly outperforms the state-of-the-art [19], based on the ResNet family, the performance gap consistently increasing when larger distances are considered.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    DDAD is a cross-continental dataset with urban scenes from the United States and Japan.

    “DDAD is a cross-continental dataset with scenes drawn from urban settings in the United States (San Francisco Bay Area, Detroit and Ann Arbor) and Japan (Tokyo and Odaiba).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    DDAD has six calibrated cameras synchronized at 10 Hz that together give 360° coverage around the vehicle.

    “Additionally, it contains six calibrated cameras time-synchronized at 10 Hz, that together produce a 360 • coverage around the vehicle.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    In self-supervised monocular SfM training, depth and pose are estimated only up to an unknown scale factor because of photometric loss ambiguity.

    “Note that in the case of monocular SfM both depth and pose are estimated up to an unknown scale factor, due to the inherent ambiguity of the photometric loss.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The velocity loss allows the pose network to make metrically accurate predictions, and the depth network learns metrically accurate estimates to maintain consistency.

    “This additional velocity loss allows the pose network to make metrically accurate predictions, subsequently resulting in the depth network also learning metrically accurate estimates to maintain consistency (cf.Section 5.4).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Instantaneous velocity measurements are ubiquitous in most mobile systems today.

  • May combine two claims.

    Interpretationown data

    PackNet learns transferable discriminative features rather than merely memorizing training data.

    “as evidence that it does not simply memorize training data but learns transferable discriminative features.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    The 3D inductive bias in PackNet enables scaling with input resolution and number of parameters without overfitting, and generalizing better on out-of-domain data such as NuScenes.

    “The 3D inductive bias in PackNet enables it to scale with input resolution and number of parameters without overfitting, generalizing better on out-of-domain data such as the NuScenes dataset.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ResNet architectures gain substantially from ImageNet pretraining.

    “As mentioned in [15,19], ResNet architectures highly benefit from ImageNet pretraining, since they were originally developed for classification tasks.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [15]

      Not checked in the free run.

    • cites [19]

      Not checked in the free run.

21 of 21

  • The paper relies on this without support.

    Instantaneous velocity measurements are ubiquitous in most mobile systems today.

    “Since instantaneous velocity measurements are ubiquitous in most mobile systems today, we show that they can be directly incorporated in our self-supervised objective to learn a metrically accurate and scale-aware monocular depth estimator.”

  • The paper relies on this without support.

    The same pixel will not be occluded or out-of-bounds in all context images, and the association with minimal photometric loss is correct.

    “The intuition is that the same pixel will not be occluded or out-of-bounds in all context images, and that the association with minimal photometric loss should be the correct one.”

  • The paper relies on this without support.

    Depth maps made by projecting LiDAR pointclouds onto the camera frame are accurate enough to serve as ground truth for evaluating monocular depth estimation.

    “The depth maps generated from projecting these Luminar pointclouds onto the camera frame allow us to evaluate depth estimation methods in a much more challenging way, both in terms of denseness and longer ranges.”
    “At test-time, all monocular methods (M) scale estimated depths with median ground-truth LiDAR information.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some methods incorporate more 3D computations into the object detection pipeline by applying a 3D reconstruction method like [4,5,6] to create a pseudo-LiDAR or range input of the scene from camera images.

    “some methods incorporate more 3D computations into our object detection pipeline by applying a 3D reconstruction method like [4, 5, 6] to create a pseudo-LiDAR or range input of the scene from camera images.”

    From DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries (2021) · cites this paper as [6]

  • DDAD provides semantic segmentation labels only for the validation and test (non-public) sets, significantly restricting its use-case.

    “The dataset provides semantic segmentation labels only for the validation and test (non-public) sets, significantly restricting its use-case.”

    From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [82]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • PyTorch citation points to a different reference

    Minor

    The implementation section cites [38] for PyTorch, but reference 38 is about monocular obstacle avoidance. The PyTorch reference is numbered 39 in the provided bibliography.

1 of 1

Cites

Flagged first, then by citations

2 of the 59 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. kingma, 0, Glow: Generative flow with invertible 1&#x00D7;1 convolutions, Advances in neural information processing systems
  2. graham, 2015, Fractional Max-Pooling

Cited by

Sorted by citations

Related

Authors
Guizilini, V., Ambrus, R., Pillai, S., Raventos, A., & Gaidon, A.
Affiliations
Toyota Research Institute (US)
ORCID
Venue
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2020-06
Pages
2482-2491
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr42600.2020.00256
OpenAlex
W2991840778
Semantic Scholar
43fcb6cf679cd2ae412d7cb53e85312600cc20dd
Cited by
741
References
59
Citations by year
2026: 51 · 2025: 108 · 2024: 140 · 2023: 155 · 2022: 139 · 2021: 125 · 2020: 19 · 2019: 4
FWCI
33.78
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
Keywords
monocular depth estimation, self-supervised learning, 3D convolution, out-of-domain generalization, real-time depth estimation, autonomous driving, KITTI dataset, nuScenes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review