OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving
Open accessAbstract
Surround View fisheye cameras are commonly deployed in automated driving for 360°near-field sensing around the vehicle. This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. We demonstrate that the jointly trained model performs better than the respective single task versions. Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other. We propose a novel camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference. This was crucial to enable training on the WoodScape dataset, comprised of data from different parts of the world collected by 12 different cameras mounted on three different cars with different intrinsics and viewpoints. Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices. We additionally evaluate our model on standard automotive datasets, namely KITTI and Cityscapes. We obtain the state-of-the-art results on KITTI for depth estimation and pose estimation tasks and competitive performance on the other tasks. We perform extensive ablation studies on various architecture choices and task weighting methodologies. A short video at https://youtu.be/xbSjZ5OfPes provides qualitative results.
AI snapshot
Written from the paper’s full text.
TL;DR
OmniDet is a multi-task visual perception network for unrectified surround-view fisheye cameras that jointly performs depth/distance estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. It reports that joint training with a shared encoder and synergized decoders outperforms single-task baselines and achieves state-of-the-art KITTI depth and pose results among monocular methods.
Key findings
- The final WoodScape OmniDet (SAN10-patch) model reports RMSE 1.210, δ<1.25 0.929, semantic mIoU 78.9, motion mIoU 79.2, and detection mAP 74.1, improving over its own ablations without the proposed components.
- Multi-task learning outperformed single-task models on WoodScape, KITTI, and Cityscapes; for example, on WoodScape the multi-task setting reached distance RMSE 1.332, semantic mIoU 76.6, motion mIoU 75.3, and detection mAP 68.4, while single-task values included distance RMSE 1.681, semantic mIoU 72.5, motion mIoU 68.1, and detection mAP 63.5.
- VarNorm task weighting gave the best WoodScape task-weighting result: Sq.Rel 0.046, Abs.Rel 0.276, semantic mIoU 76.6, motion mIoU 75.3, and detection mAP 68.4, compared with equal weighting at Sq.Rel 0.058, Abs.Rel 0.302, semantic mIoU 70.3, motion mIoU 67.3, and detection mAP 64.6.
- On KITTI, OmniDet reports state-of-the-art monocular depth and pose results: depth Abs.Rel 0.092, Sq.Rel 0.657, RMSE 3.984 on the Eigen split, and pose errors of 0.009 ± 0.004 on sequence 09 and 0.008 ± 0.005 on sequence 10 with GT; with online refinement, depth improves to Abs.Rel 0.077, Sq.Rel 0.641, RMSE 3.859.
Methods
- The model uses a shared self-attention encoder with task-specific decoders for six tasks on unrectified fisheye images: distance/depth, pose/visual odometry, semantic segmentation, motion segmentation, object detection, and soiling segmentation.
- Key components include a camera geometry tensor to encode fisheye distortion and camera intrinsics, VarNorm variance-based task weighting, semantically guided distance and detection decoders via pixel-adaptive convolutions, dynamic-object masking from motion/semantic segmentation, and a 24-sided non-uniform polygon representation for fisheye object detection.
- Training and evaluation use WoodScape (10,000 images split 6:1:3, 12 cameras, three cars), KITTI, and Cityscapes; the model is trained in PyTorch for 20 epochs with batch size 24 on a Titan RTX, and inference is measured with TensorRT FP16 on NVIDIA Jetson AGX.
Limitations
- The paper states that scaling to more tasks still faces practical challenges: building a diverse and balanced dataset, corner case mining, stable training mechanisms, and designing an optimal map representation that combines all tasks while reducing post-processing.
- The paper states that the soiling dataset is independently built and cannot be trained jointly in a traditional manner; the shared encoder is frozen and only the soiling decoder is trained, and asynchronous backpropagation achieved the same accuracy.
- Evident from the text: polygon detection on raw fisheye is limited to only two classes, pedestrians and vehicles; motion labels for KITTI and Cityscapes are available only for the cars category; and additional proprietary data was used for pre-training and initialization on WoodScape.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
4 of 21 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
- Interpretationcitation
Feature-metric losses prevent the training objective from getting stuck at multiple local minima for homogeneous areas.
“The main goal of these losses is to prevent the training objective from getting stuck at multiple local minima for homogeneous areas as fisheye images have considerably larger homogeneous areas than their rectilinear counterpart.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [11] source
The Multi-scale Estimation section says multi-scale depth prediction and image reconstruction are used to keep training from getting stuck in local minima, and discusses ambiguity in large low-texture regions. It does not attribute that effect to feature-metric losses or compare homogeneous areas in fisheye and rectilinear images. Page numbers are unavailable in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measured
- Rests on an assumption: Fisheye images have considerably larger homogeneous areas than their rectilinear counterpart.
- Findingown data
A multitask network with 6 tasks (5 diverse tasks) performs better than the equivalent single-task models on the evaluated datasets.
“Quantitative results from our experiments indicate that a multitask network with 6 tasks, 5 diverse tasks perform better than the single task models along with our proposed synergies explained in Section III-B.”
Quote in the paperSubstantial claim - Findingown data
Adding the camera geometry tensor to the training setting significantly increases distance estimation accuracy.
“When adding the camera geometry tensor to this setting, we observe a significant increase in accuracy since we train multiple cameras with different camera intrinsics and viewing angles.”
Quote in the paperSubstantial claim - Findingown data
The novel fusion technique in OmniDet significantly improves the detection decoder's accuracy, as seen in Table I.
“This novel fusion technique in our OmniDet framework significantly improves the detection decoder's accuracy, which can be seen in Table I.”
Quote in the paperSubstantial claim - Findingnone given
Using semantically guided features in the detection decoder yields significant gains in mAP.
“we use semantically guided features to the detection decoder as described in Section III-B2, which yields significant gains in mAP”
Quote in the paperSubstantial claim
- Findingown data
OmniDet outperforms the previous methods for pose estimation on the KITTI odometry benchmark sequences 09 and 10.
“We outperform the previous methods listed in Table VI”
Quote in the paperSubstantial claim - Findingnone given
The paper demonstrates the first real-time six-task model for surround-view fisheye camera perception.
“We demonstrate the first real-time six-task model for surround-view fisheye camera perception.”
Quote in the paperSubstantial claim - Findingown data
Training soiling jointly using asynchronous backpropagation achieved the same accuracy as using the frozen encoder.
“We also trained soiling jointly using asynchronous backpropagation [22], but it achieved the same accuracy as using the frozen encoder.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [22] source
- Findingnone given
Uniform sampling cannot efficiently represent high curvature variations in the fisheye image object contours.
“We observe that uniform sampling cannot efficiently represent high curvature variations in the fisheye image object contours.”
Quote in the paperSubstantial claim - Findingown data
Evaluating the network on five tasks on KITTI and Cityscapes achieves competitive results.
“we evaluate our network on five tasks on KITTI and Cityscapes, achieving competitive results.”
Quote in the paperSubstantial claim - Findingown data
With the synergy of perception tasks, OmniDet obtains state-of-the-art depth and pose estimation results on the KITTI dataset.
“With this synergy of perception tasks, we obtain state-of-the-art depth and pose estimation results on the KITTI dataset, as shown in Table V and Table VI respectively.”
Quote in the paperSubstantial claim - Findingreasoning
OmniDet outperforms all previous monocular methods for depth estimation on the KITTI Eigen split.
“we outperform all the previous monocular methods.”
Quote in the paperSubstantial claim - Findingreasoning
Using the discriminative features loss L_dis yields significant gains in distance estimation accuracy.
“We achieve significant gains in this setting by attributing better-supervised signal provided by using discriminative features loss L dis .”
Quote in the paperSubstantial claim - Findingown data
A six-task network with a shared encoder and synergized decoders was successfully demonstrated on fisheye surround-view images.
“We successfully demonstrated a six-task network with a shared encoder and synergized decoders on fisheye surroundview images in this work.”
Quote in the paperSubstantial claim - Findingreasoning
Motion mask based filtering yields superior gains, together with the camera geometry tensor, compared to using semantic masks.
“Motion mask based filtering yields superior gains along with CGT compared to using semantic masks as semantics might not contain all the dynamic objects in its set of classes as indicated in Eq. 4.”
Quote in the paperSubstantial claim - Interpretationreasoning
The camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference was crucial to enable training on the WoodScape dataset.
“We propose a novel camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference.This was crucial to enable training on the WoodScape dataset, comprised of data from different parts of the world collected by 12 different cameras mounted on three different cars with different intrinsics and viewpoints.”
Quote in the paperSubstantial claim- Rests on an assumption: Although the underlying camera intrinsics model is the same for a particular family of vehicles, manufacturing variations require calibration of each camera instance.
- Rests on an assumption: Calibration can vary after deployment due to high environmental temperature or aging.
- Interpretationcitation
The pose estimation improvement over previous methods is mainly due to the bundle adjustment framework using the cross-sequence distance consistency loss, which induces more constraints and simultaneously optimizes distances and camera pose over an implicitly extended training input sequence.
“mainly by applying our bundle adjustment framework using our cross-sequence distance consistency loss [9] that induces more constraints and simultaneously optimizes distances and camera pose for an implicitly extended training input sequence.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationnone given
The paper proposes novel design techniques, including VarNorm task weighting.
“We propose novel design techniques, including VarNorm task weighting.”
Quote in the paperSubstantial claim - Premisecitation
In the WoodScape fisheye dataset, there are 12 different cameras with slight intrinsic variations to evaluate this effect.
“In the Woodscape fisheye dataset [29], there are 12 different cameras with slight intrinsic variations to evaluate this effect.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [29] source
- Interpretationnone given
The paper employs a novel camera geometry tensor to handle multiple viewpoints and change in the camera's intrinsic distance estimation.
“Firstly, we employ our novel camera geometry tensor to handle multiple viewpoints and change in the camera's intrinsic distance estimation.”
Quote in the paperSubstantial claim - Interpretationown data
Freezing the shared encoder trained using five other tasks and training only the decoder for soiling demonstrates the potential of reusing encoder features for additional tasks.
“Thus we freeze the shared encoder trained using five other tasks and train only the decoder for soiling.This demonstrates the potential of reusing the encoder features for additional tasks.”
Quote in the paperSubstantial claim - Interpretationnone given
The paper proposes a novel method called VarNorm for variance normalization.
“Secondly, we propose a novel method called VarNorm for variance normalization.”
Quote in the paperSubstantial claim - Interpretationnone given
The paper designs synergized decoders where different tasks help each other in addition to a shared encoder.
“We design synergized decoders where different tasks help each other in addition to a shared encoder.”
Quote in the paperNot a substantial claim - Interpretationnone given
The paper employs synergized decoders via cross task connections to improve each other's performance.
“Secondly, we employ synergized decoders via cross task connections to improve each other's performance.”
Quote in the paperNot a substantial claim - Interpretationreasoning
A calibration adaptation mechanism in the model is essential.
“Thus a calibration adaptation mechanism in the model is essential.”
Quote in the paperNot a substantial claim- Rests on an assumption: Although the underlying camera intrinsics model is the same for a particular family of vehicles, manufacturing variations require calibration of each camera instance.
- Rests on an assumption: Calibration can vary after deployment due to high environmental temperature or aging.
- Interpretationreasoning
Adding semantic guidance to the distance decoder helps reason about geometry and content within the same shared features and disambiguate photometric ambiguities.
“It helps to reason about geometry and content within the same shared features and disambiguate photometric ambiguities.”
Quote in the paperSubstantial claim - Premisereasoning
Bounding boxes are not a good representation for distorted fisheye images.
“Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices.”
- Also stated in: § B. Generalized Object Detection · plain
“The standard bounding box representation fails in fisheye cameras due to heavy radial distortion, particularly in the periphery.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [15] source
- Also stated in: § B. Generalized Object Detection · plain
- Interpretationreasoning
The paper extends object detection to use a polygon with non-uniformly sampled vertices.
“Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices.”
Quote in the paperSubstantial claim - Interpretationnone given
The paper proposes a novel camera tensor representation of radial distortion to enable adaptation of a CNN for the 12 different camera models in the WoodScape dataset.
“We propose a novel camera tensor representation of radial distortion to enable the adaptation of CNN for the 12 different camera models in the WoodScape dataset.”
Quote in the paperSubstantial claim
- The quote only partly states this claim. 2
- May combine two claims. 24
- Classified as probably not a substantial claim. 13
The quote only partly states this claim.
FindingcitationUsing the online refinement method of Casser et al. yields a significant improvement in depth estimation.
“Using the online refinement method from [42], we obtain a significant improvement.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [42] source
The Conclusion (p. 8) attributes improved performance to sparse convolution modules, while Boosting Single-Image Depth Prediction (pp. 14–15) discusses training with different depth maps. Neither passage addresses Casser et al.’s online refinement method or an improvement from using it; searches for “Casser” and “online refinement” found no such passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationThere is minimal work in the area of fisheye perception; specifically, the only previous work on multi-task learning is FisheyeMultiNet, which discusses a more straightforward three-task network.
“In general, there is minimal work in the area of fisheye perception.Specifically, there is only our previous work on multi-task learning: FisheyeMultiNet [6] which discusses a more straight forward three task network.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [6] source
The Related Work section discusses single-view depth prediction and networks that jointly predict depth and segmentation, but does not address how much work exists on fisheye perception. Searches for “fisheye,” “FisheyeMultiNet,” and fisheye multi-task learning found no passage identifying FisheyeMultiNet or describing a three-task network. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
PremisecitationShu et al. proposed a convergent loss to have a relatively large convergence radius to enable gradient descent from a remote distance.
“Shu et al. [11] proposed a convergent loss to have a relatively large convergence radius to enable gradient descent from a remote distance.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [11] source
The Per-Pixel Minimum Reprojection Loss and Multi-scale Estimation sections describe Godard et al.’s depth-estimation losses, not a convergent loss proposed by Shu et al. Searches for “convergent loss,” “convergence radius,” and “Shu et al.” found no passage in our copy addressing the attributed claim. Page numbers were not available in the indexed passages. (LLM, conf=0.75)
Experimental — accuracy not yet measured
May combine two claims.
Findingown dataVarNorm achieved the best results and was employed for all further experiments.
“We employ the proposed VarNorm method for all the further experiments as it achieved the best results.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataEqual weighting is the worst, and the multi-task network performs better than the single-task networks by using any dynamic task weighting method presented.
“As shown in Table III, equal weighting is the worst, and our multitask network performs better than the single task networks by using any dynamic task weighting method presented above.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAmong the evaluated object detection representations, the 24-sided polygon achieves the highest ground-truth instance segmentation mIoU (86.6) and the highest network mIoU (44.6).
“24-sided Polygon 86.6 44.6 48”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe jointly trained multi-task model performs better than the respective single-task versions, including a six-task model on WoodScape and a five-task model on KITTI and Cityscapes performing better than single-task baselines.
“We demonstrate that the jointly trained model performs better than the respective single task versions.”
- Also stated in: § I. INTRODUCTION · plain
“We showcase a six-task model on WoodScape and fivetask model on KITTI and Cityscapes performing better than the single task baselines.”
Quote in the paperSubstantial claim- Also stated in: § I. INTRODUCTION · plain
May combine two claims. The quote only partly states this claim.
Findingown dataThe model obtains state-of-the-art results on KITTI for depth estimation and pose estimation among monocular methods and competitive performance on the other tasks.
“We obtain the state-of-the-art results on KITTI for depth estimation and pose estimation tasks and competitive performance on the other tasks.”
- Also stated in: § I. INTRODUCTION · plain
“We obtain state-of-the-art results for depth and pose estimation tasks on KITTI among monocular methods.”
Quote in the paperSubstantial claim- Also stated in: § I. INTRODUCTION · plain
May combine two claims.
PremisecitationFisheye cameras have strong radial distortion that cannot be corrected without disadvantages, including reduced field-of-view and resampling distortion artifacts at the periphery.
“Fisheye cameras have a strong radial distortion that cannot be corrected without disadvantages, including reduced field-of-view and resampling distortion artifacts at the periphery [1].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [1] source
Classified as probably not a substantial claim. May combine two claims.
InterpretationreasoningPedestrians and vehicles are non-rigid and diverse in appearance, unlike traffic lights and traffic signs, making them suitable for polygon regression.
“Unlike traffic lights and traffic signs, these categories are non-rigid in nature and quite diverse in appearance, making them suitable for polygon regression.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe contributed features and the synergy between tasks help OmniDet achieve good scene understanding with high accuracy in each task's predictions.
“All our contributed features and the synergy between tasks help the OmniDet framework to achieve a good scene understanding with high accuracy in each task's predictions.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCylindrical rectification provides a good trade-off between loss of field-of-view and reducing distortion.
“We also experiment using cylindrical rectification (Cyl Rect.), which provides a good trade-off between loss of field-of-view and reducing distortion [29].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [29] source
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
InterpretationreasoningSampling 24 points with high curvature values from each object instance contour helps regress better polygon shapes, because points at high curvature define the shape of the object contours.
“Learning these points helps to regress better polygon shapes, as these points at high curvature define the shape of the object contours.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningAs weights are shared in the Siamese encoder, the previous frame's encoder can be saved and re-used instead of re-computing.
“As the weights are shared in the Siamese encoder, the previous frame's encoder can be saved and re-used instead of re-computing.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationMotion is a dominant cue in automotive scenes, and it requires at least two frames or the use of dense optical flow.
“Motion is a dominant cue in automotive scenes, and it requires at least two frames or the use of dense optical flow [8].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [8] source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe semantic and motion mask contribution has the potential to solve the infinite distance issue.
“Additionally, this contribution possesses the potential to solve the infinite distance issue.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationFisheye cameras are mounted low on a vehicle and are susceptible to lens soiling due to splashing of mud or water from the road.
“Fisheye cameras are mounted low on a vehicle and are susceptible to lens soiling due to splashing of mud or water from the road.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [7] source
May combine two claims.
InterpretationreasoningInformation about depth/distance of dynamic objects is essential in autonomous driving; otherwise, an infinite depth issue occurs during inference and contaminates reconstruction losses.
“As dynamic objects violate the static world assumption, information about their depth/distance is essential in autonomous driving; else, we would encounter the infinite depth issue during the inference stage and incur contamination to the reconstruction losses.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Dynamic objects violate the static world assumption.
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationThe proposed approach can, in principle, be applied to any fisheye projection model of choice.
“The proposed approach can, in principle, be applied to any fisheye projection model of choice explained in [1].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [1] source
May combine two claims.
InterpretationreasoningGeneric object detection using geometric cues like motion or depth for rare objects complements the detection of standard objects and provides more robustness.
“Thus, generic object detection using geometric cues like motion or depth for rare objects is typically used.They will also complement the detection of standard objects and provide more robustness.”
Quote in the paperNot a substantial claim- Rests on an assumption: Semantic tasks typically require a large annotated dataset covering various objects, and it is infeasible to practically cover every possible object.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationSelf-supervised methods have recently dominated depth estimation, which has also been demonstrated on fisheye images.
“Self-supervised methods have recently dominated depth estimation, which has also been demonstrated on fisheye images [9].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationcitationThe paper builds upon CAM-Convs and generalizes to arbitrary camera geometries, including fisheye cameras.
“We build upon this work and generalize to arbitrary camera geometries, including fisheye cameras.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningTraining with the camera geometry tensor makes the network camera-independent and better generalizes to images taken from a different camera.
“The aforementioned training strategy makes the network camera-independent and better generalizes to images taken from a different camera.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationMao et al. illustrated that multi-task learning improves adversarial robustness, which is critical for safety applications.
“Mao et al. [4] illustrated that multi-task learning improves adversarial robustness, which is critical for safety applications.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [4] source
May combine two claims.
InterpretationreasoningThe discriminative features loss and the L_cvt loss help gradient descent approaches transit smoothly towards optimal solutions.
“These losses help the gradient descent approaches to transit smoothly towards optimal solutions.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningA single model for these four cameras instead of four individual models would have practical advantages including improved efficiency on the embedded system, improved training by access to a larger dataset and regularization through different views, and maintenance and certification of one model instead of four.
“A single model for these four cameras instead of four individual models would also have several practical advantages such as (1) an improved efficiency on the embedded system requiring less memory and data rate to transmit, (2) an improved training by access to a larger dataset and regularization through different views, and (3) maintenance and certification of one model instead of four.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationcitationPixel-adaptive convolutions break up the spatial invariance of convolutions and allow incorporation of location-specific semantic knowledge into multi-level distance features.
“This, in particular, breaks up the spatial invariance of the convolutions and allows the incorporation of location-specific semantic knowledge into the multi-level distance features.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCAM-Convs uses camera-aware convolutions for pinhole cameras.
“The closest work is CAM-Convs [30], which uses camera-aware convolutions for pinhole cameras.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Interpretationnone givenThe multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other.
“Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationMultiNet was one of the first to demonstrate a three-task network on KITTI, and further works have primarily worked on a three-task setting.
“In the automotive multi-task setting, MultiNet [5] was one of the first to demonstrate a three task network on KITTI, and further works have primarily worked on a three task setting.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisecitationMulti-task learning is an efficient design pattern commonly used where most of the computation is shared across all tasks.
“Multi-task learning (MTL) is an efficient design pattern commonly used where most of the computation is shared across all the tasks [2], [3].”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [2] source
- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningVariance normalization re-scales the dispersion between different task loss distributions based on the previous n epochs; a large dispersion leads to a lower task weight, and a small dispersion to a higher one, tending to homogenize the learning speed of tasks.
“Variance normalization re-scales the dispersion between the different task loss distributions based on the previous n epochs.A large dispersion leads to a lower task weight, whereas a small dispersion to a higher one.Its final effect tends to homogenize the learning speed of tasks.”
Quote in the paperSubstantial claim
The paper relies on this without support.
Appearance variations of objects are larger due to spatially variant distortion, particularly for close-by objects.
“Appearance variations of objects are larger due to the spatially variant distortion, particularly for close-by objects.”
The paper relies on this without support.
Fisheye perception is relatively less explored despite its prevalence.
“Thus fisheye perception is a challenging task, however it is relatively less explored despite its prevalence.”
The paper relies on this without support.
Learning features for multiple tasks can act as a regularizer, improving generalization.
“Furthermore, learning features for multiple tasks can act as a regularizer, improving generalization.”
The paper relies on this without support.
Semantic tasks typically require a large annotated dataset covering various objects, and it is infeasible to practically cover every possible object.
“The semantic tasks typically require a large annotated dataset covering various objects.It is infeasible to practically cover every possible object.”
The paper relies on this without support.
The visual odometry task is required to place the detected objects in a temporally consistent map.
“Finally, the visual odometry task is required to place the detected objects in a temporally consistent map.”
The paper relies on this without support.
Fisheye images have considerably larger homogeneous areas than their rectilinear counterpart.
“as fisheye images have considerably larger homogeneous areas than their rectilinear counterpart.”
The paper relies on this without support.
Although the underlying camera intrinsics model is the same for a particular family of vehicles, manufacturing variations require calibration of each camera instance.
“Although the underlying camera intrinsics model is the same for a particular family of vehicles, there are variations due to manufacturing processes, which require the calibration of each camera instance.”
The paper relies on this without support.
Calibration can vary after deployment due to high environmental temperature or aging.
“Even after deployment, calibration can vary due to high environmental temperature or due to aging.”
The paper relies on this without support.
Public datasets have a single camera instance for both the training and test dataset.
“This contrasts with public datasets, which have a single camera instance for both the training and test dataset.”
The authors state this as an assumption.
Dynamic objects violate the static world assumption.
“As dynamic objects violate the static world assumption, information about their depth/distance is essential in autonomous driving; else, we would encounter the infinite depth issue during the inference stage and incur contamination to the reconstruction losses.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Equal weighting is not worst on every reported measure
MinorTable III reports a lower (better) squared relative error for equal weighting (0.058) than for geometric weighting (0.061), contradicting the unqualified statement that equal weighting is the worst.
“As shown in Table III, equal weighting is the worst”
“Single Task 0.060 0.304 72.5 94.8 68.1 94.1 63.5 Equal 0.058 0.302 70.3 92.7 67.3 93.3 64.6 DTP [25] 0.047 0.281 75.8 95.6 75.3 95.3 67.9 DWA [26] 0.054 0.293 75.4 95.2 74.7 95.1 67.5 Geometric [27] 0.061 0.297 74.2 94.1 73.2 94.3 66.7”
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1× · see citing works
“It is a linear combination of the general robust pixel-wise loss term [12] and the Structural Similarity (SSIM) [13] as described in [14].”
MethodFocal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · cited 1× · see citing works
“During training, the network predicts the posterior probability Y t which is optimized in a supervised fashion by Lovasz-Softmax [20] loss, and Focal [21] loss for handling class imbalance instead of the cross-entropy loss used in our previous work.”
MethodYOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141 · cited 1× · see citing works
“We adapted the YOLOv3 [19] decoder to output polygons and other representations listed above for a uniform comparison.”
MethodAre we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 1× · see citing works
“We systematically train and test all our single and multi-task models on Woodscape [29], a Surround View fisheye dataset with annotations for multiple tasks and on the pinhole camera datasets KITTI [35] and Cityscapes [36].”
MethodThe Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 1× · see citing works
“We systematically train and test all our single and multi-task models on Woodscape [29], a Surround View fisheye dataset with annotations for multiple tasks and on the pinhole camera datasets KITTI [35] and Cityscapes [36].”
MethodVision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · cited 1× · see citing works
“To facilitate comparison to previous methods, we also train our distance estimation method in the classical depth estimation setting on the KITTI Eigen split [39] whose results are shown in Table V.”
MethodMultiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · see citing works
Douglas, D. H., & Peucker, T. K. (1973) · Cartographica The International Journal for Geographic Information and Geovisualization · doi:10.3138/fm57-6770-u75u-7727 · Cited by 4,252 · cited 1× · see citing works
“Then we reduce the set of points using the algorithm in [18] to get the most representative simplified curves.”
MethodMulti-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
Cipolla, R., Gal, Y., & Kendall, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00781 · Cited by 3,657 · cited 1× · see citing works
“We evaluate the uncertainty loss from Kendall [23], the gradient magnitude normalization GradNorm [24], the dynamic task prioritization DTP [25], the dynamic weight average DWA [26] and the geometric loss [27].”
MethodDepth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · cited 1× · see citing works
“We also evaluate using the Original [40] as well as Improved [41] ground truth depth maps.”
MethodUnsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · cited 1× · see citing works
“In Table VI, we report the average trajectory error in meters of the pose estimation network by following the same protocols by Zhou [43] on the official KITTI odometry split (containing 11 sequences with ground-truth (GT) odometry acquired with the IMU/GPS measurements, which is used for…”
MethodEigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · cited 1× · see citing works
“[37] for self-supervised depth estimation with an input size of 640 × 192 px for all the tasks.”
MethodDigging Into Self-Supervised Monocular Depth Estimation
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00393 · Cited by 2,561 · cited 1× · see citing works
“The total loss comprised of a reconstruction matching term L r , regularization term L s , which enforces edge-aware smoothness inside the distance map ˆ D t as introduced in [10].”
BackgroundOn the Variance of the Adaptive Learning Rate and BeyondPreprint
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Han, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03265 · Cited by 1,529 · cited 1× · see citing works
“We employ the Ranger (RAdam [33] + LookAhead [34]) optimizer to minimize the training objective function.”
MethodEnd-To-End Multi-Task Learning With Attention
Liu, S., Johns, E., & Davison, A. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00197 · Cited by 1,299 · cited 1× · see citing works
“We evaluate the uncertainty loss from Kendall [23], the gradient magnitude normalization GradNorm [24], the dynamic task prioritization DTP [25], the dynamic weight average DWA [26] and the geometric loss [27].”
MethodGeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
Yin, Z., & Shi, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00212 · Cited by 1,280 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017) · arXiv · doi:10.48550/arxiv.1708.02002 · Cited by 1,227 · see citing works
Berman, M., Triki, A. R., & Blaschko, M. B. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00464 · Cited by 1,114 · cited 2× · see citing works
“During training, the network predicts the posterior probability Yt which is optimized in a supervised fashion by Lovasz-Softmax [20] loss, and Focal [21] loss for handling class imbalance instead of the cross-entropy loss used in our previous work.”
MethodExploring Self-Attention for Image Recognition
Zhao, H., Jia, J., & Koltun, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01009 · Cited by 1,002 · cited 2× · see citing works
“3) Linking Self-Attention and Semantic features to 2D detection: To leverage the multi-task learning setup, at first, we extract the Self-Attention Network (SAN) [28] encoder features and feed it as an input signal to the Eq.”
MethodUhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924 · cited 1× · see citing works
“We also evaluate using the Original [40] as well as Improved [41] ground truth depth maps.”
MethodMulti-Task Learning for Dense Prediction Tasks: A Survey
Vandenhende, S., Georgoulis, S., Gansbeke, W. V., Proesmans, M., Dai, D., & Gool, L. V. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3054719 · Cited by 881 · cited 1× · see citing works
“Multi-task learning (MTL) is an efficient design pattern commonly used where most of the computation is shared across all the tasks [2], [3].”
BackgroundGradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask NetworksPreprint
Chen, Z., Badrinarayanan, V., Lee, C., & Rabinovich, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02257 · Cited by 797 · see citing works
An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
Liu, R., Lehman, J., Molino, P., Such, F. P., Frank, E., Sergeev, A., & Yosinski, J. (2018) · arXiv · doi:10.48550/arxiv.1807.03247 · Cited by 784 · see citing works
3D Packing for Self-Supervised Monocular Depth Estimation
Guizilini, V., Ambrus, R., Pillai, S., Raventos, A., & Gaidon, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00256 · Cited by 741 · see citing works
A General and Adaptive Robust Loss Function
Barron, J. T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00446 · Cited by 737 · cited 1× · see citing works
“It is a linear combination of the general robust pixel-wise loss term [12] and the Structural Similarity (SSIM) [13] as described in [14].”
MethodOn the detection of dominant points on digital curves
Teh, C.-H., & Chin, R. T. (1989) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.31447 · Cited by 730 · cited 1× · see citing works
“We adopt the algorithm in [17] to detect the dominant points in a given curved shape, which best represents the object.”
MethodMultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving
Teichmann, M., Weber, M., Zollner, M., Cipolla, R., & Urtasun, R. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500504 · Cited by 681 · cited 1× · see citing works
“In the automotive multi-task setting, MultiNet [5] was one of the first to demonstrate a three task network on KITTI, and further works have primarily worked on a three task setting.”
MethodKokkinos, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.579 · Cited by 657 · cited 1× · see citing works
“We also trained soiling jointly using asynchronous backpropagation [22], but it achieved the same accuracy as using the frozen encoder.”
MethodRanjan, A., Jampani, V., Balles, L., Kim, K., Sun, D., Wulff, J., & Black, M. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01252 · Cited by 654 · see citing works
Casser, V., Pirk, S., Mahjourian, R., & Angelova, A. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33018001 · Cited by 473 · cited 2× · see citing works
“Using the online refinement method from [42], we obtain a significant improvement.”
MethodLookahead Optimizer: k steps forward, 1 step backPreprint
Zhang, M. R., Lucas, J., Hinton, G., & Ba, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08610 · Cited by 470 · cited 1× · see citing works
“We employ the Ranger (RAdam [33] + LookAhead [34]) optimizer to minimize the training objective function.”
MethodDynamic Task Prioritization for Multitask Learning
Guo, M., Haque, A., Huang, D.-A., Yeung, S. Y., & Fei-Fei, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_17 · Cited by 377 · cited 1× · see citing works
“We evaluate the uncertainty loss from Kendall [23], the gradient magnitude normalization GradNorm [24], the dynamic task prioritization DTP [25], the dynamic weight average DWA [26] and the geometric loss [27].”
MethodPixel-Adaptive Convolutional Neural Networks
Su, H., Jampani, V., Sun, D., Gallo, O., Learned-Miller, E., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01142 · Cited by 348 · cited 1× · see citing works
“…work [14], to better incorporate the semantic knowledge extracted from the segmentation branch of the multi-task network into the distance estimation, we incorporate it using pixel-adaptive convolutions [32] (PAC) to distill the knowledge from the semantic features into the distance decoder.”
MethodWoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving
Yogamani, S., Hughes, C., Horgan, J., Sistu, G., Chennupati, S., Uricar, M., Milz, S., Simon, M., Amende, K., Witt, C., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00940 · Cited by 303 · cited 3× · see citing works
“In the Woodscape fisheye dataset [29], there are 12 different cameras with slight intrinsic variations to evaluate this effect.”
BackgroundMethodSelf-citationGradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
Zhao, C., Badrinarayanan, V., Lee, C., & Rabinovich, A. (2018) · International Conference on Machine Learning · Cited by 275 · see citing works
Chen, Y., Schmid, C., & Sminchisescu, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00716 · Cited by 264 · see citing works
Feature-Metric Loss for Self-supervised Learning of Depth and Egomotion
Shu, C., Yu, K., Duan, Z., & Yang, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_34 · Cited by 254 · cited 3× · see citing works
“We incorporate feature-metric losses from [11] where L dis and L cvt are computed on I t ’s feature representation, where we learn the features using a self-attention autoencoder.”
BackgroundMethodInfluentialPoly-YOLO: higher speed, more precise detection and instance segmentation for YOLOv3
Hurtík, P., Molek, V., Hůla, J., Vajgl, M., Vlašánek, P., & Nejezchleba, T. (2022) · Neural Computing and Applications · doi:10.1007/s00521-021-05978-9 · Cited by 176 · cited 1× · see citing works
“The object contour can be uniformly sampled in the range of 360 ° split into N equal polygon vertices, each represented by the radial distance r from the object’s centroid as used in PolyYOLO [16].”
MethodCAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth
Facil, J. M., Ummenhofer, B., Zhou, H., Montesano, L., Brox, T., & Civera, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01210 · Cited by 130 · cited 2× · see citing works
“5) Normalized Coordinates ( nc ): Additionally, we add two channels of normalized coordinates [31], [30] whose values vary linearly between − 1 and 1 with respect to the image coordinates.”
MethodRashed, H., Mohamed, E., Sistu, G., Kumar, V., Eising, C., El-Sallab, A., & Yogamani, S. (2020) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00232 · Cited by 98 · cited 2× · see citing works
“We briefly summarize the details here and refer to our expanded paper [15] for more details on generalized object detection.”
BackgroundSelf-citationMODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving
Siam, M., Mahgoub, H., Zahran, M., Yogamani, S., Jagersand, M., & El-Sallab, A. (2018) · 2018 21st International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569744 · Cited by 95 · cited 2× · see citing works
“Motion is a dominant cue in automotive scenes, and it requires at least two frames or the use of dense optical flow [8].”
BackgroundMethodSelf-citationMultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning
Chennupati, S., Sistu, G., Yogamani, S. K., & Rawashdeh, S. A. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00159 · Cited by 94 · cited 1× · see citing works
“We evaluate the uncertainty loss from Kendall [23], the gradient magnitude normalization GradNorm [24], the dynamic task prioritization DTP [25], the dynamic weight average DWA [26] and the geometric loss [27].”
MethodSelf-citationKumar, V. R., Klingner, M., Yogamani, S. K., Milz, S., Fingscheidt, T., & Maeder, P. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00011 · Cited by 86 · cited 2× · see citing works
“2) Semantically-Guided Distance and Detection Decoder: Following our previous work [14], to better incorporate the semantic knowledge extracted from the segmentation branch of the multi-task network into the distance estimation, we incorporate it using pixel-adaptive convolutions [32] (PAC) to…”
MethodSelf-citationKumar, V. R., Hiremath, S. A., Bach, M., Milz, S., Witt, C., Pinard, C., Yogamani, S. K., & Mäder, P. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197319 · Cited by 73 · see citing works
Self-citationSMSnet: Semantic motion segmentation using deep convolutional neural networks
Vertens, J., Valada, A., & Burgard, W. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202211 · Cited by 69 · cited 1× · see citing works
“For the motion segmentation, we use the annotations provided by DeepMotion [38] for Cityscapes and KITTI MoSeg [8].”
MethodMultitask Learning Strengthens Adversarial Robustness
Mao, C., Gupta, A., Nitin, V., Ray, B., Song, S., Yang, J., & Vondrick, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_10 · Cited by 60 · see citing works
SoilingNet: Soiling Detection on Automotive Surround-View Cameras
Uricar, M., Krizek, P., Sistu, G., & Yogamani, S. (2019) · 2019 IEEE Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc.2019.8917178 · Cited by 55 · cited 2× · see citing works
“Compared to our previous work SoilingNet [7], we moved from the tiled output to pixel-level segmentation.”
BackgroundMethodSelf-citationKumar, V. R., Yogamani, S. K., Bach, M., Witt, C., Milz, S., & Mäder, P. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340732 · Cited by 48 · cited 3× · see citing works
“Fisheye cameras have a strong radial distortion that cannot be corrected without disadvantages, including reduced field-of-view and resampling distortion artifacts at the periphery [1].”
BackgroundMethodInfluentialSelf-citationNeurAll: Towards a Unified Visual Perception Model for Automated Driving
Sistu, G., Leang, I., Chennupati, S., Yogamani, S. K., Eising, C., Milz, S., & Rawashdeh, S. A. (2019) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2019.8917043 · Cited by 30 · cited 1× · see citing works
“Multi-task learning (MTL) is an efficient design pattern commonly used where most of the computation is shared across all the tasks [2], [3].”
BackgroundSelf-citationPoly-YOLO: higher speed, more precise detection and instance segmentation for YOLOv3Preprint
Hurtík, P., Molek, V., Hůla, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13243 · Cited by 25 · see citing works
Multitask Learning Strengthens Adversarial RobustnessPreprint
Mao, C., Gupta, A., Nitin, V., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07236 · Cited by 12 · see citing works
CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View DepthPreprint
Fácil, J. M., Ummenhofer, B., Zhou, H., Montesano, L., Brox, T., & Civera, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02028 · Cited by 9 · cited 2× · see citing works
“5) Normalized Coordinates ( nc ): Additionally, we add two channels of normalized coordinates [31], [30] whose values vary linearly between − 1 and 1 with respect to the image coordinates.”
MethodMaddu, P., Doherty, W., Sistu, G., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11066 · Cited by 8 · cited 1× · see citing works
“Specifically, there is only our previous work on multi-task learning: FisheyeMultiNet [6] which discusses a more straight forward three task network.”
BackgroundSelf-citation
Flagged first, then by citations
19 of the 72 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- geiger, 0, Proc Conf Comput Vis Pattern Recognit, 3354
Cited by
Sorted by citations
Review on Panoramic Imaging and Its Applications in Scene Understanding
Gao, S., Yang, K., Shi, H., Wang, K., & Bai, J. Y. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3216675 · Cited by 121 · see citing works
“other fisheye [324], [325], [326], [327], panoramic [328], [329], [330], and omnidirectional [331], [332], [333], [334]”
Surround-View Fisheye Camera Perception for Automated Driving: Overview, Survey & Challenges
Kumar, V. R., Eising, C., Witt, C., & Yogamani, S. K. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3235057 · Cited by 121 · see citing works
“Finally, a holistic real-time scene understanding for the near-field perception of the environment using cameras only was presented in OmniDet [123].”
MethodInfluentialSelf-citationNeuromorphic computing chip with spatiotemporal elasticity for multi-intelligent-tasking robots
Ma, S., Pei, J., Zhang, W., et al. (2022) · Science Robotics · doi:10.1126/scirobotics.abk2948 · Cited by 68 · see citing works
“Moreover, the application of multiple models ( 11 – 13 ) and multi-tasking NNs ( 14 – 17 ) can further improve the ability and adaptability of robots.”
MethodSynWoodScape: Synthetic Surround-View Fisheye Camera Dataset for Autonomous Driving
Sekkat, A. R., Dupuis, Y., Kumar, V. R., Rashed, H., Yogamani, S. K., Vasseur, P., & Honeiné, P. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3188106 · Cited by 64 · see citing works
Self-citationZhang, J., Ma, C., Yang, K., Roitberg, A., Peng, K., & Stiefelhagen, R. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3123070 · Cited by 52 · see citing works
“…of panoramic data, which is often captured through distortion-pronounced fisheye lenses [38], [39], [40] or multiple surround-view cameras [41], [42], [43], is challenging as it entails a set of hard tasks like distortion elimination, camera synchronization and calibration, as well as data…”
Near-Field Perception for Low-Speed Vehicle Automation Using Surround-View Fisheye Cameras
Eising, C., Horgan, J., & Yogamani, S. K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3127646 · Cited by 53 · see citing works
Self-citationEfficient Multi-Task RGB-D Scene Analysis for Indoor Environments
Seichter, D., Fischedick, S. B., Köhler, M., & Grob, H.-M. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892852 · Cited by 60 · see citing works
“With uncertainty weighting [52], GradNorm [53], Dynamic Weight Average [54], and VarNorm [55], several approaches for determining the weights automatically have been proposed.”
MethodVision-Based Learning for Drones: A Survey
Xiao, J., Zhang, R., Zhang, Y., & Feroskhan, M. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3564184 · Cited by 55 · see citing works
“With dual-fisheye cameras, the drone can achieve better navigation capability [50] and omnidirectional visual perception [48], [51].”
Weather and Light Level Classification for Autonomous Driving: Dataset, Baseline and Active Learning
Dhananjaya, M. M., Kumar, V. R., & Yogamani, S. K. (2021) · doi:10.1109/itsc48978.2021.9564689 · Cited by 42 · see citing works
“enables easy integration with multi-task shared encoder models where this task can be added to existing perception model with incremental increase in complexity [22], [23].”
BackgroundSelf-citationTowards Compact Autonomous Driving Perception With Balanced Learning and Multi-Sensor Fusion
Natan, O., & Miura, J. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3149370 · Cited by 39 · see citing works
“We consider using the multi-task learning (MTL) approach since handling each task with a single-task model can be very costly and inefficient [12] [13] [14].”
Ma, C., Zhang, J., Yang, K., et al. (2021) · doi:10.1109/itsc48978.2021.9564920 · Cited by 38 · see citing works
“Segmentation of panoramic data, which is often captured through distortion-heavy fisheye lenses [22], [23] or multiple cameras [2], [24], is especially challenging as it requires eliminating distortions, synchronizing and calibrating the cameras, and fusing the data, which leads to higher latency.”
Multisensor Information Fusion: Future of Environmental Perception in Intelligent Vehicles
Zhang, Y., Tu, C., Gao, K., & Wang, L. (2024) · Journal of Intelligent and Connected Vehicles · doi:10.26599/jicv.2023.9210049 · Cited by 45 · see citing works
Survey on Multi-Task Learning in Smart Transportation
Alzahrani, M. Y., Wang, Q., Liao, W., Chen, X., & Yu, W. (2024) · IEEE Access · doi:10.1109/access.2024.3355034 · Cited by 28 · see citing works
“…motion segmentation to differentiate between car movement and other objects movement, object detection, and lens soiling detection to detect when the camera vision is not clear, for example, by dirt [76], synergized decoders was used in this method to make the tasks related to each other.”
MethodLiMoSeg: Real-time Bird’s Eye View based LiDAR Motion Segmentation
Mohapatra, S., Hodaei, M., Yogamani, S. K., Milz, S., Gotzig, H., Simón, M., Rashed, H., & Maeder, P. (2022) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010866000003124 · Cited by 31 · see citing works
Self-citationDeep Learning for Omnidirectional Vision: A Survey and New PerspectivesPreprint
Ai, H., Cao, Z., Zhu, J., Bai, H., Chen, Y., & Wang, L. (2022) · doi:10.36227/techrxiv.19807699 · Cited by 36 · see citing works
LF-VIO: A Visual-Inertial-Odometry Framework for Large Field-of-View Cameras with Negative Plane
Wang, Z., Yang, K., Shi, H.-W., Li, P., Gao, F., & Wang, K.-W. (2022) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros47612.2022.9981217 · Cited by 24 · see citing works
“Moreover, based on different sensor types, many other odometry estimation methods emerge in the field, including fisheye-based [29]– [33], multi-camera-based [3], [34]–[38], and systems that combine omnidirectional imaging with other sensing units like GPS [39] and LiDAR sensors [40], [41].”
BackgroundJaus, A., Yang, K., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3232897 · Cited by 20 · see citing works
“(FoV), many surrounding understanding platforms are based on fisheye images or multiple cameras [63], [64], [65], [66],”
BackgroundReview on Functional Testing Scenario Library Generation for Connected and Automated Vehicles
Zhu, Y., Wang, J., Meng, F., & Liu, T. (2022) · Sensors · doi:10.3390/s22207735 · Cited by 19 · see citing works
Ai, H., & Wang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00947 · Cited by 19 · see citing works
“…environ ment in one shot with an ultra-wide field-of-view (FoV) of 180°x360°, which is broadly applied to applications, e.g., autonomous driving [1, 19, 42, 43, 49] and virtual reality *Corresponding author (e-mail: linwang@ust.hk) With only lM more parameters, our depth result is more accurate.”
BackgroundZhou, J. N., Yang, D., Song, T., et al. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.104966 · Cited by 18 · see citing works
A Review of Vision-Based Multi-Task Perception Research Methods for Autonomous Vehicles
Wang, H., Li, J., & Dong, H. (2025) · Sensors · doi:10.3390/s25082611 · Cited by 17 · see citing works
“OmniDet [88] is a multi-task visual perception system for surround-view fisheye lenses.”
An Online Learning System for Wireless Charging Alignment Using Surround-View Fisheye Cameras
Dahal, A., Kumar, V. R., Yogamani, S. K., & Eising, C. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3182165 · Cited by 19 · see citing works
“While we provide a short overview of the baseline model here, the reader is referred to [31] for more complete details.”
BackgroundMethodInfluentialSelf-citationAdversarial Attacks on Multi-task Visual Perception for Autonomous Driving
Sobh, I., Hamed, A., Kumar, V. R., & Yogamani, S. K. (2021) · Journal of Imaging Science and Technology · doi:10.2352/j.imagingsci.technol.2021.65.6.060408 · Cited by 18 · see citing works
Self-citationWu, Z., Gan, Y., Li, X., Wu, Y., Wang, X., Xu, T., & Wang, F. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3218594 · Cited by 16 · see citing works
“areas, valet parking aims to drive the vehicle into the drop-off area such as parking slots, which faces high requirements in perception [12].”
BackgroundMethodFishDreamer: Towards Fisheye Semantic Completion via Unified Image Outpainting and Segmentation
Shi, H., Li, Y., Yang, K., et al. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00684 · Cited by 16 · see citing works
A Hybrid Sparse-Dense Monocular SLAM System for Autonomous Driving
Gallagher, L., Kumar, V. R., Yogamani, S. K., & McDonald, J. (2021) · European Conference on Mobile Robots · doi:10.1109/ecmr50962.2021.9568797 · Cited by 17 · see citing works
Self-citationMumuni, F., Mumuni, A., & Amuzuvi, C. K. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2022.100416 · Cited by 19 · see citing works
PGDS-YOLOv8s: An Improved YOLOv8s Model for Object Detection in Fisheye Images
Yang, D., Zhou, J. N., Song, T., Zhang, X., & Song, Y. (2023) · Applied Sciences · doi:10.3390/app14010044 · Cited by 14 · see citing works
SlaBins: Fisheye Depth Estimation using Slanted Bins on Road Environments
Lee, J.-S., Cho, G., Park, J., Kim, K., Lee, S., Kim, J.-H., Jeong, S.-G., & Joo, K. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00805 · Cited by 16 · see citing works
“Overall, SlaBins outperforms the other baseline approaches [3, 20, 21].”
BackgroundMethodResultInfluentialOptical flow for autonomous driving: Applications, challenges and improvements
Shen, S., Kerofsky, L. J., & Yogamani, S. K. (2023) · Electronic Imaging · doi:10.2352/ei.2023.35.16.avm-128 · Cited by 16 · see citing works
Self-citationDeep Learning for Omnidirectional Vision: A Survey and New PerspectivesPreprint
Ai, H., Cao, Z., Zhu, J., Bai, H., Chen, Y., & Wang, L. (2022) · doi:10.36227/techrxiv.19807699.v1 · Cited by 15 · see citing works
Cross-task and time-aware adversarial attack framework for perception of autonomous driving
Lu, Y., Liu, N., Li, Y., et al. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111652 · Cited by 14 · see citing works
Djenouri, Y., Srivastava, G., Djenouri, D., et al. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3165156 · Cited by 14 · see citing works
“[27] developed a deep learning network to identify vehicles captured in fisheye images.”
BackgroundBEV-MODNet: Monocular Camera based Bird's Eye View Moving Object Detection for Autonomous Driving
Rashed, H., Essam, M., Mohamed, M., Sallab, A. E., & Yogamani, S. K. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9564667 · Cited by 15 · see citing works
“On the other hand, the model in [10] takes two sequential images as input and tries to understand motion implicitly.”
BackgroundMethodSelf-citationInfluence of Camera-LiDAR Configuration on 3D Object Detection for Autonomous Driving
Li, Y., Hu, H., Liu, Z., Xu, X., Huang, X., & Zhao, D. X. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610896 · Cited by 16 · see citing works
“With this respect, Li-DARs and cameras are widely used for the perception of autonomous vehicles due to their capability to obtain rich information from the environment in real-time settings [21]– [23].”
BackgroundSong, F., Ma, R., Leí, T., & Peng, Z. (2023) · Remote Sensing · doi:10.3390/rs15092364 · Cited by 13 · see citing works
Hernandez‐Diaz, K., Alonso‐Fernandez, F., & Bigün, J. (2023) · IEEE Access · doi:10.1109/access.2023.3315234 · Cited by 12 · see citing works
“…editor coordinating the review of this manuscript and approving it for publication was Genoveffa Tortora . and extract highly discriminative features, as well as to easily adapt to different applications such as medical data [6], autonomous driving [7], or, in our case, biometric recognition [8].”
BackgroundAdvanced Techniques for Perception and Localization in Autonomous Driving Systems: A Survey
Sellat, Q., & Ramasubramanian, K. (2022) · Optical Memory and Neural Networks · doi:10.3103/s1060992x22020084 · Cited by 11 · see citing works
DarSwin: Distortion Aware Radial Swin Transformer
Athwale, A., Afrasiyabi, A., Lagüe, J., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00545 · Cited by 9 · see citing works
Adaptive Attention Module for Image Recognition Systems in Autonomous Driving
Ma, X., Hu, K., Sun, X., & Chen, S. (2024) · International Journal of Intelligent Systems · doi:10.1155/2024/3934270 · Cited by 9 · see citing works
Video Waterdrop Removal via Spatio-Temporal Fusion in Driving Scenes
Wen, Q., Wu, Y., & Chen, Q. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161240 · Cited by 13 · see citing works
“With the increasing applications of visual perception methods [1], [2], [3], [4] in robotics and autonomous driving, the robustness of these methods has become more and more important in their visual system.”
MethodSemantics and Contour Based Interactive Learning Network for Building Footprint Extraction
Zhu, X., Zhang, X., Zhang, T., et al. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3317080 · Cited by 9 · see citing works
“OmniDet [39] is aimed at a multitasking visual perception system for circumferential fisheye lenses, integrating multiple tasks of depth estimation, visual odometry, semantic and motion segmentation, object detection, and lens contamination detection.”
BackgroundFisheyeSuperPoint: Keypoint Detection and Description Network for Fisheye Images
Konrad, A., Eising, C., Sistu, G., McDonald, J., Villing, R. C., & Yogamani, S. K. (2022) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010795400003124 · Cited by 10 · see citing works
“Such multi-task models share the same encoder, which can improve performance and reduce latency for all tasks (Leang et al., 2020), (Ravikumar et al., 2021).”
BackgroundMethodSelf-citationLearning Panoptic Segmentation from Instance Contours
Chennupati, S., Narayanan, V., Sistu, G., Yogamani, S. K., & Rawashdeh, S. A. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560798 · Cited by 8 · see citing works
Self-citationFisheyePixPro: Self-supervised pretraining using Fisheye images for semantic segmentation
Cheke, R., Sistu, G., Eising, C., van de Ven, P., Kumar, V. R., & Yogamani, S. (2022) · Autonomous Vehicles and Machines · doi:10.2352/ei.2022.34.16.avm-147 · Cited by 11 · see citing works
“Several experiments have been conducted to enhance the performance of CNN on fisheye dataset by investigating the impact of adversarial attacks[15] on a multi-task visual perception network [16].”
BackgroundSelf-citationCityscapes TL++: Semantic Traffic Light Annotations for the Cityscapes Dataset
Janosovits, J. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812144 · Cited by 9 · see citing works
Neuroevolutionary multi-objective approaches to trajectory prediction in autonomous vehicles
Stapleton, F., Galván, E., Sistu, G., & Yogamani, S. K. (2022) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3520304.3528984 · Cited by 8 · see citing works
Self-citationSwin‐fisheye: Object detection for fisheye images
Zhang, D., Yang, T., & Zhao, B. (2024) · IET Image Processing · doi:10.1049/ipr2.13216 · Cited by 8 · see citing works
Automotive DNN-Based Object Detection in the Presence of Lens Obstruction and Video Compression
Baris, G., Li, B., Chan, P. H., Avizzano, C. A., & Donzella, V. (2025) · IEEE Access · doi:10.1109/access.2025.3544773 · Cited by 7 · see citing works
A Review On Thermal Infrared Semantic Distribution for Nightfall Drive
Bandi, M., & Reeja, S. R. (2022) · International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT) · doi:10.1109/icicict54557.2022.9917651 · Cited by 8 · see citing works
“WoodScape[31] is the very first dataset with a fisheye lens acquired by four a vehicle with fisheye cameras for autonomous driving.”
BackgroundInfluentialLu, R., Shang, J., Wu, J. Y., et al. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3523287 · Cited by 7 · see citing works
“Thecontinuous development of optical imaging technology and computer vision has enabled the wide usage of panoramic vision in fields such as robotics [4], [5] and autonomous driving [6], [7].”
Internal thread defect detection system based on multi-vision
Dou, X., Xue, C., Zhang, G., & Jiang, Z. (2024) · PLoS ONE · doi:10.1371/journal.pone.0304224 · Cited by 8 · see citing works
Ensemble-Based Semi-Supervised Learning to Improve Noisy Soiling Annotations in Autonomous Driving
Uřičář, M., Sistu, G., Yahiaoui, L., & Yogamani, S. K. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9565086 · Cited by 6 · see citing works
“In general, there is limited work on fisheye camera perception for tasks such as object detection [1], depth estimation [2], SLAM [3] and multi-task learning [4].”
BackgroundSelf-citationTrained Trajectory based Automated Parking System using Visual SLAM on Surround View CamerasPreprint
Tripathi, N., & Yogamani, S. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.02161 · Cited by 7 · see citing works
Self-citationLiDAR-BEVMTN: Real-Time LiDAR Bird’s-Eye View Multi-Task Perception Network for Autonomous Driving
Mohapatra, S., Yogamani, S. K., Kumar, V. R., Milz, S., Gotzig, H., & Mäder, P. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3510642 · Cited by 5 · see citing works
“It builds on our previous work OmniDet [42] is a multi-task camera perception model.”
MethodSelf-citationWu, Z., Zhang, W., Wang, J., Wang, M., Gan, Y., Gou, X., Fang, M., & Song, J. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636707 · Cited by 5 · see citing works
“Moreover, OmniDet [33] presents a multi-task visual perception network to capture the surrounding environment.”
BackgroundCyber-Attacks on Wheeled Mobile Robotic Systems with Visual Servoing Control
Jokić, A., Khazraei, A., Petrović, M., et al. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341376 · Cited by 5 · see citing works
“Robotic systems rely heavily on visual information for a wide variety of tasks such as control [21], perception [22], decision-making, and planning [23].”
BackgroundSTRIDE: Street View-based Environmental Feature Detection and Pedestrian Collision Prediction
González, C., Ayobi, N., Escallón, F., Baldovino-Chiquillo, L., Wilches-Mogollon, M. A., Pasos, D., Ramírez, N., Pinzón, J. D., Sarmiento, O. L., Quistberg, D. A., et al. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00347 · Cited by 5 · see citing works
“Further approaches use data gathered with fisheye and surround-view cameras annotated for semantic scene parsing [24, 92, 29, 43, 49], thus introducing significant distortion and deformation caused by this type of cameras.”
MethodLearning Cross-modality Interaction for Robust Depth Perception of Autonomous Driving
Liang, Y., Chen, N., Yu, Z., Tang, L., Yu, H., Guo, B., & Zeng, D. D. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3650039 · Cited by 5 · see citing works
“For example, although depth sensing based on RGB images can provide high-resolution depth information, its performance is limited by lower accuracy and short perception distance [14, 19, 33, 52, 57].”
BackgroundCao, H., Li, Y., Liu, Y., et al. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802087 · Cited by 5 · see citing works
“The training process remains consistent with the source code released by OmniDet [18].”
InfluentialMa, C., Zhang, J., Yang, K., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06383 · Cited by 5 · see citing works
“Segmentation of panoramic data, which is often captured through distortion-heavy fisheye lenses [22], [23] or multiple cameras [2], [24], is especially challenging as it requires eliminating distortions, synchronizing and calibrating the cameras, and fusing the data, which leads to higher latency.”
VM-MODNet: Vehicle Motion aware Moving Object Detection for Autonomous Driving
Rashed, H., Sallab, A. E., & Yogamani, S. K. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9564662 · Cited by 4 · see citing works
“We start with the OmniDet [18] motion segmentation network using two-stream RGB only network.”
MethodSelf-citationA Comparison of Spherical Neural Networks for Surround-View Fisheye Image Semantic Segmentation
Manzoor, A., Mohandas, R., Scanlan, A., Grua, E. M., Collins, F., Sistu, G., & Eising, C. (2025) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2025.3541891 · Cited by 5 · see citing works
“A common yet simplistic approach for applying CNNs to fisheye imagery involves rectifying fish-eye images before applying standard algorithms; however, this method introduces significant drawbacks, such as a reduced field of view and resampling distortion artifacts near the edges [19].”
SpikiLi: A Spiking Simulation of LiDAR based Real-time Object Detection for Autonomous Driving
Mohapatra, S., Mesquida, T., Hodaei, M., Yogamani, S. K., Gotzig, H., & Maeder, P. (2022) · International Conference on Event-based Control, Communication, and Signal Processing · doi:10.1109/ebccsp56922.2022.9845647 · Cited by 4 · see citing works
Self-citationDarSwin-Unet: Distortion Aware Architecture
Athwale, A., Shili, I., Bergeron, É., et al. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv61041.2025.00840 · Cited by 3 · see citing works
“Similar to our method, [20,21,23,25] propose a camera-aware depth estimation network to handle the severe distortion of fisheye cameras: [25] encode the camera intrinsic parameters as a tensor; and [23] propose a self-supervised depth estimation method which relies on the lens distortion parameter…”
A Feature-based Approach for the Recognition of Image Quality Degradation in Automotive Applications
Bauer, F. (2023) · International Conference on Systems, Signals and Image Processing · doi:10.1109/iwssip58668.2023.10180245 · Cited by 3 · see citing works
“They allow the system to issue warnings about malfunction or even to disable affected functions [2]–[4].”
BackgroundFishSegSSL: A Semi-Supervised Semantic Segmentation Framework for Fish-Eye Images
Paul, S., Patterson, Z., & Bouguila, N. (2024) · Journal of Imaging · doi:10.3390/jimaging10030071 · Cited by 3 · see citing works
“Another key observation we find is that the very recent literature is based on multimodal or multitask models where the signals from different tasks or domains help the model to learn representation from fish-eye images [4,27].”
BackgroundMethodA comprehensive study of fisheye image compression and perception for autonomous driving
Barakat, B., Sobh, I., Wong, C.-C., et al. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10831-w · Cited by 3 · see citing works
Fisheye object detection based on standard image datasets with 24-points regression strategy
Xu, X., Gao, Y., Liang, H. Z., Yang, Y. M., & Fu, M. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981891 · Cited by 3 · see citing works
“Rashed et. al [1], [14] adjusted the encoder part of the standard image object detection network.”
BackgroundCooperative Environmental Perception Task Offloading for Connected and Autonomous Vehicles
Lv, P., Huang, J., & Liu, H. (2023) · Electronics · doi:10.3390/electronics12173714 · Cited by 3 · see citing works
Autonomous Driving of Personal Mobility by Imitation Learning from Small and Noisy Dataset
Kobayashi, T., & Enomoto, T. (2024) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii58957.2024.10417607 · Cited by 3 · see citing works
“Note that although comparisons with the state-of-the-art methods (e.g. [23], [24]) are desirable in terms of autonomous driving technology, they are omitted in this paper because it is difficult to match the implementation conditions, making fair comparisons difficult.”
ZSPose: Instance-Level Zero-Shot Object Pose Estimation With Segment Anything Model
Yu, S. W., Zhai, D., Yin, J., et al. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3618671 · Cited by 2 · see citing works
“I NTRODUCTION 6 D POSE estimation is an important task in the field of computer vision, widely applied in areas such as robotic grasping [1], [2], [3], autonomous driving [4], and augmented reality [5].”
BNDCNet: Bilateral nonlocal decoupled convergence network for semantic segmentation
Ye, M., Chen, Z., Guo, Y., et al. (2023) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2023.104028 · Cited by 2 · see citing works
Robust Detection for Fisheye Camera Based on Contrastive Learning
Zhang, J., Tang, L., & Zhou, X. (2025) · Comput. Mater. Continua · doi:10.32604/cmc.2025.061690 · Cited by 3 · see citing works
“Fisheye cameras offer an omnidirectional field of view and can capture comprehensive visual information, making them particularly advantageous for applications in intelligent surveillance [1–3], unmanned aerial vehicles [4,5], virtual reality [6,7], autonomous driving [8–10], and robotics [11–13].”
Design of a preprocessing system suitable for Surround View Monitor based on Zynq
Oh, H., Hwang, G. H., Sim, H., et al. (2025) · International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC) · doi:10.1109/itc-cscc66376.2025.11137632 · Cited by 2 · see citing works
“Recently, SVMs have become a core component of advanced driver assistance systems (ADAS), and SVM technology has been actively researched in autonomous driving beyond parking [1].”
Zhang, Z., Xu, R., Zhang, W., Zhang, Z., Tan, X., Gong, J., Xie, Y., & Ma, L. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755159 · Cited by 2 · see citing works
“Mean-while, current multi-view depth estimation methods primarily target homogeneous setups, either pinhole-only [15, 27] or fisheye-only [16, 30], leaving the great potential of heterogeneous pinhole-fisheye depth estimation unexplored.”
Advancements in Fisheye Lens Image Stitching for Enhanced ADAS in Automated Vehicles
Harsha Vardhan Reddy., K., & Hegde, V. (2023) · International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS) · doi:10.1109/csitss60515.2023.10334173 · Cited by 2 · see citing works
“In order to embed the fisheye distortion model in both training and inference, they proposed a new adaption approach[10] based on camera geometry.”
MethodSemantic Segmentation Using Transfer Learning on Fisheye Images
Paul, S., Patterson, Z., & Bouguila, N. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00068 · Cited by 2 · see citing works
“While some prior works have focused on semantic segmentation of fisheye images, they either rely on multi-task or multi-modal approaches, leveraging learning signals from multiple tasks or modalities [5], [13].”
BackgroundDialesiotis, S., Papatsimouli, M., Asimopoulos, K. N., et al. (2024) · AIP conference proceedings · doi:10.1063/5.0234982 · Cited by 1 · see citing works
Sequential Cooperative Distillation for Imbalanced Multi-Task Learning
Feng, Q., Yao, J.-Y., Xie, M.-K., et al. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-2264-z · Cited by 1 · see citing works
Fisheye omnidirectional stereo depth estimation assisted with edge-awareness
Sun, J., Xue, H., Guo, S., et al. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1555785 · Cited by 1 · see citing works
Omnidirectional Spatial Modeling from Correlated Panoramas
Zhang, X., Fu, T., & Xu, Z. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3770977 · Cited by 1 · see citing works
“Omnidirectional scene understanding with 360 ◦ cameras plays important roles in a wide range of applications, such as embodied AI, autonomous driving, and virtual/augmented reality (VR/AR) [7, 13, 16].”
Distortion-Aware Depth Self-Updating for Self-Supervised Fisheye Monocular Depth Estimation
Xu, Y., & Dong, Q. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3661813 · Cited by 1 · see citing works
“To alleviate the above distortion problem in fisheye images, self-supervised fisheye monocular depth estimation methods for fisheye cameras [2], [10], [11], [12], [13], [14] have been explored in recent years.”
InfluentialMulti-lane modelling using convolutional neural networks and conditional random fields
Babu, G., Cheke, R., Sistu, G., & Yogamani, S. K. (2022) · Electronic Imaging · doi:10.2352/ei.2022.34.16.avm-148 · Cited by 1 · see citing works
Self-citationRobust Single-Cam Surround View Object Detection and Localization Using Memory Maps
Quan, Y., Kiefer, B., Messmer, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78113-1_8 · Cited by 1 · see citing works
Mumuni, F., Mumuni, A., & Amuzuvi, C. K. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4122364 · Cited by 1 · see citing works
Ramachandran, S., Sistu, G., McDonald, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08246
Self-citationPerformance monitoring of deep learning vision systems during deployment
Rahman, Q. M. (2022) · Queensland University of Technology · doi:10.5204/thesis.eprints.229733
Relaxing the Forget Constraints in Open World Recognition
Fontanel, D., Cermelli, F., Geraci, A., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-06427-2_62
Analysis of Faster R-CNN network prediction in the presence of lens occlusion and video compressionPreprint
Baris, G., Li, B., Chan, P. H., Avizzano, C. A., & Donzella, V. (2023) · doi:10.36227/techrxiv.23047412
Bndcnet: Bilateral Nonlocal Decoupled Convergence Network for Semantic SegmentationPreprint
Ye, M., Chen, Z., Guo, Y., et al. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4505116
Impact of Misperception on Emergence of Risk in Platoon of Autonomous Vehicles
Amini, A., Liu, G., Pandey, V. K., & Motee, N. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10383863
“On the other hand, with recent advancements in visual perception algorithms[9], almost all cars are equipped with cameras that allow them to perceive the environments for safety measures and collision avoidance.”
BackgroundJoint-Loss Enhanced Self-Supervised Learning for Refinement-Coupled Object 6D Pose Estimation
Mu, F., Sun, S., Huang, R., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611061
“6D object pose estimation holds great potential in various domains such as robotic grasping [1][2], autonomous driving [3][4], and augmented reality [5][6].”
BackgroundSynthetic Soiling Augmentation Using the Diffusion Model for Automotive Camera System
Heo, J., Kim, T., & Hwang, Y. (2024) · International Conference on Platform Technology and Service · doi:10.1109/platcon63925.2024.10830729
“Many existing researches attempt to perform soil detection or segmentation [11], [22].”
DepthFisheye: Efficient Fine-Tuning of Depth Estimation Models for Fisheye Cameras
Wu, W., Zhang, Z., Tan, X., Zhang, Z., & Ma, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-5815-2_1
Research on a Vision-Based Perception Strategy for Autonomous Vehicle Control
Xie, S. (2024) · International Conference on Information Technologies and Electrical Engineering (ICITEE) · doi:10.1145/3717934.3717994
UST-SU: a U-shaped video prediction network based on partial autoregression
Cui, Z., Tian, W., Luo, F., et al. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07486-2
“The modeling capability of motion patterns within videos has made it applicable in diverse tasks such as traffic flow prediction [1, 2], precipitation nowcasting [3, 4], human action recognition [5–7] and autonomous driving [8, 9].”
Unified interest point detection and description for perspective and Fisheye images
Xu, J., Han, D., Li, K., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-02487-w
GF-Net: Ghosting-fusion network for autonomous driving
Liu, C., Zhang, Z., Yao, J., et al. (2025) · Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering · doi:10.1177/09544070251350929
Weakly Supervised Monocular Fisheye Camera Distance Estimation with Segmentation Constraints
Zhang, Z., & Yang, X. (2025) · Electronics · doi:10.3390/electronics14173429
- Authors
- Ravi Kumar, V., Yogamani, S., Rashed, H., Sitsu, G., Witt, C., Leang, I., Milz, S., & Mader, P.
- Affiliations
- Valeo (Ireland) (IE)
- Valeo (Germany) (DE)
- Valeo (France) (FR)
- Technische Universität Ilmenau (DE)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2021-04
- Volume
- 6
- Issue
- 2
- Pages
- 2830-2837
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W3129414859
- Semantic Scholar
- 58fce566cc744fac4851353ee3e6132e8fbe20e5
- Cited by
- 117
- References
- 72
- Citations by year
- 2026: 13 · 2025: 24 · 2024: 18 · 2023: 21 · 2022: 27 · 2021: 13 · 2020: 1
- FWCI
- 6.33
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- surround-view cameras, fisheye images, multi-task learning, autonomous driving, depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, shared encoder, KITTI, cityscape
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
