CoChat
Sign Up
Cited by 15Advanced Vision and ImagingAdvanced Neural Network ApplicationsRobotics and Sensor-Based Localization

MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation

Open access
Yasarla, Rajeev, Cai, Hong, Jeong, Jisoo, Shi, Yunxiao, Garrepalli, Risheek, Porikli, Fatih·2023·2023 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv51070.2023.00804
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose MAMo, a novel memory and attention framework for monocular video depth estimation. MAMo can augment and improve any single-image depth estimation networks into video depth estimation models, enabling them to take advantage of the temporal information to predict more accurate depth. In MAMo, we augment model with memory which aids the depth prediction as the model streams through the video. Specifically, the memory stores learned visual and displacement tokens of the previous time instances. This allows the depth network to cross-reference relevant features from the past when predicting depth on the current frame. We introduce a novel scheme to continuously update the memory, optimizing it to keep tokens that correspond with both the past and the present visual information. We adopt attention-based approach to process memory features where we first learn the spatiotemporal relation among the resultant visual and displacement memory tokens using self-attention module. Further, the output features of self-attention are aggregated with the current visual features through cross-attention. The cross-attended features are finally given to a decoder to predict depth on the current frame. Through extensive experiments on several benchmarks, including KITTI, NYU-Depth V2, and DDAD, we show that MAMo consistently improves monocular depth estimation networks and sets new state-of-the-art (SOTA) accuracy. Notably, our MAMo video depth estimation provides higher accuracy with lower latency, when comparing to SOTA cost-volume-based video depth models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv51070.2023.00804

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MAMo augments networks with memory to store learned visual and displacement tokens from previous frames, allowing for more accurate depth predictions through cross-referencing past features.

    “MAMo (Yasarla et al., 2023) augments networks with memory to store learned visual and displacement tokens from previous frames, allowing for more accurate depth predictions through cross-referencing past features.”

    From The AI Hippocampus: How Far are We From Human Memory? (2026) · cites this paper as Yasarla et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 65

  1. Ashutosh Agarwal and Chetan Arora. Depthformer: Multi- scale vision transformer for monocular depth estimation with global local information fusion. In Proceedings of the IEEE International Conference on Image Processing (ICIP), pages 3873-3877, 2022. 6, 10
  2. Ashutosh Agarwal and Chetan Arora. Attention attention ev- erywhere: Monocular depth prediction with skip attention. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision (WACV), pages 5861-5870, January 2023. 2, 5, 6, 7, 9, 10
  3. Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4009-4018, 2021. 1, 2, 6, 10
  4. Shubhankar Borse, Debasmit Das, Hyojin Park, Hong Cai, Risheek Garrepalli, and Fatih Porikli. Dejavu: Conditional regenerative learning to enhance dense prediction. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19466-19477, June 2023.
  5. Shubhankar Borse, Hyojin Park, Hong Cai, Debasmit Das, Risheek Garrepalli, and Fatih Porikli. Panoptic, instance and semantic relations: A relational context encoder to enhance panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1269-1279, June 2022. 3
  6. Hong Cai, Janarbek Matai, Shubhankar Borse, Yizhe Zhang, Amin Ansari, and Fatih Porikli. X-distill: Improving self- supervised monocular depth via cross-task distillation. In British Machine Vision Conference, 2021. 2
  7. Jiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Memot: multi-object tracking with memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8090-8100, 2022. 3
  8. Yuanzhouhan Cao, Yidong Li, Haokui Zhang, Chao Ren, and Yifan Liu. Learning structure affinity for video depth estima- tion. In Proceedings of the 29th ACM International Confer- ence on Multimedia, pages 190-198, 2021. 2, 3, 6, 10
  9. Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. Attend to you: Personalized image captioning with con- text sequence memory networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 895-903, 2017. 3
  10. Debasmit Das, Shubhankar Borse, Hyojin Park, Kambiz Azarian, Hong Cai, Risheek Garrepalli, and Fatih Porikli. Transadapt: A transformative framework for online test time adaptive semantic segmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023. 3
  11. David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep net- work. Advances in neural information processing systems, 27, 2014. 1, 4, 5, 6
  12. Chanho Eom, Hyunjong Park, and Bumsub Ham. Tempo- rally consistent depth prediction with flow-guided memory units. IEEE Transactions on Intelligent Transportation Sys- tems, 21(11):4626-4636, 2019. 2, 3, 6, 10
  13. Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski. Towards internet-scale multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1434-1441, 2010. 1
  14. Risheek Garrepalli. Oracle analysis of representations for deep open set detection. arXiv preprint arXiv:2209.11350, 2022. 3 · arXiv
  15. Risheek Garrepalli, Jisoo Jeong, Rajeswaran C. Ravindran, Jamie Menjay Lin, and Fatih Porikli. Dift: Dynamic iterative field transforms for memory efficient optical flow. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2219- 2228, June 2023. 3
  16. Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 3354- 3361, 2012. 2, 5
  17. Clément Godard, Oisin Mac Aodha, and Gabriel J. Bros- tow. Unsupervised monocular depth estimation with left- right consistency. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2017. 2
  18. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. 3 · arXiv
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 37(9):1904-1916, 2015. 9
  20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. 9
  21. Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In Computer Vision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 668-685. Springer, 2022. 3, 9
  22. Jisoo Jeong, Hong Cai, Risheek Garrepalli, and Fatih Porikli. Distractflow: Improving optical flow estimation via real- istic distractions and pseudo-labeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13691-13700, June 2023. 3
  23. Jisoo Jeong, Jamie Menjay Lin, Fatih Porikli, and Nojun Kwak. Imposing consistency for optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 3181-3191, 2022. 3
  24. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In Pro- ceedings of the International Conference on Machine Learn- ing (ICML), pages 1378-1387, 2016. 3
  25. Hyunmin Lee and Jaesik Park. Stad: Stable video depth es- timation. In Proceedings of the IEEE International Confer- ence on Image Processing (ICIP), pages 3213-3217. IEEE, 2021. 6, 10
  26. Siyuan Li, Yue Luo, Ye Zhu, Xun Zhao, Yu Li, and Ying Shan. Enforcing temporal consistency in video depth esti- mation. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 1145-1154, 2021. 9
  27. Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. arXiv preprint arXiv:2204.00987, 2022. 2, 6, 10 · arXiv
  28. Chao Liu, Jinwei Gu, Kihwan Kim, Srinivasa G Narasimhan, and Jan Kautz. Neural rgb (r) d sensing: Depth and uncer- tainty from a video camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10986-10995, 2019. 6, 10
  29. Si Liu, Risheek Garrepalli, Thomas Dietterich, Alan Fern, and Dan Hendrycks. Open category detection with PAC guarantees. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3169-3178. PMLR, 10-15 Jul 2018. 3
  30. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021. 9
  31. Xiaoxiao Long, Lingjie Liu, Wei Li, Christian Theobalt, and Wenping Wang. Multi-view depth estimation using epipolar spatio-temporal networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8258-8267, June 2021. 2
  32. Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040-4048, 2016. 6, 10
  33. Jiaxu Miao, Yunchao Wei, and Yi Yang. Memory aggrega- tion networks for efficient interactive video object segmenta- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 10366- 10375, 2020.
  34. Jeff Michels, Ashutosh Saxena, and Andrew Y Ng. High speed obstacle avoidance using monocular vision and rein- forcement learning. In Proceedings of the International Con- ference on Machine Learning (ICML), pages 593-600, 2005. 2
  35. Takaaki Nagai, Takumi Naruse, Masaaki Ikehara, and Akira Kurematsu. Hmm-based surface reconstruction from single images. In Proceedings of the IEEE International Confer- ence on Image Processing (ICIP), volume 2, pages II-II. IEEE, 2002. 2
  36. Vaishakh Patil, Wouter Van Gansbeke, Dengxin Dai, and Luc Van Gool. Don't forget the past: Recurrent depth estimation from monocular video. IEEE Robotics and Automation Let- ters, 5(4):6813-6820, 2020. 2, 3, 6, 10
  37. Giovanni Pintore, Marco Agus, Eva Almansa, Jens Schnei- der, and Enrico Gobbetti. Slicenet: deep dense depth esti- mation from a single indoor panorama using a slice-based representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11536- 11545, 2021. 9
  38. Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal. Make-a-story: Vi- sual memory conditioned consistent story generation. arXiv preprint arXiv:2211.13319, 2022. 3 · arXiv
  39. Lukas Ruff, Jacob R Kauffmann, Robert A Vander- meulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Pro- ceedings of the IEEE, 109(5):756-795, 2021. 3
  40. Patrick Ruhkamp, Daoyi Gao, Hanzhi Chen, Nassir Navab, and Beniamin Busam. Attention meets geometry: Geom- etry guided spatial-temporal attention for consistent self- supervised monocular depth estimation. In Proceedings of the International Conference on 3D Vision (3DV), pages 837-847, 2021. 6, 10
  41. Ashutosh Saxena, Sung Chung, and Andrew Ng. Learning depth from single monocular images. Advances in Neural Information Processing Dystems, 18, 2005. 2
  42. Ashutosh Saxena, Jamie Schulte, Andrew Y Ng, et al. Depth estimation using monocular and stereo cues. In Proceedings of the International Joint Conferences on Artificial Intelli- gence (IJCAI), volume 7, pages 2197-2203, 2007. 1
  43. Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5):824-840, 2008. 2
  44. Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard. Simplere- con: 3d reconstruction without 3d convolutions. In Pro- ceedings of the European Conference on Computer Vision (ECCV), 2022. 2
  45. Shuwei Shao, Zhongcai Pei, Weihai Chen, Ran Li, Zhong Liu, and Zhengguo Li. Urcdc-depth: Uncertainty rectified cross-distillation with cutflip for monocular depth estima- tione. https://arxiv.org/abs/2302.08149, 2023. 10 · arXiv
  46. Yunxiao Shi, Hong Cai, Amin Ansari, and Fatih Porikli. Ega- depth: Efficient guided attention for self-supervised multi- camera depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 119-129, 2023. 2
  47. Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of the European Conference on Computer Vision (ECCV), volume 7576, pages 746-760, 2012. 2, 6
  48. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End- to-end memory networks. Advances in Neural Information Processing Systems, 28, 2015. 3
  49. Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8934- 8943, 2018. 3
  50. Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 6105-6114. PMLR, 2019. 6, 10
  51. Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402-419. Springer, 2020. 3, 5, 7
  52. Vibashan VS, Poojan Oza, and Vishal M Patel. Towards on- line domain adaptive object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 478-488, 2023. 3
  53. Vibashan VS, Ning Yu, Chen Xing, Can Qin, Mingfei Gao, Juan Carlos Niebles, Vishal M Patel, and Ran Xu. Mask-free ovis: Open-vocabulary instance segmentation without man- ual mask annotations. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 23539-23549, 2023. 3
  54. Yiran Wang, Zhiyu Pan, Xingyi Li, Zhiguo Cao, Ke Xian, and Jianming Zhang. Less is more: Consistent video depth estimation with masked frames modeling. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6347-6358, 2022. 2, 3, 6, 10
  55. Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman. The temporal opportunist: Self-supervised multi-frame monocular depth. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1164-1174, 2021. 2, 5, 6, 10
  56. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014. 3 · arXiv
  57. Chien-Sheng Wu, Richard Socher, and Caiming Xiong. Global-to-local memory pointer networks for task-oriented dialogue. arXiv preprint arXiv:1901.04713, 2019. 3 · arXiv
  58. Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers. Advances in Neural Information Processing Systems, 34:12077-12090, 2021. 6, 10
  59. Jiaxin Xie, Chenyang Lei, Zhuwen Li, Li Erran Li, and Qifeng Chen. Video depth estimation by fusing flow-to- depth proposals. In Proceedings of the IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, pages 10100-10107, 2020. 2, 3, 6, 10
  60. Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked im- age modeling. pages 14475-14485, 2023. 10
  61. Seungjoo Yoo, Hyojin Bahng, Sunghyo Chung, Junsoo Lee, Jaehyuk Chang, and Jaegul Choo. Coloring with limited data: Few-shot colorization via memory augmented net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11283-11292, 2019. 3
  62. Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Newcrfs: Neural window fully-connected crfs for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 2, 5, 6, 7, 9, 10
  63. Haokui Zhang, Chunhua Shen, Ying Li, Yuanzhouhan Cao, Yu Liu, and Youliang Yan. Exploiting temporal consis- tency for real-time video depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1725-1734, 2019. 2, 3, 6, 10
  64. Jing Zhu, Yunxiao Shi, Mengwei Ren, and Yi Fang. Mda- net: memorable domain adaptation network for monocu- lar depth estimation. In British Machine Vision Conference 2020, 2020. 2
  65. Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text- to-image synthesis. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 5802-5810, 2019. 3

Cited by

Sorted by citations

Authors
Yasarla, R., Cai, H., Jeong, J., Shi, Y., Garrepalli, R., & Porikli, F.
Affiliations
Qualcomm (United States) (US)
ORCID
Venue
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2023-10-01
Pages
8720-8730
DOI
10.1109/iccv51070.2023.00804
OpenAlex
W4390873091
Cited by
15
References
5
Citations by year
2026: 1 · 2025: 6 · 2024: 8
FWCI
1.61
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
Keywords
memory-augmented networks, attention mechanism, self-attention, cross-attention, temporal information, monocular depth estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review