The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
GMFlow: Learning Optical Flow via Global Matching
Open accessAbstract
Learning-based optical flow estimation has been dominated with the pipeline of cost volume with convolutions for flow regression, which is inherently limited to local correlations and thus is hard to address the long-standing challenge of large displacements. To alleviate this, the state-of-the-art framework RAFT gradually improves its prediction quality by using a large number of iterative refinements, achieving remarkable performance but introducing linearly increasing inference time. To enable both high accuracy and efficiency, we completely revamp the dominant flow regression pipeline by reformulating optical flow as a global matching problem, which identifies the correspondences by directly comparing feature similarities. Specifically, we propose a GMFlow framework, which consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. We further introduce a refinement step that reuses GMFlow at higher feature resolution for residual flow prediction. Our new framework outperforms 31-refinements RAFT on the challenging Sintel benchmark, while using only one refinement and running faster, suggesting a new paradigm for accurate and efficient optical flow estimation. Code is available at https://github.com/haofeixu/gmflow.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · cited 2×
“Following previous methods [16, 36, 39], we first train on the FlyingChairs (Chairs) [10] and FlyingThings3D (Things) [27] datasets, and then evaluate Sintel [4] and KITTI [29] training sets.”
MethodFlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
SuperGlue: Learning Feature Matching With Graph Neural Networks
Sarlin, P.-E., DeTone, D., Malisiewicz, T., & Rabinovich, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00499 · Cited by 2,830
(2020) · doi:10.1007/978-3-030-58536-5_24 · Cited by 2,824
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00931 · Cited by 2,648
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570
Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.438 · Cited by 2,488
Object scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270
A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation
Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., & Sorkine-Hornung, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.85 · Cited by 1,989
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
LoFTR: Detector-Free Local Feature Matching with Transformers
Sun, J., Shen, Z., Wang, Y., Bao, H., & Zhou, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00881 · Cited by 1,719
End-to-End Learning of Geometry and Context for Deep Stereo Regression
Kendall, A., Martirosyan, H., Dasgupta, S., Henry, P., Kennedy, R., Bachrach, A., & Bry, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.17 · Cited by 1,470
Optical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
Brox, T., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.143 · Cited by 1,295
Mayer, N., Ilg, E., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · Cited by 1,214
Scale-Aware Trident Networks for Object Detection
Li, Y., Chen, Y., Wang, N., & Zhang, Z.-X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00615 · Cited by 1,087
VirtualWorlds as Proxy for Multi-object Tracking Analysis
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.470 · Cited by 1,079
DeepFlow: Large Displacement Optical Flow with Deep Matching
Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.175 · Cited by 997
ConViT: improving vision transformers with soft convolutional inductive biases
d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., & Sagun, L. (2022) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac9830 · Cited by 782
EpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298720 · Cited by 697
Fast Cost-Volume Filtering for Visual Correspondence and Beyond
Hosni, A., Rhemann, C., Bleyer, M., Rother, C., & Gelautz, M. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.156 · Cited by 673
AANet: Adaptive Aggregation Network for Efficient Stereo Matching
Xu, H., & Zhang, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00203 · Cited by 583
Self-citationUnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss
Meister, S., Hur, J., & Roth, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12276 · Cited by 520
Virtual KITTI 2Preprint
Cabon, Y., Murray, N., & Humenberger, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.10773 · Cited by 470
TartanAir: A Dataset to Push the Limits of Visual SLAM
Wang, W., Zhu, D., Wang, X., Hu, Y., Qiu, Y., Wang, C., Hu, Y., Kapoor, A., & Scherer, S. (2020) · 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341801 · Cited by 456
(2021) · International Conference on Learning Representations · Cited by 449
Richter, S. R., Hayder, Z., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.243 · Cited by 419
Learning to Estimate Hidden Motions with Global Motion Aggregation
Jiang, S., Campbell, D., Lu, Y., Li, H., & Hartley, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00963 · Cited by 365
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers
Li, Z., Liu, X., Drenkow, N., Ding, A., Creighton, F. X., Taylor, R. H., & Unberath, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00614 · Cited by 335
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation
Hur, J., & Roth, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00590 · Cited by 288
COTR: Correspondence Transformer for Matching Across Images
Jiang, W., Trulls, E., Hosang, J., Tagliasacchi, A., & Yi, K. M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00615 · Cited by 282
Large displacement optical flow
Brox, T., Bregler, C., & Malik, J. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206697 · Cited by 277
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond
Zhang, Q., Xu, Y., Zhang, J., & Tao, D. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-022-01739-w · Cited by 277
Self-citationHierarchical Discrete Distribution Decomposition for Match Density Estimation
Yin, Z., Darrell, T., & Yu, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00620 · Cited by 252
Bailer, C., Taetz, B., & Stricker, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.457 · Cited by 233
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive BiasPreprint
Xu, Y., Zhang, Q., Zhang, J., & Tao, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03348 · Cited by 228
Self-citationGLU-Net: Global-Local Universal Network for Dense Flow and Correspondences
Truong, P., Danelljan, M., & Timofte, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00629 · Cited by 206
Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894353 · Cited by 195
Efficient Coarse-to-Fine Patch Match for Large Displacement Optical Flow
Hu, Y., Song, R., & Li, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.615 · Cited by 188
Volumetric Correspondence Networks for Optical Flow
Yang, G., & Ramanan, D. (2019) · neural information processing systems · Cited by 179
Learning Feature Descriptors Using Camera Pose Supervision
Wang, Q., Zhou, X., Hariharan, B., & Snavely, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_44 · Cited by 162
Kondermann, D., Nair, R., Honauer, K., Krispin, K., Andrulis, J., Brock, A., Gussefeld, B., Rahimimoghaddam, M., Hofmann, S., Brenner, C., et al. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.10 · Cited by 160
Large Displacement Optical Flow from Nearest Neighbor Fields
Chen, Z., Jin, H., Lin, Z., Cohen, S. D., & Wu, Y. N. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.316 · Cited by 155
Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation
Zhang, F., Woodford, O. J., Prisacariu, V. A., & Torr, P. H. S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01063 · Cited by 113
AutoFlow: Learning a Better Training Set for Optical Flow
Sun, D., Vlasic, D., Herrmann, C., Jampani, V., Krainin, M., Chang, H., Zabih, R., Freeman, W. T., & Liu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00996 · Cited by 107
High-Resolution Optical Flow from 1D Attention and Correlation
Xu, H., Yang, J., Cai, J., Zhang, J., & Tong, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01033 · Cited by 91
Self-citationLearning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Lv, Z., Kim, K., Troccoli, A. J., Sun, D., Rehg, J. M., & Kautz, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01228-1_29 · Cited by 86
Cited by
CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow
Weinzaepfel, P., Lucas, T., Leroy, V., Cabon, Y., Arora, V., Brégier, R., Csurka, G., Antsfeld, L., Chidlovskii, B., & Revaud, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01647 · Cited by 110
MemFlow: Optical Flow Estimation and Prediction with Memory
Dong, Q., & Fu, Y. (2024) · arXiv · doi:10.48550/arxiv.2404.04808 · Cited by 49
Metadata
- Authors
- Xu, H., Zhang, J., Cai, J., Rezatofighi, H., & Tao, D.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 8111-8120
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3215023725
- Cited by
- 418
- References
- 55
- FWCI
- 34.48
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.