CoChat
Sign Up
Cited by 288Domain Adaptation and Few-Shot LearningRemote-Sensing Image ClassificationAdvanced Neural Network Applications

Geography-Aware Self-Supervised Learning

Open access
Ayush, Kumar, Uzkent, Burak, Meng, Chenlin, Tanmay, Kumar, Burke, Marshall, Lobell, David, Ermon, Stefano·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv48922.2021.01002
Open this paper in CoChat — free accountOpen PDF

Abstract

Contrastive learning methods have significantly narrowed the gap between supervised and unsupervised learning on computer vision tasks. In this paper, we explore their application to geo-located datasets, e.g. remote sensing, where unlabeled data is often abundant but labeled data is scarce. We first show that due to their different characteristics, a non-trivial gap persists between contrastive and supervised learning on standard benchmarks. To close the gap, we propose novel training methods that exploit the spatio-temporal structure of remote sensing data. We leverage spatially aligned images over time to construct temporal positive pairs in contrastive learning and geo-location to design pre-text tasks. Our experiments show that our proposed method closes the gap between contrastive and supervised learning on image classification, object detection and semantic segmentation for remote sensing. Moreover, we demonstrate that the proposed method can also be applied to geo-tagged ImageNet images, improving downstream performance on various tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.01002

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GASSL and SeCo were trained on a larger pre-train dataset (1M Sentinel-2 images vs. 713k) and with all 13 bands than SatMAE's fMoW Sentinel.

    “GASSL and SeCo were actually trained on a larger pre-train dataset (1M Sentinel-2 images v.s. 713k) and with all 13 bands than our fMoW Sentinel.”

    From SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022) · cites this paper as [34]

    Section 5 (Experiments) says the paper learns representations on fMoW and GeoImageNet, not a million-image Sentinel-2 dataset. Section 3.1 describes fMoW as having approximately 363,571 training images. Our copy also does not substantiate the claim that GASSL used all 13 Sentinel-2 bands; the paper does not address SeCo or SatMAE's dataset comparison. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Existing self-supervised learning methods for satellite imagery cannot learn general representations for both temporal and multi-spectral remote sensing data.

    “While self-supervised learning methods for satellite imagery exist [34, 35], these approaches cannot learn general representations for both temporal and multi-spectral remote sensing data.”

    From SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022) · cites this paper as [34]

    The Introduction reports a performance gap for an existing self-supervised method on satellite imagery, and the Problem Definition explains why the paper seeks to use remote sensing data’s temporal structure. But the paper does not establish that existing methods cannot learn general representations, nor does it address a combined limitation involving temporal and multispectral data. No page numbers are available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The timestamp of a satellite image is useful for many pre-training or downstream vision tasks.

    “As shown in many prior works [34, 35], the timestamp of a satellite image is useful for many pre-training or downstream vision tasks.”

    From SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022) · cites this paper as [34]

  • SatMAE achieves a larger performance gain from supervised learning from scratch compared to GASSL on SpaceNet v1 building segmentation.

    “The results in table 7 show that our method achieves a larger performance gain from supervised learning from scratch compared to [34].”

    From SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022) · cites this paper as [34]

  • Using test-time augmentations similar to [34] is beneficial.

    “In addition, using test-time augmentations similar to [34] is beneficial.”

    From SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022) · cites this paper as [34]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. Kumar Ayush, Burak Uzkent, Marshall Burke, David Lobell, and Stefano Ermon. Generating interpretable poverty maps using object detection in satellite images. arXiv preprint arXiv:2002.01612, 2020. 2 · arXiv
  2. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020. 1, 2, 4 · arXiv
  3. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. 1, 2, 4, 6, 7, 8 · arXiv
  4. Anil M Cheriyadat. Unsupervised feature learning for aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 52(1):439-451, 2013. 2
  5. Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6172-6180, 2018. 1, 2, 4
  6. Grace Chu, Brian Potetz, Weijun Wang, Andrew Howard, Yang Song, Fernando Brucher, Thomas Leung, and Hartwig Adam. Geo-aware networks for fine-grained recognition. In Proceedings of the IEEE International Conference on Com- puter Vision Workshops, pages 0-0, 2019. 3
  7. Terrance de Vries, Ishan Misra, Changhan Wang, and Lau- rens van der Maaten. Does object recognition work for ev- eryone? In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition Workshops, pages 52- 59, 2019. 2, 4
  8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009. 4
  9. Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un- supervised representation learning by predicting image rota- tions. arXiv preprint arXiv:1803.07728, 2018. 2 · arXiv
  10. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020. 1
  11. James Hays and Alexei A Efros. Im2gps: estimating geo- graphic information from a single image. In 2008 ieee con- ference on computer vision and pattern recognition, pages 1-8. IEEE, 2008. 1, 3
  12. James Hays and Alexei A Efros. Large-scale image geolo- calization. In Multimodal location estimation of videos and images, pages 41-62. Springer, 2015. 1, 3
  13. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020. 1, 2, 4, 5, 7, 8
  14. Hayate Iso, Shoko Wakamiya, and Eiji Aramaki. Density estimation for geolocation via convolutional mixture density network. arXiv preprint arXiv:1705.02750, 2017. 3 · arXiv
  15. Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, and Stefano Ermon. Tile2vec: Unsupervised representation learning for spatially distributed data. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3967-3974, 2019. 2
  16. Darius Lam, Richard Kuzma, Kevin McGee, Samuel Doo- ley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord. xview: Objects in context in overhead imagery. arXiv preprint arXiv:1802.07856, 2018. 1, 7 · arXiv
  17. Yansheng Li, Chao Tao, Yihua Tan, Ke Shang, and Jinwen Tian. Unsupervised multilayer feature learning for satellite image scene classification. IEEE Geoscience and Remote Sensing Letters, 13(2):157-161, 2016. 2
  18. Xiaoqiang Lu, Xiangtao Zheng, and Yuan Yuan. Remote sensing scene classification by unsupervised representation learning. IEEE Transactions on Geoscience and Remote Sensing, 55(9):5148-5157, 2017. 2
  19. Oisin Mac Aodha, Elijah Cole, and Pietro Perona. Presence- only geographical priors for fine-grained image classifica- tion. In Proceedings of the IEEE International Conference on Computer Vision, pages 9596-9606, 2019. 3
  20. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Con- ference on Computer Vision (ECCV), pages 181-196, 2018.
  21. Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6707-6717, 2020. 2
  22. Lichao Mou, Pedram Ghamisi, and Xiao Xiang Zhu. Un- supervised spectral-spatial feature learning via deep resid- ual conv-deconv network for hyperspectral image classifica- tion. IEEE Transactions on Geoscience and Remote Sensing, 56(1):391-406, 2017. 2
  23. Eric Muller-Budack, Kader Pustu-Iren, and Ralph Ewerth. Geolocation estimation of photos using a hierarchical model and scene classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 563-579, 2018. 3
  24. Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84. Springer, 2016. 2
  25. Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Dar- rell, and Bharath Hariharan. Learning features by watch- ing objects move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2701- 2710, 2017. 2
  26. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2536-2544, 2016. 2
  27. Adriana Romero, Carlo Gatta, and Gustau Camps-Valls. Un- supervised deep feature extraction for remote sensing image classification. IEEE Transactions on Geoscience and Remote Sensing, 54(3):1349-1362, 2015. 2
  28. Evan Sheehan, Chenlin Meng, Matthew Tan, Burak Uzkent, Neal Jean, Marshall Burke, David Lobell, and Stefano Er- mon. Predicting economic development using geolocated wikipedia articles. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2698-2706, 2019. 1
  29. Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685, 2020. 2 · arXiv
  30. Kevin Tang, Manohar Paluri, Li Fei-Fei, Rob Fergus, and Lubomir Bourdev. Improving image classification with lo- cation context. In Proceedings of the IEEE international conference on computer vision, pages 1008-1016, 2015. 3
  31. Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. arXiv preprint arXiv:1906.05849, 2019. 4 · arXiv
  32. Burak Uzkent, Aneesh Rangnekar, and Matthew J Hoffman. Tracking in aerial hyperspectral videos using deep kernelized correlation filters. IEEE Transactions on Geoscience and Remote Sensing, 57(1):449-461, 2018. 2
  33. Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang, Marshall Burke, David Lobell, and Stefano Ermon. Learning to interpret satellite images in global scale using wikipedia. arXiv preprint arXiv:1905.02506, 2019. 1, 2 · arXiv
  34. Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang, Marshall Burke, David Lobell, and Stefano Ermon. Learning to interpret satellite images using wikipedia. In Proceedings of the Twenty-Eighth International Joint Conference on Arti- ficial Intelligence, 2019. 1
  35. Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang, Marshall Burke, David B Lobell, and Stefano Ermon. Learn- ing to interpret satellite images using wikipedia. In IJCAI, pages 3620-3626, 2019. 1
  36. Burak Uzkent, Christopher Yeh, and Stefano Ermon. Effi- cient object detection in large images using deep reinforce- ment learning. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 1824- 1833, 2020. 2
  37. Adam Van Etten, Dave Lindenbaum, and Todd M Bacastow. Spacenet: A remote sensing dataset and challenge series. arXiv preprint arXiv:1807.01232, 2018. 8 · arXiv
  38. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008. 2
  39. Nam Vo, Nathan Jacobs, and James Hays. Revisiting im2gps in the deep learning era. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pages 2621-2630, 2017. 3
  40. Tobias Weyand, Ilya Kostrikov, and James Philbin. Planet- photo geolocation with convolutional neural networks. In European Conference on Computer Vision, pages 37-55. Springer, 2016. 3
  41. Yi Yang and Shawn Newsam. Bag-of-visual-words and spa- tial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pages 270-279, 2010. 2
  42. Xiwen Yao, Junwei Han, Gong Cheng, Xueming Qian, and Lei Guo. Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Transactions on Geoscience and Remote Sensing, 54(6):3660-3671, 2016.
  43. Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016. 2
  44. Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel pre- diction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058-1067, 2017. 2
  45. Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 267-283, 2018.

Cited by

Sorted by citations

Authors
Ayush, K., Uzkent, B., Meng, C., Tanmay, K., Burke, M., Lobell, D., & Ermon, S.
Affiliations
  • Stanford University (US)
  • Indian Institute of Technology Kharagpur (IN)
ORCID
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
10161-10170
DOI
10.1109/iccv48922.2021.01002
OpenAlex
W3105496885
Cited by
288
References
4
Citations by year
2026: 53 · 2025: 73 · 2024: 60 · 2023: 49 · 2022: 41 · 2021: 12
FWCI
18.15
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
  • arXiv (Cornell University)
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
Funders
  • National Science Foundation (1651565, 1733686)
  • Office of the Director of National Intelligence
  • Intelligence Advanced Research Projects Activity (2021-2011000004)
  • Office of Naval Research (N00014-19, N00014-19-1-2145, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Media Technology › Remote-Sensing Image Classification
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
contrastive learning, self-supervised learning, remote sensing images, spatiotemporal structure, geolocation, image classification, object detection, semantic segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review