The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
SpectralGPT: Spectral Remote Sensing Foundation Model
Open accessAbstract
The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 3×
“SpectralGPT is constructed following the MAE architecture [25] and incorporates spectral-wise tensor (3D) masking, where 90% of the tokens are masked out.”
BackgroundMethodInfluentialSelf-citationDeep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673 · cited 1×
“…(MS/HS) remote sensing (RS) data the preferred tool of choice and a key component in a wide range of Earth Observation (EO) applications [4], [5], [6], [7], including land use/land cover mapping, ecosystem monitoring, weather forecasting, energy resource development, biodiversity…”
Background(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationUnified Perceptual Parsing for Scene Understanding
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_26 · Cited by 2,201 · cited 1×
“On the SegMunich dataset, we employ the UperNet framework [41] in conjunction with the pretrained foundation models, initially consolidating the four tokens per pixel from the encoder’s final layer into a single token.”
MethodImproved Baselines with Momentum Contrastive Learning
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 1,956
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
Helber, P., Bischke, B., Dengel, A., & Borth, D. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2918242 · Cited by 1,951 · cited 1×
“For the downstream single-label RS scene classification task, we employ the EuroSAT dataset [39].”
MethodImaging Spectrometry for Earth Remote Sensing
Goetz, A. F. H., Vane, G., Solomon, J. E., & Rock, B. N. (1985) · Science · doi:10.1126/science.228.4704.1147 · Cited by 1,928
Hong, D., Zhang, B., Li, H., Li, Y., Yao, J., Li, C., Werner, M., Chanussot, J., Zipf, A., & Zhu, X. X. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113856 · Cited by 545
Self-citationBigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding
Sumbul, G., Charfuelan, M., Demir, B., & Markl, V. (2019) · IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium · doi:10.1109/igarss.2019.8900532 · Cited by 534
Urban Change Detection for Multispectral Earth Observation Using Convolutional Neural Networks
Daudt, R. C., Le Saux, B., Boulch, A., & Gousseau, Y. (2018) · IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium · doi:10.1109/igarss.2018.8518015 · Cited by 466
Hyperspectral Image Classification Using Group-Aware Hierarchical Transformer
Mei, S., Song, C., Ma, M., & Xu, F. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3207933 · Cited by 384
Xie, Z., Lin, Y., Zhang, Z., Cao, Y., Lin, S., & Hu, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01641 · Cited by 370
Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
Manas, O., Lacoste, A., Giro-i-Nieto, X., Vazquez, D., & Rodriguez, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00928 · Cited by 354
RingMo: A Remote Sensing Foundation Model With Masked Image Modeling
Sun, X., Wang, P., Lu, W., Zhu, Z., Lu, X., He, Q., Li, J., Rong, X., Yang, Z., Chang, H., et al. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3194732 · Cited by 323 · cited 1×
“Unlike MAE-based methods that rely on only a few visible image patches to infer the entire image, Sun et al. [27] considered all image patches, whether masked or unmasked, by implementing a MIM strategy in their RS pretraining models.”
MethodChristie, G., Fendley, N., Wilson, J., & Mukherjee, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00646 · Cited by 312
SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery
Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., & Ermon, S. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0015 · Cited by 303
Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model
Wang, D., Zhang, Q., Xu, Y., Zhang, J., Du, B., Tao, D., & Zhang, L. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3222818 · Cited by 287
Non-local Meets Global: An Integrated Paradigm for Hyperspectral Image Restoration
He, W., Yao, Q., Li, C., Yokoya, N., Zhao, Q., Zhang, H., & Zhang, L. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3027563 · Cited by 199
Self-citationHong, D., Yao, J., Li, C., Meng, D., Yokoya, N., & Chanussot, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3324497 · Cited by 196
Self-citationRecent Advances for Quantum Neural Networks in Generative Learning
Tian, J., Sun, X., Du, Y., Zhao, S., Liu, Q., Zhang, K., Yi, W., Huang, W., Wang, C., Wu, X., et al. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3272029 · Cited by 189
Mei, S., Jiang, R., Ma, M., & Song, C. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3233726 · Cited by 103
Modality Translation in Remote Sensing Time Series
Liu, X., Hong, D., Chanussot, J., Zhao, B., & Ghamisi, P. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3079294 · Cited by 46
Self-citationDu, D., Gu, Y., Liu, T., & Li, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3285893 · Cited by 34
Self-citationRethinking Pan-Sharpening in Closed-Loop Regularization
Zhou, M., Huang, J., Hong, D., Zhao, F., Li, C., & Chanussot, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3279931 · Cited by 26
Self-citation
Cited by
Artificial intelligence for geoscience: Progress, challenges, and perspectives
Zhao, T., Wang, S., Ouyang, C., Chen, M., Liu, C., Zhang, J., Yu, L., Wang, F., Xie, Y., Li, J., et al. (2024) · The Innovation · doi:10.1016/j.xinn.2024.100691 · Cited by 360
Self-citationGuo, X., Lao, J., Dang, B., Zhang, Y., Yu, L., Ru, L., Zhong, L., Huang, Z., Wu, K., Hu, D., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02613 · Cited by 290
Self-citationNeural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
Xiong, Z., Wang, Y., Zhang, F., Stewart, A. J., Hanna, J., Borth, D., Papoutsis, I., Saux, B. L., Camps-Valls, G., & Zhu, X. X. (2024) · arXiv · doi:10.48550/arxiv.2403.15356 · Cited by 26
Guo, J., Zimmer-Dauphinee, J. R., Nieusma, J. M., Lu, S., Liu, Q., Deng, R., Cui, C., Yue, J., Lin, Y., Yao, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.20303 · Cited by 7
“FMs leverage large-scale unlabeled data through task-agnostic pretraining, which has been shown in [27] and [43] to alleviate data imbalance and reduce overfitting in downstreamtasks.”
Metadata
- Authors
- Hong, D., Zhang, B., Li, X., Li, Y., Li, C., Yao, J., Yokoya, N., Li, H., Ghamisi, P., Jia, X., Plaza, A., Gamba, P., Benediktsson, J. A., & Chanussot, J.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-08
- Volume
- 46
- Issue
- 8
- Pages
- 5227-5244
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4393906060
- Cited by
- 930
- References
- 33
- FWCI
- 155.72
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.