EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Multi-modal large language models (MLLMs) have demonstrated remarkable success in vision and visual-language tasks within the natural image domain. Owing to the significant domain gap between natural and remote sensing (RS) images, the development of MLLMs in the RS domain is still in the infant stage. To fill the gap, a pioneer MLLM named EarthGPT integrating various multi-sensor RS interpretation tasks uniformly is proposed in this paper for universal RS image comprehension. Firstly, a visual-enhanced perception mechanism is constructed to refine and incorporate coarse-scale semantic perception information and fine-scale detailed perception information. Secondly, a cross-modal mutual comprehension approach is proposed, aiming at enhancing the interplay between visual perception and language comprehension and deepening the comprehension of both visual and language content. Finally, a unified instruction tuning method for multi-sensor multi-task in the RS domain is proposed to unify a wide range of tasks including scene classification, image captioning, region-level captioning, visual question answering (VQA), visual grounding, object detection, etc. More importantly, a dataset named MMRS-1M featuring large-scale multi-sensor multi-modal RS instruction-following is constructed, comprising over 1M image-text pairs based on 34 existing diverse RS datasets and including multi-sensor images such as optical, synthetic aperture radar (SAR), and infrared. The MMRS-1M dataset addresses the drawback of MLLMs on RS expert knowledge and stimulates the development of MLLMs in the RS domain. Extensive experiments are conducted, demonstrating the EarthGPT’s superior performance in various RS visual interpretation tasks compared with the other specialist models and MLLMs, proving the effectiveness of the proposed EarthGPT and offering a versatile paradigm for open-set reasoning tasks. Our code and dataset are available at https://github.com/wivizhang/EarthGPT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“The Word2Vec [79] technique is employed to map each subword to a 1-”
MethodA Comprehensive Survey on Transfer Learning
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3004555 · Cited by 6,678 · cited 1×
“During the two-step training, the domain transfer learning strategy [3], [14] is adopted, allowing the proposed model to leverage generalized knowledge learned from the broader natural datasets during the cross-modal mutual comprehension phase, and then continue fine-tuning to transfer the general…”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources
Zhu, X. X., Tuia, D., Mou, L., Xia, G.-S., Zhang, L., Xu, F., & Fraundorfer, F. (2017) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2017.2762307 · Cited by 3,400
Remote Sensing Image Scene Classification: Benchmark and State of the Art
Cheng, G., Han, J., & Lu, X. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2675998 · Cited by 2,926 · cited 2×
“It is comprised of 45 scene categories, with each category containing 700 RS images.”
BackgroundMethodLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Deep learning in remote sensing applications: A meta-analysis and review
Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., & Johnson, B. A. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2019.04.015 · Cited by 2,456
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72970-6_3 · Cited by 2,074
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
Helber, P., Bischke, B., Dengel, A., & Borth, D. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2918242 · Cited by 1,951 · cited 2×
“For classification, commonly used datasets include AID [56], EuroSAT [57], NWPU-RESISC45 [58], UCMerced LandUse [59], and WHU-RS19 [60].”
BackgroundMethodMicrosoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · cited 2×
“Specifically, the self-attention and normalization layers are unfrozen for training on the most common domain data (LAION-400M [80] and COCO Caption [81]).”
BackgroundMethodFlamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Detection and tracking meet drones challenge
Zhu, P., Wen, L., Du, D., Bian, X., Fan, H., Hu, Q., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3119563 · Cited by 1,336 · cited 1×
“For other tasks such as VQA, RSVG, and HBB detection, the two-step training strategy brings EarthGPT an increase of 3.82% (OA), 4.06% (mIoU), and 3.74% (AP@50) on RSVQA-HR, DIOR-RSVG, and MAR20 datasets, respectively.”
BackgroundInfluentialDINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · cited 1×
“In the visual-enhanced perception part, the DINOv2 ViT-L/14 [94] is adopted as the ViT encoder, and the frozen CLIP ConvNeXt-L [95] is utilized as the CNN encoder.”
MethodLi, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · cited 4×
“Models like VisualGPT [4], BLIP [5], Flamingo [7], and Kosmos-1 [26] showcase strong multimodal reasoning potential after aligning LLMs with image modalities.”
BackgroundMethodInfluentialLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682
Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges
Ding, J., Xue, N., Xia, G.-S., Bai, X., Yang, W., Yang, M. Y., Belongie, S., Luo, J., Datcu, M., Pelillo, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117983 · Cited by 678
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing
Liu, F., Chen, D., Guan, Z., Zhou, X., Zhu, J., Ye, Q., Fu, L., & Zhou, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3390838 · Cited by 507 · cited 2×
“Note that four pivotal techniques are developed.”
BackgroundOrientation robust object detection in aerial images using deep convolutional neural network
Zhu, H., Chen, X., Dai, W., Fu, K., Ye, Q., & Jiao, J. (2015) · 2015 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2015.7351502 · Cited by 483 · cited 1×
“For other tasks such as VQA, RSVG, and HBB detection, the two-step training strategy brings EarthGPT an increase of 3.82% (OA), 4.06% (mIoU), and 3.74% (AP@50) on RSVQA-HR, DIOR-RSVG, and MAR20 datasets, respectively.”
BackgroundInfluentialMiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · cited 3×
“To achieve modality alignment, many works, such as MiniGPT-4 [9], LLAMA-Adapter V2 [27], mPLUG-Owl [28], employed projection layers, zero-shot attention mechanism, and intermediate networks to fuse LLaMA and visual modality.”
BackgroundMethodLAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358
Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
Manas, O., Lacoste, A., Giro-i-Nieto, X., Vazquez, D., & Rodriguez, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00928 · Cited by 354
Deep semantic understanding of high resolution remote sensing image
Qu, B., Li, X., Tao, D., & Lu, X. (2016) · 2016 International Conference on Computer, Information and Telecommunication Systems (CITS) · doi:10.1109/cits.2016.7546397 · Cited by 350
GeoChat:Grounded Large Vision-Language Model for Remote Sensing
Kuckreja, K., Danish, M. S., Naseer, M., Das, A., Khan, S., & Khan, F. S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02629 · Cited by 290 · cited 2×
“In addition, GeoChat [11] has been proposed to explore RS MLLMs and to solve multiple tasks on optical RS images.”
BackgroundMethodLanguage Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235 · cited 1×
“Models like VisualGPT [4], BLIP [5], Flamingo [7], and Kosmos-1 [26] showcase strong multimodal reasoning potential after aligning LLMs with image modalities.”
MethodRSGPT: A Remote Sensing Vision Language Model and Benchmark
Hu, Y., Yuan, J., Wen, C., Lu, X., & Li, X. (2023) · arXiv · doi:10.48550/arxiv.2307.15266 · Cited by 219 · cited 2×
“For instance, RSGPT [10] fine-tunes Instruct-BLIP [36] on a high-quality image–text pair dataset, showing good image–text caption and VQA capabilities.”
BackgroundQwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2308.12966 · Cited by 193
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183
RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
Zhan, Y., Xiong, Z., & Yuan, Y. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3250471 · Cited by 179 · cited 2×
“In addition, the DIOR-RSVG [17] is collected for visual grounding and region-level caption.”
MethodMutual Attention Inception Network for Remote Sensing Visual Question Answering
Zheng, X., Wang, B., Du, X., & Lu, X. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3079918 · Cited by 159 · cited 2×
“The ViT model splits the input image into patch sequences, allowing models to learn image structures independently.”
MethodResultInfluentialMMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., Liu, Z., Wang, S., Han, W., & Chang, B. (2023) · arXiv · doi:10.48550/arxiv.2309.07915 · Cited by 120 · cited 1×
“…[5], [6], [7], has shown great success in the natural image domain, as they not only possess robust language interaction capabilities but also exhibit impressive multitask reasoning skills in real-world scenarios [8], compared to smaller and domain-specific models tailored for particular tasks.”
BackgroundMacaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2306.09093 · Cited by 38
Research works printed but not linked: 72
- 10.3390/rs14225675 · DOI
- 10.1109/CVPR52688.2022.01750 · DOI · Paper page
- Dosovitskiy, 2020, An image is worth 16×16 words: Transformers for image recognition at scale, arXiv:2010.11929 · arXiv
- 10.48550/ARXIV.1511.08458 · arXiv · doi:10.48550/arxiv.1511.08458 · Paper page
- 10.1109/TGRS.2022.3201474 · DOI
- 10.1109/TGRS.2023.3312479 · DOI
- 10.3390/s20041226 · DOI
- 10.1109/JSTARS.2021.3063096 · DOI
- 10.11834/jrs.20222139 · DOI · Paper page
- Achiam, 2023, GPT-4 technical report
- Zhang, 2023, LLaMA-adapter: Efficient fine-tuning of language models with zero-init attention, arXiv:2303.16199 · arXiv
- Gao, 2023, LLaMA-adapter v2: Parameter-efficient visual instruction model, arXiv:2304.15010 · arXiv
- Ye, 2023, MPLUG-owl: Modularization empowers large language models with multimodality, arXiv:2304.14178 · arXiv
- Tang, 2023, CoDi-2: In-context, interleaved, and interactive any-to-any generation, arXiv:2311.18775 · arXiv
- Xu, 2023, PointLLM: Empowering large language models to understand point clouds, arXiv:2308.16911 · arXiv
- Guo, 2023, Point-bind & point-LLM: Aligning point cloud with multi-modality for 3D understanding, generation, and instruction following, arXiv:2309.00615 · arXiv
- Hong, 2023, 3D-LLM: Injecting the 3D world into large language models, arXiv:2307.12981 · arXiv
- Yang, 2023, LiDAR-LLM: Exploring the potential of large language models for 3D LiDAR understanding, arXiv:2312.14074 · arXiv
- Ahn, 2024, AutoRT: Embodied foundation models for large scale orchestration of robotic agents, arXiv:2401.12963 · arXiv
- Dai, 2023, InstructBLIP: Towards general-purpose vision-language models with instruction tuning, arXiv:2305.06500 · arXiv
- 10.1109/JSTARS.2021.3117857 · DOI
- 10.1109/TGRS.2023.3295797 · DOI
- 10.1109/TGRS.2022.3190934 · DOI
- 10.1109/TGRS.2021.3081421 · DOI
- 10.1109/LGRS.2019.2893772 · DOI
- 10.3390/rs11060612 · DOI · Paper page
- 10.1109/CVPR.2019.00331 · DOI · Paper page
- 10.1109/TGRS.2020.2988782 · DOI · Paper page
- 10.1109/CVPR.2016.538 · DOI · Paper page
- 10.1109/ICCV.2019.00479 · DOI
- 10.1109/ICCV.2019.00478 · DOI · Paper page
- 10.1109/CVPR46437.2021.01661 · DOI
- 10.1109/ICCV48922.2021.00179 · DOI · Paper page
- 10.1109/TGRS.2021.3062048 · DOI · Paper page
- 10.1109/CVPR46437.2021.00868 · DOI · Paper page
- 10.1109/ICCV48922.2021.00350 · DOI · Paper page
- 10.1609/aaai.v35i4.16426 · DOI · Paper page
- 10.1609/aaai.v36i1.19975 · DOI · Paper page
- 10.1109/TGRS.2017.2685945 · DOI · Paper page
- 10.1145/1869790.1869829 · DOI · Paper page
- 10.1109/LGRS.2010.2055033 · DOI · Paper page
- 10.1109/TGRS.2014.2374218 · DOI · Paper page
- 10.1016/j.isprsjprs.2021.12.004 · DOI · Paper page
- 10.1109/TGRS.2019.2900302 · DOI · Paper page
- 10.1109/TGRS.2017.2778300 · DOI · Paper page
- 10.12000/JR19097 · DOI
- 10.1109/ACCESS.2020.3005861 · DOI · Paper page
- 10.3390/rs13183690 · DOI · Paper page
- Zamir, iSAID: A large-scale dataset for instance segmentation in aerial images, 28
- Wang, LoveDA: A remote sensing land-cover dataset for domain adaptive semantic segmentation, 1
- 10.1109/TGRS.2017.2776321 · DOI · Paper page
- 10.1109/TGRS.2021.3078451 · DOI · Paper page
- 10.1109/ACCESS.2021.3090981 · DOI · Paper page
- Chen, 2023, MiniGPT-v2: Large language model as a unified interface for vision-language multi-task learning, arXiv:2310.09478 · arXiv
- Lin, 2023, SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, arXiv:2311.07575 · arXiv
- 10.18653/v1/D18-2012 · DOI · Paper page
- 10.1109/LGRS.2015.2475299 · DOI · Paper page
- 10.3390/rs13040747 · DOI
- 10.1117/12.2532741 · DOI
- 10.1109/TGRS.2016.2645610 · DOI · Paper page
- 10.1038/s41597-023-02066-6 · DOI · Paper page
- 2021, Sea-Shipping
- 2021, Infrared-Security
- 2021, Aerial-Mancar
- 2021, Double-Light-Vehicle
- 2020, Oceanic-Ship
- Ilharco, 2021, OpenCLIP, Zenodo
- 10.1007/978-3-030-58568-6_23 · DOI · Paper page
- 10.1109/CVPR52688.2022.00928 · DOI · Paper page
- Zhao, 2024, An open and comprehensive pipeline for unified object grounding and detection, arXiv:2401.02361 · arXiv
- Wei, 2023, Lenna: Language enhanced reasoning detection assistant, arXiv:2312.02433 · arXiv
- 10.1109/CVPR52688.2022.00187 · DOI · Paper page
Cited by
RSGPT: A Remote Sensing Vision Language Model and Benchmark
Hu, Y., Yuan, J., Wen, C., Lu, X., & Li, X. (2023) · arXiv · doi:10.48550/arxiv.2307.15266 · Cited by 219
VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
Li, X., Ding, J., & Elhoseiny, M. (2024) · arXiv · doi:10.48550/arxiv.2406.12384 · Cited by 47
“Recent works explore integrating multiple datasets to accommodate a wider array of tasks [28, 29].”
Glob-Diffusion: A Global Consistent Diffusion Model for Large-Scale Image Generation
Kang, Y., Shi, H., Liu, H., Xie, W., Fang, L., & Bruzzone, L. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3643743 · Cited by 1
Metadata
- Authors
- Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X.
- Venue
- IEEE Transactions on Geoscience and Remote Sensing
- Published
- 2024
- Volume
- 62
- Pages
- 1-20
- ISSN
- 0196-2892 (ISSN-L)
- 1558-0644
- OpenAlex
- W4399399400
- Cited by
- 185
- References
- 33
- FWCI
- 26.13
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Natural Science Foundation of China (92152109)
- General Program of National Natural Science Foundation of China (62371048)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.