EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Multi-modal large language models (MLLMs) have demonstrated remarkable success in vision and visual-language tasks within the natural image domain. Owing to the significant domain gap between natural and remote sensing (RS) images, the development of MLLMs in the RS domain is still in the infant stage. To fill the gap, a pioneer MLLM named EarthGPT integrating various multi-sensor RS interpretation tasks uniformly is proposed in this paper for universal RS image comprehension. Firstly, a visual-enhanced perception mechanism is constructed to refine and incorporate coarse-scale semantic perception information and fine-scale detailed perception information. Secondly, a cross-modal mutual comprehension approach is proposed, aiming at enhancing the interplay between visual perception and language comprehension and deepening the comprehension of both visual and language content. Finally, a unified instruction tuning method for multi-sensor multi-task in the RS domain is proposed to unify a wide range of tasks including scene classification, image captioning, region-level captioning, visual question answering (VQA), visual grounding, object detection, etc. More importantly, a dataset named MMRS-1M featuring large-scale multi-sensor multi-modal RS instruction-following is constructed, comprising over 1M image-text pairs based on 34 existing diverse RS datasets and including multi-sensor images such as optical, synthetic aperture radar (SAR), and infrared. The MMRS-1M dataset addresses the drawback of MLLMs on RS expert knowledge and stimulates the development of MLLMs in the RS domain. Extensive experiments are conducted, demonstrating the EarthGPT’s superior performance in various RS visual interpretation tasks compared with the other specialist models and MLLMs, proving the effectiveness of the proposed EarthGPT and offering a versatile paradigm for open-set reasoning tasks. Our code and dataset are available at https://github.com/wivizhang/EarthGPT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
A Comprehensive Survey on Transfer Learning
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3004555 · Cited by 6,678 · cited 1×
“During the two-step training, the domain transfer learning strategy [3], [14] is adopted, allowing the proposed model to leverage generalized knowledge learned from the broader natural datasets during the cross-modal mutual comprehension phase, and then continue fine-tuning to transfer the general…”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources
Zhu, X. X., Tuia, D., Mou, L., Xia, G.-S., Zhang, L., Xu, F., & Fraundorfer, F. (2017) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2017.2762307 · Cited by 3,400
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Deep learning in remote sensing applications: A meta-analysis and review
Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., & Johnson, B. A. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2019.04.015 · Cited by 2,456
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72970-6_3 · Cited by 2,074
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
Helber, P., Bischke, B., Dengel, A., & Borth, D. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2918242 · Cited by 1,951 · cited 2×
“For classification, commonly used datasets include AID [56], EuroSAT [57], NWPU-RESISC45 [58], UCMerced LandUse [59], and WHU-RS19 [60].”
BackgroundMethod
Research works printed but not linked: 76
- 10.3390/rs14225675 · DOI
- 10.1109/CVPR52688.2022.01750 · DOI · Paper page
- Dosovitskiy, 2020, An image is worth 16×16 words: Transformers for image recognition at scale, arXiv:2010.11929 · arXiv
- 10.48550/ARXIV.1511.08458 · arXiv · doi:10.48550/arxiv.1511.08458 · Paper page
- 10.1109/TGRS.2022.3201474 · DOI
- 10.1109/TGRS.2023.3312479 · DOI
- 10.3390/s20041226 · DOI
- 10.1109/JSTARS.2021.3063096 · DOI
- 10.11834/jrs.20222139 · DOI
- Achiam, 2023, GPT-4 technical report
- Zhang, 2023, LLaMA-adapter: Efficient fine-tuning of language models with zero-init attention, arXiv:2303.16199 · arXiv
- Peng, 2023, Instruction tuning with GPT-4, arXiv:2304.03277 · arXiv
- Gao, 2023, LLaMA-adapter v2: Parameter-efficient visual instruction model, arXiv:2304.15010 · arXiv
- Ye, 2023, MPLUG-owl: Modularization empowers large language models with multimodality, arXiv:2304.14178 · arXiv
- Lyu, 2023, Macaw-LLM: Multi-modal language modeling with image, audio, video, and text integration, arXiv:2306.09093 · arXiv
- Tang, 2023, CoDi-2: In-context, interleaved, and interactive any-to-any generation, arXiv:2311.18775 · arXiv
- Xu, 2023, PointLLM: Empowering large language models to understand point clouds, arXiv:2308.16911 · arXiv
- Guo, 2023, Point-bind & point-LLM: Aligning point cloud with multi-modality for 3D understanding, generation, and instruction following, arXiv:2309.00615 · arXiv
- Hong, 2023, 3D-LLM: Injecting the 3D world into large language models, arXiv:2307.12981 · arXiv
- Yang, 2023, LiDAR-LLM: Exploring the potential of large language models for 3D LiDAR understanding, arXiv:2312.14074 · arXiv
- Ahn, 2024, AutoRT: Embodied foundation models for large scale orchestration of robotic agents, arXiv:2401.12963 · arXiv
- Dai, 2023, InstructBLIP: Towards general-purpose vision-language models with instruction tuning, arXiv:2305.06500 · arXiv
- 10.1109/JSTARS.2021.3117857 · DOI
- 10.1109/TGRS.2023.3295797 · DOI
- 10.1109/TGRS.2022.3190934 · DOI
- 10.1109/TGRS.2021.3081421 · DOI
- 10.1109/LGRS.2019.2893772 · DOI
- 10.3390/rs11060612 · DOI · Paper page
- 10.1109/CVPR.2019.00331 · DOI · Paper page
- 10.1109/TGRS.2020.2988782 · DOI · Paper page
- 10.1109/CVPR.2016.538 · DOI · Paper page
- 10.1109/ICCV.2019.00479 · DOI
- 10.1109/ICCV.2019.00478 · DOI · Paper page
- 10.1109/CVPR46437.2021.01661 · DOI
- 10.1109/ICCV48922.2021.00179 · DOI · Paper page
- 10.1109/TGRS.2021.3062048 · DOI · Paper page
- 10.1109/CVPR46437.2021.00868 · DOI · Paper page
- 10.1109/ICCV48922.2021.00350 · DOI · Paper page
- 10.1609/aaai.v35i4.16426 · DOI · Paper page
- 10.1609/aaai.v36i1.19975 · DOI · Paper page
- 10.1109/TGRS.2017.2685945 · DOI · Paper page
- 10.1109/JPROC.2017.2675998 · DOI · Paper page
- 10.1145/1869790.1869829 · DOI · Paper page
- 10.1109/LGRS.2010.2055033 · DOI · Paper page
- 10.1109/TGRS.2014.2374218 · DOI · Paper page
- 10.1016/j.isprsjprs.2021.12.004 · DOI · Paper page
- 10.1109/TGRS.2019.2900302 · DOI · Paper page
- 10.1109/TGRS.2017.2778300 · DOI · Paper page
- 10.12000/JR19097 · DOI
- 10.1109/ACCESS.2020.3005861 · DOI · Paper page
- 10.3390/rs13183690 · DOI · Paper page
- Zamir, iSAID: A large-scale dataset for instance segmentation in aerial images, 28
- Wang, LoveDA: A remote sensing land-cover dataset for domain adaptive semantic segmentation, 1
- 10.1109/CITS.2016.7546397 · DOI · Paper page
- 10.1109/TGRS.2017.2776321 · DOI · Paper page
- 10.1109/TGRS.2021.3078451 · DOI · Paper page
- 10.1109/ACCESS.2021.3090981 · DOI · Paper page
- Chen, 2023, MiniGPT-v2: Large language model as a unified interface for vision-language multi-task learning, arXiv:2310.09478 · arXiv
- Lin, 2023, SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, arXiv:2311.07575 · arXiv
- 10.18653/v1/D18-2012 · DOI · Paper page
- 10.1109/LGRS.2015.2475299 · DOI · Paper page
- 10.3390/rs13040747 · DOI
- 10.1117/12.2532741 · DOI
- 10.1109/TGRS.2016.2645610 · DOI · Paper page
- 10.1038/s41597-023-02066-6 · DOI · Paper page
- 2021, Sea-Shipping
- 2021, Infrared-Security
- 2021, Aerial-Mancar
- 2021, Double-Light-Vehicle
- 2020, Oceanic-Ship
- Ilharco, 2021, OpenCLIP, Zenodo
- 10.1007/978-3-030-58568-6_23 · DOI · Paper page
- 10.1109/CVPR52688.2022.00928 · DOI · Paper page
- Zhao, 2024, An open and comprehensive pipeline for unified object grounding and detection, arXiv:2401.02361 · arXiv
- Wei, 2023, Lenna: Language enhanced reasoning detection assistant, arXiv:2312.02433 · arXiv
- 10.1109/CVPR52688.2022.00187 · DOI · Paper page
Cited by
RSGPT: A Remote Sensing Vision Language Model and Benchmark
Hu, Y., Yuan, J., Wen, C., Lu, X., & Li, X. (2023) · arXiv · doi:10.48550/arxiv.2307.15266 · Cited by 219
VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
Li, X., Ding, J., & Elhoseiny, M. (2024) · arXiv · doi:10.48550/arxiv.2406.12384 · Cited by 47
“Recent works explore integrating multiple datasets to accommodate a wider array of tasks [28, 29].”
Glob-Diffusion: A Global Consistent Diffusion Model for Large-Scale Image Generation
Kang, Y., Shi, H., Liu, H., Xie, W., Fang, L., & Bruzzone, L. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3643743 · Cited by 1
Metadata
- Authors
- Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X.
- Venue
- IEEE Transactions on Geoscience and Remote Sensing
- Published
- 2024
- Volume
- 62
- Pages
- 1-20
- ISSN
- 0196-2892 (ISSN-L)
- 1558-0644
- OpenAlex
- W4399399400
- Cited by
- 185
- References
- 29
- FWCI
- 26.13
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Natural Science Foundation of China (92152109)
- General Program of National Natural Science Foundation of China (62371048)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain. https://app.cochat.ai/papers/10.1109/tgrs.2024.3409624
- The paper
- Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X. (2024). EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/tgrs.2024.3409624
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.