Large Language Models for Captioning and Retrieving Remote Sensing Images
Open accessAbstract
Image captioning and cross-modal retrieval are examples of tasks that involve the joint analysis of visual and linguistic information. In connection to remote sensing imagery, these tasks can help non-expert users in extracting relevant Earth observation information for a variety of applications. Still, despite some previous efforts, the development and application of vision and language models to the remote sensing domain have been hindered by the relatively small size of the available datasets and models used in previous studies. In this work, we propose RS-CapRet, a Vision and Language method for remote sensing tasks, in particular image captioning and text-image retrieval. We specifically propose to use a highly capable large decoder language model together with image encoders adapted to remote sensing imagery through contrastive language-image pre-training. To bridge together the image encoder and language decoder, we propose training simple linear layers with examples from combining different remote sensing image captioning datasets, keeping the other parameters frozen. RS-CapRet can then generate descriptions for remote sensing images and retrieve images from textual descriptions, achieving SOTA or competitive performance with existing methods. Qualitative results illustrate that RS-CapRet can effectively leverage the pre-trained large language model to describe remote sensing images, retrieve them based on different types of queries, and also show the ability to process interleaved sequences of images and text in a dialogue manner.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM-based methods can produce explanations that are more detailed, coherent, and linguistically flexible than those from earlier vision-language models.
“Recent research shows that LLM-based methods can produce explanations that are more detailed, coherent, and linguistically flexible than those from earlier vision-language models [19].”
From Human vs LLM-Generated Alt Text for STEM Image Accessibility: A Comparative Study (2026) · cites this paper as [19]
The Image Captioning Results section (page unavailable) reports that the LLM-based RS-CapRet model outperforms an earlier captioning model on several metrics, and the Qualitative Analysis section shows that it generates image descriptions. Neither section establishes that its output is more detailed, coherent, or linguistically flexible than explanations from earlier vision-language models. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 77
- C. Wen, Y. Hu, X. Li, Z. Yuan, and X. X. Zhu, "Vision-language models in remote sensing: Current progress and future trends," arXiv:2305.05726, 2023. · arXiv
- G. Mai, W. Huang, J. Sun, S. Song, D. Mishra, N. Liu, S. Gao, T. Liu, G. Cong, Y. Hu et al., "On the opportunities and challenges of foun- dation models for geospatial artificial intelligence," arXiv:2304.06798, 2023. · arXiv
- F. Liu, D. Chen, Z. Guan, X. Zhou, J. Zhu, and J. Zhou, "Re- moteclip: A vision language foundation model for remote sensing," arXiv:2306.11029, 2023. · arXiv
- M. M. A. Rahhal, Y. Bazi, N. A. Alsharif, L. Bashmal, N. Alajlan, and F. Melgani, "Multilanguage transformer for improved text to remote sensing image retrieval," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 9115-9126, 2022.
- L. Mi, S. Li, C. Chappuis, and D. Tuia, "Knowledge-aware cross-modal text-image retrieval for remote sensing images," in Proceedings of the Second Workshop on Complex Data Challenges in Earth Observation (CDCEO 2022), 2022.
- Z. Yuan, W. Zhang, C. Tian, X. Rong, Z. Zhang, H. Wang, K. Fu, and X. Sun, "Remote sensing cross-modal text-image retrieval based on global and local information," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-16, 2022.
- Q. Cheng, H. Huang, Y. Xu, Y. Zhou, H. Li, and Z. Wang, "Nwpu- captions dataset and mlca-net for remote sensing image captioning," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1- 19, 2022.
- T. Wei, W. Yuan, J. Luo, W. Zhang, and L. Lu, "VLCA: Vision-language aligning model with cross-modal attention for bilingual remote sensing image captioning," Journal of Systems Engineering and Electronics, vol. 34, no. 1, pp. 9-18, 2023.
- R. Ramos and B. Martins, "Using Neural Encoder-Decoder Models With Continuous Outputs for Remote Sensing Image Captioning," IEEE Access, vol. 10, 2022.
- Z. Shi and Z. Zou, "Can a Machine Generate Humanlike Language Descriptions for a Remote Sensing Image?" IEEE Transactions on Geoscience and Remote Sensing, vol. 55, 2017.
- S. Lobry, D. Marcos, J. Murray, and D. Tuia, "RSVQA: Visual Question Answering for Remote Sensing Data," IEEE Transactions on Geoscience and Remote Sensing, vol. 58, 2020.
- J. D. Silva, J. Magalhães, D. Tuia, and B. Martins, "Remote sensing visual question answering with a self-attention multi-modal encoder," in Proceedings of the 5th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. Association for Computing Machinery, 2022, pp. 40-49.
- Y. Bazi, M. M. Al Rahhal, M. L. Mekhalfi, M. A. Al Zuair, and F. Melgani, "Bi-modal transformer-based approach for visual question answering in remote sensing imagery," IEEE Transactions on Geo- science and Remote Sensing, vol. 60, pp. 1-11, 2022.
- Z. Zhang, L. Jiao, L. Li, X. Liu, P. Chen, F. Liu, Y. Li, and Z. Guo, "A spatial hierarchical reasoning network for remote sensing visual question answering," IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1-15, 2023.
- D. Tuia, R. Roscher, J. D. Wegner, N. Jacobs, X. Zhu, and G. Camps- Valls, "Toward a collective agenda on AI for earth science data analysis," IEEE Geoscience and Remote Sensing Magazine, vol. 9, no. 2, 2021.
- Y. Hu, J. Yuan, C. Wen, X. Lu, and X. Li, "Rsgpt: A remote sensing vision language model and benchmark," arXiv:2307.15266, 2023. · arXiv
- Y. Zhan, Z. Xiong, and Y. Yuan, "Skyeyegpt: Unifying remote sensing vision-language tasks via instruction tuning with large language model," arXiv:2401.09712, 2024. · arXiv
- K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan, "Geochat: Grounded large vision-language model for remote sensing," arXiv:2311.15826, 2023. · arXiv
- Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao, "SimVLM: Simple Visual Language Model Pretraining with Weak Supervision," arXiv:2108.10904, 2021. · arXiv
- J. Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation," in International Conference on Machine Learning. PMLR, 2022, pp. 12 888-12 900.
- J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al., "Flamingo: A visual language model for few-shot learning," Advances in Neural Information Processing Systems, vol. 35, pp. 23 716-23 736, 2022.
- S. Hochreiter and J. Schmidhuber, "Long Short-term Memory," Neural computation, vol. 9, pp. 1735-1780, 1997.
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre- training of Deep Bidirectional Transformers for Language Understand- ing," arXiv:1810.04805, 2019. · arXiv
- C. Chappuis, V. Zermatten, S. Lobry, B. Le Saux, and D. Tuia, "Prompt- rsvqa: Prompting visual context to a language model for remote sensing visual question answering," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1372-1381.
- X. Lu, B. Wang, X. Zheng, and X. Li, "Exploring models and data for remote sensing image caption generation," IEEE Transactions on Geoscience and Remote Sensing, vol. 56, 2018.
- U. Zia, M. M. Riaz, and A. Ghafoor, "Transforming remote sensing images to textual descriptions," International Journal of Applied Earth Observation and Geoinformation, vol. 108, p. 102741, 2022.
- W. Huang, Q. Wang, and X. Li, "Denoising-based multiscale feature fusion for remote sensing image captioning," IEEE Geoscience and Remote Sensing Letters, vol. 18, no. 3, pp. 436-440, 2021.
- Y. Li, S. Fang, L. Jiao, R. Liu, and R. Shang, "A multi-level attention model for remote sensing image captions," Remote Sensing, vol. 12, no. 6, 2020.
- Z. Yuan, X. Li, and Q. Wang, "Exploring multi-level attention and se- mantic relationship for remote sensing image captioning," IEEE Access, vol. 8, pp. 2608-2620, 2020.
- R. Zhao, Z. Shi, and Z. Zou, "High-resolution remote sensing image captioning based on structured attention," IEEE Transactions on Geo- science and Remote Sensing, vol. 60, pp. 1-14, 2021.
- F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, "Language models as knowledge bases?" arXiv:1909.01066, 2019. · arXiv
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language mod- els are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
- J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler et al., "Emergent abilities of large language models," arXiv:2206.07682, 2022. · arXiv
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in Neural Information Processing Systems, vol. 35, pp. 24 824-24 837, 2022.
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large lan- guage models are zero-shot reasoners," Advances in neural information processing systems, vol. 35, pp. 22 199-22 213, 2022.
- J. Huang and K. C.-C. Chang, "Towards reasoning in large language models: A survey," arXiv:2212.10403, 2022. · arXiv
- J. Guo, J. Li, D. Li, A. M. H. Tiong, B. Li, D. Tao, and S. C. Hoi, "From images to textual prompts: Zero-shot VQA with frozen large language models," arXiv:2212.10846, 2022. · arXiv
- Z. Yang, Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang, "An empirical study of gpt-3 for few-shot knowledge-based vqa," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 3081-3089.
- M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill, "Multimodal few-shot learning with frozen language models," Advances in Neural Information Processing Systems, vol. 34, pp. 200- 212, 2021.
- E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," arXiv:2106.09685, 2021. · arXiv
- B. Lester, R. Al-Rfou, and N. Constant, "The power of scale for parameter-efficient prompt tuning," in Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. Association for Computational Linguistics, 2021, pp. 3045-3059.
- J. Y. Koh, R. Salakhutdinov, and D. Fried, "Grounding language models to images for multimodal inputs and outputs," in International Conference on Machine Learning. PMLR, 2023, pp. 17 283-17 300.
- C. Eichenberg, S. Black, S. Weinbach, L. Parcalabescu, and A. Frank, "MAGMA -multimodal augmentation of generative models through adapter-based finetuning," in Findings of the Association for Computational Linguistics: EMNLP 2022, Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 2416-2428. [Online]. Available: https://aclanthology.org/2022.findings-emnlp.179
- R. Ramos, B. Martins, D. Elliott, and Y. Kementchedjhieva, "Smallcap: lightweight image captioning prompted with retrieval augmentation," pp. 2840-2849, 2023.
- J. Li, D. Li, S. Savarese, and S. Hoi, "Blip-2: bootstrapping language- image pre-training with frozen image encoders and large language mod- els," in Proceedings of the 40th International Conference on Machine Learning, ser. ICML'23. JMLR.org, 2023.
- B. Qu, X. Li, D. Tao, and X. Lu, "Deep semantic understanding of high resolution remote sensing image," in Proceedings of the International Conference on Computer, Information and Telecommunication Systems, 2016.
- G. Sumbul, S. Nayak, and B. Demir, "Sd-rsic: Summarization-driven deep remote sensing image captioning," IEEE Transactions on Geo- science and Remote Sensing, vol. 59, no. 8, pp. 6922-6934, 2021.
- X. Li, X. Zhang, W. Huang, and Q. Wang, "Truncation cross entropy loss for remote sensing image captioning," IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 6, pp. 5246-5257, 2021.
- K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv:1409.1556, 2014. · arXiv
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
- N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using Siamese BERT-networks," in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP- IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3982-3992.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning Transferable Visual Models From Natural Language Supervi- sion," in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol. 139. PMLR, 2021, pp. 8748-8763.
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv:2010.11929, 2020. · arXiv
- V. Zermatten, J. C. Navarro, L. Hughes, T. Kellenberger, and D. Tuia, "Text as a richer source of supervision in semantic segmentation tasks," in IGARSS 2023 -2023 IEEE International Geoscience and Remote Sensing Symposium, 2023, pp. 2219-2222.
- J. Jakubik, M. Muszynski, M. Vössing, N. Kühl, and T. Brunschwiler, "Toward foundation models for earth monitoring: Generalizable deep learning models for natural hazard segmentation," arXiv:2301.09318, 2023. · arXiv
- D. Wang, Q. Zhang, Y. Xu, J. Zhang, B. Du, D. Tao, and L. Zhang, "Advancing plain vision transformer towards remote sensing foundation model," IEEE Transactions on Geoscience and Remote Sensing, 2022.
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. B. Girshick, "Masked Autoencoders Are Scalable Vision Learners," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 2022, pp. 15 979-15 988.
- Y. Long, G.-S. Xia, S. Li, W. Yang, M. Y. Yang, X. X. Zhu, L. Zhang, and D. Li, "On creating benchmark dataset for aerial image interpre- tation: Reviews, guidances and million-aid," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 4205-4230, 2021.
- X. Sun, P. Wang, W. Lu, Z. Zhu, X. Lu, Q. He, J. Li, X. Rong, Z. Yang, H. Chang et al., "Ringmo: A remote sensing foundation model with masked image modeling," IEEE Transactions on Geoscience and Remote Sensing, 2022.
- K. Cha, J. Seo, and T. Lee, "A billion-scale foundation model for remote sensing images," arXiv:2304.05215, 2023. · arXiv
- M. Mendieta, B. Han, X. Shi, Y. Zhu, C. Chen, and M. Li, "GFM: Building geospatial foundation models via continual pretraining," arXiv:2302.04476, 2023. · arXiv
- A. Lacoste, N. Lehmann, P. Rodriguez, E. D. Sherwin, H. Kerner, B. Lütjens, J. A. Irvin, D. Dao, H. Alemohammad, A. Drouin, M. Gun- turkun, G. Huang, D. Vazquez, D. Newman, Y. Bengio, S. Ermon, and X. X. Zhu, "GEO-bench: Toward foundation models for earth moni- toring," in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
- W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, "Instructblip: Towards general-purpose vision- language models with instruction tuning," arXiv:2305.06500, 2023. · arXiv
- H. Liu, C. Li, Y. Li, and Y. J. Lee, "Improved baselines with visual instruction tuning," arXiv:2310.03744, 2023. · arXiv
- J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V. Chandra, Y. Xiong, and M. Elhoseiny, "Minigpt-v2: large language model as a unified interface for vision-language multi-task learning," arXiv:2310.09478, 2023. · arXiv
- H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv:2307.09288, 2023. · arXiv
- H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning," arXiv:2304.08485, 2023. · arXiv
- A. Roberts, C. Raffel, and N. Shazeer, "How much knowledge can you pack into the parameters of a language model?" arXiv:2002.08910, 2020. · arXiv
- D. Ganguli, D. Hernandez, L. Lovitt, A. Askell, Y. Bai, A. Chen, T. Conerly, N. Dassarma, D. Drain, N. Elhage et al., "Predictability and surprise in large generative models," in 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 1747-1764.
- C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, "Scaling up visual and vision-language representation learning with noisy text supervision," in International Conference on Machine Learning. PMLR, 2021, pp. 4904-4916.
- Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao, "Eva: Exploring the limits of masked visual representation learning at scale," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 358-19 369.
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, "LLaMA: Open and Efficient Foundation Language Models," arXiv:2302.13971, 2023. · arXiv
- J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., "Training compute-optimal large language models," arXiv:2203.15556, 2022. · arXiv
- A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, vol. 32, 2019.
- T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A Simple Framework for Contrastive Learning of Visual Representations," arXiv:2002.05709, 2020. · arXiv
- J. Merullo, L. Castricato, C. Eickhoff, and E. Pavlick, "Linearly mapping from image to text space," arXiv:2209.15162, 2022. · arXiv
- K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, "Shikra: Unleashing multimodal llm's referential dialogue magic," arXiv:2306.15195, 2023. · arXiv
Cited by
Sorted by citations
Human vs LLM-Generated Alt Text for STEM Image Accessibility: A Comparative Study
Cardia, M., Angileri, L., Poggianti, C., & Leporini, B. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800457 · Cited by 2 · see citing works
- Authors
- Silva, J. D., Magalhães, J., Tuia, D., & Martins, B.
- Affiliations
- Instituto de Engenharia de Sistemas e Computadores Investigação e Desenvolvimento (PT)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.06475
- OpenAlex
- W4391766847
- Cited by
- 20
- References
- 0
- Citations by year
- 2026: 1 · 2025: 9 · 2024: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- remote sensing image captioning, image-text retrieval, vision-language models, large language models, contrastive language-image pretraining, linear probing, cross-modal retrieval, Earth observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review