CoChat
Sign Up
PreprintCited by 4Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationDomain Adaptation and Few-Shot Learning

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

Open access
Xu, Weiwen, Chan, Hou Pong, Li, Long, Aljunied, Mahani, Yuan, Ruifeng, Wang, Jianyu, Xiao, Chenghao, Chen, Guizhen +10 more·2025·arXiv
DOI 10.48550/arxiv.2506.07044arXiv 2506.07044
Open this paper in CoChat — free accountOpen PDF

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.07044

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing medical MLLMs focus on visual question answering and are largely confined to image-level understanding tasks, failing to achieve fine-grained pixel-level comprehension.

    “Prevailing medical MLLMsfocus on visual question answering (VQA) and are largely confined to image-level understanding tasks, thus failing to achieve fine-grained pixel-level comprehension.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [43]

    The Abstract/Introduction (page unavailable) says existing medical MLLMs have a narrow focus on visual modalities and reports their results on medical VQA. It also describes their use in report generation and diagnostic support, so it does not establish that they are confined to image-level tasks. The searched passages do not state that prevailing models fail at pixel-level comprehension; that more specific limitation is unsupported. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 96

  1. OpenAI, "Gpt-4o system card," ArXiv, 2024.
  2. G. DeepMind, "Gemini: A family of highly capable multimodal models," ArXiv, 2025.
  3. J. Zhu, W. Wang, Z. Chen et al., "Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models," ArXiv, 2025.
  4. Z. Yan, K. Zhang, R. Zhou et al., "Multimodal chat- gpt for medical applications: an experimental study of gpt4v," ArXiv, 2023.
  5. P. Lee, S. Bubeck, and J. Petro, "Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine," NEJM, vol. 388, 2023.
  6. T. Li, Y. Su, W. Li et al., "Gmai-vl & gmai-vl-5.5m: A large vision-language model and a comprehensive multimodal dataset towards general medical ai," ArXiv, 2025.
  7. C. Li, C. Wong, S. Zhang et al., "LLaVA-med: Training a large language-and-vision assistant for biomedicine in one day," in NeurIPS, 2023.
  8. J. Chen, C. Gui, R. Ouyang et al., "Huatuogpt-vision, towards injecting medical visual knowledge into multi- modal llms at scale," ArXiv, 2024.
  9. C. Wu, X. Zhang, Y. Zhang et al., "Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data," ArXiv, 2023.
  10. X. Zhang, C. Wu, Z. Zhao et al., "Pmc-vqa: Visual in- struction tuning for medical visual question answering," ArXiv, 2024.
  11. M. S. Seyfioglu, W. O. Ikezogwo, F. Ghezloo et al., "Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos," ArXiv, 2025.
  12. T. Lin, W. Zhang, S. Li et al., "Healthgpt: A medi- cal large vision-language model for unifying compre- hension and generation via heterogeneous knowledge adaptation," ArXiv, 2025.
  13. DeepSeek, "Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning," ArXiv, 2025.
  14. Z. Shao, P. Wang, Q. Zhu et al., "Deepseekmath: Pushing the limits of mathematical reasoning in open language models," ArXiv, 2024.
  15. Y. Lai, J. Zhong, M. Li et al., "Med-r1: Reinforcement learning for generalizable medical reasoning in vision- language models," ArXiv, 2025.
  16. J. Pan, C. Liu, J. Wu et al., "Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning," ArXiv, 2025.
  17. H. Liu, C. Li, Y. Li et al., "Improved baselines with visual instruction tuning," in CVPR, 2024.
  18. M. Deitke, C. Clark, S. Lee et al., "Molmo and pixmo: Open weights and open data for state-of-the-art vision- language models," ArXiv, 2024.
  19. G. H. Chen, S. Chen, R. Zhang et al., "Allava: Har- nessing gpt4v-synthesized data for lite vision-language models," ArXiv, 2024.
  20. Y. Xie, C. Zhou, L. Gao et al., "Medtrinity-25m: A large-scale multimodal dataset with multigranular an- notations for medicine," in ICLR, 2025.
  21. S. Zhang, Y. Xu, N. Usuyama et al., "Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs," ArXiv, 2025.
  22. S. Bai, K. Chen, X. Liu et al., "Qwen2.5-vl technical report," ArXiv, 2025.
  23. W. Lin, Z. Zhao, X. Zhang et al., "Pmc-clip: Con- trastive language-image pre-training using biomedical documents," in MICCAI, 2023.
  24. Z. Chen, W. Wang, Y. Cao et al., "Expanding perfor- mance boundaries of open-source multimodal models with model, data, and test-time scaling," ArXiv, 2025.
  25. K. Zhang, B. Li, P. Zhang et al., "LMMs-eval: Reality check on the evaluation of large multimodal models," in Findings of NAACL, 2025.
  26. W. Kwon, Z. Li, S. Zhuang et al., "Efficient memory management for large language model serving with pagedattention," in SOSP, 2023.
  27. J. J. Lau, S. Gayen, A. Ben Abacha et al., "A dataset of clinically generated visual questions and answers about radiology images," Scientific data, vol. 5, 2018.
  28. X. He, Y. Zhang, L. Mou et al., "Pathvqa: 30000+ ques- tions for medical visual question answering," ArXiv, 2020.
  29. Y. Hu, T. Li, Q. Lu et al., "Omnimedvqa: A new large- scale comprehensive evaluation benchmark for medical lvlm," in CVPR, 2024.
  30. X. Yue, Y. Ni, K. Zhang et al., "Mmmu: A massive multi-discipline multimodal understanding and reason- ing benchmark for expert agi," in CVPR, 2024.
  31. Y. Zuo, S. Qu, Y. Li et al., "Medxpertqa: Benchmark- ing expert-level medical reasoning and understanding," ArXiv, 2025.
  32. D. Jin, E. Pan, N. Oufattole et al., "What disease does this patient have? a large-scale open domain question answering dataset from medical exams," Applied Sci- ences, vol. 11, 2021.
  33. D. Hendrycks, C. Burns, S. Basart et al., "Measuring massive multitask language understanding," in ICLR, 2021.
  34. Q. Jin, B. Dhingra, Z. Liu et al., "Pubmedqa: A dataset for biomedical research question answering," ArXiv, 2019.
  35. H. Chen, Z. Fang, Y. Singla et al., "Benchmarking large language models on answering and explaining challenging medical questions," in NAACL, 2025.
  36. P. Team, X. Du, Y. Yao et al., "Supergpqa: Scaling llm evaluation across 285 graduate disciplines," ArXiv, 2025.
  37. X. Wang, N. Chen, J. Chen et al., "Apollo: Lightweight multilingual medical llms towards democratizing med- ical ai to 6b people," ArXiv, 2024.
  38. X. Zhang, H.-Y. Zhou, X. Yang et al., "Rexrank: A public leaderboard for ai-powered radiology report generation," ArXiv, 2024.
  39. R. Vedantam, C. Lawrence Zitnick, and D. Parikh, "Cider: Consensus-based image description evaluation," in CVPR, 2015.
  40. A. Smit, S. Jain, P. Rajpurkar et al., "Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT," in EMNLP, 2020.
  41. W. Zhao, C. Wu, X. Zhang et al., "RaTEScore: A metric for radiology report generation," in EMNLP, 2024.
  42. A. Sellergren, S. Kazemzadeh, T. Jaroensri et al., "Medgemma technical report," ArXiv, vol. abs/2507.05201, 2025. · arXiv
  43. S. S. Mullappilly, M. I. Kurpath, S. Pieri et al., "Bimedix2: Bio-medical expert lmm for diverse medical modalities," ArXiv, 2024.
  44. S. He, Y. Nie, H. Wang et al., "Gsco: Towards gener- alizable ai in medicine via generalist-specialist collab- oration," ArXiv, 2024.
  45. B. Zhang, K. Li, Z. Cheng et al., "Videollama 3: Frontier multimodal foundation models for image and video understanding," arXiv preprint arXiv:2501.13106, 2025. · arXiv
  46. D. Tian, S. Jiang, L. Zhang et al., "The role of large language models in medical image processing: a nar- rative review," Quantitative Imaging in Medicine and Surgery, vol. 14, 2023.
  47. R. AlSaad, A. Abd-alrazaq, S. Boughorbel et al., "Mul- timodal large language models in health care: Applica- tions, challenges, and future outlook," J Med Internet Res, vol. 26, 2024.
  48. M. Moor, Q. Huang, S. Wu et al., "Med-flamingo: a multimodal medical few-shot learner," in Machine Learning for Health Symposium, 2023.
  49. W. O. Ikezogwo, M. S. Seyfioglu, F. Ghezloo et al., "Quilt-1m: One million image-text pairs for histopathol- ogy," in NeurIPS, 2023.
  50. Y. Sun, X. Qian, W. Xu et al., "Reasonmed: A 370k multi-agent generated dataset for advancing medical reasoning," ArXiv, vol. abs/2506.09513, 2025. · arXiv
  51. V. Nath, W. Li, D. Yang et al., "Vila-m3: Enhancing vision-language models with medical expert knowl- edge," ArXiv, 2025.
  52. L. Yang, S. Xu, A. Sellergren et al., "Advancing mul- timodal medical capabilities of gemini," ArXiv, 2024.
  53. O. Aydin and E. Karaarslan, "Openai chatgpt interprets radiological images: Gpt-4 as a medical doctor for a fast check-up," ArXiv, 2025.
  54. R. K. Arora, J. Wei, R. S. Hicks et al., "Healthbench: Evaluating large language models towards improved human health," ArXiv, 2025.
  55. M. Y. Lu, B. Chen, D. F. K. Williamson et al., "A multimodal generative ai copilot for human pathology," Nature, vol. 634, 2024.
  56. T. Zhao, Y. Gu, J. Yang et al., "A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities," Nature Methods, vol. 22, 2024.
  57. S. L. Hyland, S. Bannur, K. Bouzid et al., "Maira-1: A specialised large multimodal model for radiology report generation," ArXiv, 2024.
  58. R. Tanno, D. G. T. Barrett, A. Sellergren et al., "Col- laboration between clinicians and vision-language mod- els in radiology report generation," Nature Medicine, vol. 31, 2025.
  59. Z. Deng, W. Gao, C. Chen et al., "Ophglm: An oph- thalmology large language-and-vision assistant," Artif. Intell. Med., vol. 157, 2024.
  60. J. Keyl, P. Keyl, G. Montavon et al., "Decoding pan- cancer treatment outcomes using multimodal real-world data and explainable artificial intelligence," Nature Can- cer, vol. 6, 2025.
  61. S.-J. Sammut, M. Crispin-Ortuzar, S.-F. Chin et al., "Multi-omic machine learning predictor of breast cancer therapy response," Nature, vol. 601, 2022.
  62. C. Niu, Q. Lyu, C. D. Carothers et al., "Medical multimodal multitask foundation model for lung cancer screening," Nature Communications, vol. 16, 2025.
  63. S. Schmidgall, R. Ziaei, C. Harris et al., "Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments," ArXiv, 2025.
  64. J. Li, Y. Lai, W. Li et al., "Agent hospital: A simulacrum of hospital with evolvable medical agents," ArXiv, 2025.
  65. Z. Wang, J. Wu, L. Cai et al., "Medagent-pro: Towards evidence-based multi-modal medical diagnosis via rea- soning agentic workflow," ArXiv, 2025.
  66. Y. Li, Z. Li, K. Zhang et al., "Chatdoctor: A medical chat model fine-tuned on a large language model meta- ai (llama) using medical domain knowledge," Cureus, vol. 15, 2023.
  67. A. Dubey, A. Jauhri, A. Pandey et al., "The llama 3 herd of models," ArXiv, 2024.
  68. S. Subramanian, L. L. Wang, B. Bogin et al., "MedI- CaT: A dataset of medical images, captions, and textual references," in Findings of EMNLP, 2020.
  69. Y. Luo, M. Shi, M. O. Khan et al., "Fairclip: Harnessing fairness in vision-language learning," in CVPR, 2024.
  70. I. Siragusa, S. Contino, M. Ciura et al., "Medpix 2.0: A comprehensive multimodal biomedical data set for advanced ai applications," ArXiv, 2025.
  71. P. Chambon, J. Delbrouck, T. Sounack et al., "Chexpert plus: Augmenting a large chest x-ray dataset with text radiology reports, patient demographics and additional image formats," ArXiv, 2024.
  72. A. Ben Abacha and D. Demner-Fushman, "A question- entailment approach to question answering," BMC Bioinformatics, vol. 20, 2019.
  73. J. Wu, W. Deng, X. Li et al., "Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs," ArXiv, 2025.
  74. J. Chen, Z. Cai, K. Ji et al., "Huatuogpt-o1, towards medical complex reasoning with llms," ArXiv, 2024.
  75. J. Chen, X. Wang, A. Gao et al., "Huatuogpt-ii, one- stage training for medical adaption of llms," ArXiv, 2023.
  76. M. E. H. Chowdhury, T. Rahman, A. Khandakar et al., "Can ai help in screening viral and covid-19 pneumo- nia?" IEEE Access, 2020.
  77. X. Wang, Y. Peng, L. Lu et al., "Chestx-ray8: Hospital- scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases," in CVPR, 2017.
  78. J. Irvin, P. Rajpurkar, M. Ko et al., "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison," in AAAI, vol. 33, 2019.
  79. Y. He, G. Yang, J. Yang et al., "Meta grayscale adaptive network for 3d integrated renal structures segmenta- tion," Medical Image Analysis, vol. 71, 2021.
  80. K. Yan, X. Wang, L. Lu et al., "Deeplesion: Automated deep mining, categorization and detection of significant radiology image findings using large-scale clinical le- sion annotations," ArXiv, 2017.
  81. S. Vitale, J. I. Orlando, E. Iarussi et al., "Improving realism in patient-specific abdominal ultrasound simu- lation using cyclegans," IJCARS, vol. 15, 2020.
  82. X. Mei, Z. Liu, P. M. Robson et al., "Radimagenet: An open radiologic deep learning research dataset for effective transfer learning," Radiology: Artificial Intel- ligence, vol. 0, 2022.
  83. T. Philipp, R. Cliff, and K. Harald, "The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions," Scientific data, vol. 5, 2018.
  84. M. Groh, C. Harris, L. Soenksen et al., "Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset," in CVPR, 2021.
  85. O. Kovalyk, J. Morales-Sánchez, R. Verdú-Monedero et al., "Papila: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment," Scientific Data, vol. 9, 2022.
  86. P. Wagner, M. Springenberg, M. Kröger et al., "Se- mantic modeling of cell damage prediction: a machine learning approach at human-level performance in der- matology," Scientific Reports, vol. 13, 2023.
  87. J. Gamper, N. A. Koohbanani, K. Benes et al., "Pannuke dataset extension, insights and baselines," ArXiv, 2020.
  88. L. Shi, X. Li, W. Hu et al., "Ebhi-seg: A novel entero- scope biopsy histopathological hematoxylin and eosin image dataset for image segmentation tasks," Frontiers in Medicine, vol. 10, 2023.
  89. M. C. de Verdier, R. Saluja, L. Gagnon et al., "The 2024 brain tumor segmentation (brats) challenge: glioma seg- mentation on post-treatment mri," ArXiv, 2024.
  90. P. Rajpurkar, J. Irvin, K. Zhu et al., "Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning," ArXiv, 2017.
  91. A. G. Pacheco, G. R. Lima, A. S. Salomão et al., "Pad- ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones," Data in Brief, vol. 32, 2020.
  92. F. Meng, L. Du, Z. Liu et al., "Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning," ArXiv, 2025.
  93. H. Wang, C. Qu, Z. Huang et al., "Vl-rethinker: Incen- tivizing self-reflection of vision-language models with reinforcement learning," ArXiv, 2025.
  94. H. Tan, Y. Ji, X. Hao et al., "Reason-rft: Reinforcement fine-tuning for visual reasoning," ArXiv, 2025.
  95. E. Yeo, Y. Tong, X. Niu et al., "Demystifying long chain-of-thought reasoning in LLMs," in ICLR Work- shop, 2025.
  96. J. R. Landis and G. G. Koch, "The measurement of observer agreement for categorical data," Biometrics, vol. 33, no. 1, pp. 159-174, 1977. 23 https://doi.org/10.17632/t9ndx37v5h.1 24 https://tianchi.aliyun.com/dataset/93929 25 https://www.kaggle.com/dsv/2645886 26 https://www.synapse.org/Synapse:syn53708249/wiki/ 27 https://www.kaggle.com/datasets/aryashah2k/ breast-ultrasound-images-dataset 28 https://doi.org/10.5281/zenodo.7272660 29 https://kaggle.com/competitions/isic-2024-challenge 30 https://www.kaggle.com/datasets/c7934597/ resized-2015-2019-diabetic-retinopathy-detection 31 https://github.com/Shenggan/BCCD Dataset 32 https://figshare.com/s/d6fb591f1beb4f8efa6f TABLE XXII: Doctor preferred features in the summarization stage of medical long caption synthesis for MRI, X-ray, CT, histopathology, skin, and knee X-ray images. Doctor preferred features for MRI images: 1. The sequence used; 2. Plane/orientation of the image; 3. The anatomical structures visible and their normal or abnormal appearance; · DOI

Cited by

Sorted by citations

Authors
Xu, W., Chan, H. P., Li, L., Aljunied, M., Yuan, R., Wang, J., Xiao, C., Chen, G., Liu, C., Li, Z., Sun, Y., Shen, J., Wang, C., Tan, J., Zhao, D., Xu, T., Zhang, H., & Rong, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.07044
arXiv
2506.07044
OpenAlex
W4417307773
Cited by
4
References
0
Citations by year
2026: 3 · 2025: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
Ling Shu, multimodal large language models, medical large language models, visual question answering, medical reasoning, hallucination mitigation, reinforcement learning with verifiable rewards, medical report generation, multimodal question answering, medical text understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review