CoChat
Sign Up
Cited by 14Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationAdvanced Neural Network Applications

Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine

Open access
Huang, Xiaoshuang, Shen, Lingdong, Liu, Jia, Shang, Fangxin, Li, Hongxiang, Huang, Haifeng, Yang, Yehui·2025·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v39i4.32394
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, Multimodal Large Language Models (MLLM) have achieved notable advancements, demonstrating the feasibility of developing an intelligent biomedical assistant. However, current biomedical MLLMs predominantly focus on image-level understanding and restrict interactions to textual commands, thus limiting their capability boundaries and the flexibility of usage. In this paper, we introduce a novel end-to-end multimodal large language model for the biomedical domain, named MedPLIB, which possesses pixel-level understanding. Excitingly, it supports visual question answering (VQA), arbitrary pixel-level prompts (points, bounding boxes, and free-form shapes), and pixel-level grounding. We propose a novel Mixture-of-Experts (MoE) multi-stage training strategy, which divides MoE into separate training phases for a visual-language expert model and a pixel-grounding expert model, followed by fine-tuning using MoE. This strategy effectively coordinates multitask learning while maintaining the computational cost at inference equivalent to that of a single expert model. To advance the research of biomedical MLLMs, we introduce the Medical Complex Vision Question Answering Dataset (MeCoVQA), which comprises an array of 8 modalities for complex medical imaging question answering and image region understanding. Experimental results indicate that MedPLIB has achieved state-of-the-art outcomes across multiple medical visual language tasks. More importantly, in zero-shot evaluations for the pixel grounding task, MedPLIB leads the best small and large models by margins of 19.7 and 15.6 respectively on the mDice metric.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v39i4.32394

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing medical MLLM approaches introduce implicit segmentation tokens and require joint fine-tuning of the MLLM and external pixel decoders, increasing the risk of catastrophic forgetting and limiting out-of-domain generalization.

    “First, they introduce implicit segmentation tokens and require joint fine-tuning of the MLLM and external pixel decoders, increasing the risk of catastrophic forgetting and limiting out-of-domain generalization.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [13]

    The Introduction (page unavailable) says MedPLIB uses a <SEG> token with a mask decoder, and Multi-stage Training (page unavailable) describes fine-tuning the model and pixel decoder. These passages support the architectural description as an example, but do not establish that existing medical MLLM approaches generally require joint fine-tuning. The paper discusses generalization challenges but does not link them—or catastrophic forgetting—to that fine-tuning approach. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Dependence on joint fine-tuning of the MLLM and external pixel decoders heightens the risk of catastrophic forgetting, resulting in strong in-domain performance but weak cross-domain generalization.

    “Their dependence on joint fine-tuning of the MLLM and external pixel decoders heightens the risk of catastrophic forgetting , resulting in strong in-domain performance but weak cross-domain generalization.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [13]

    The Zero-shot to Pixel Grounding section reports that smaller medical image-grounding models face generalization challenges on unseen imaging modalities, while the evaluation table shows substantially higher scores on the MeCoVQA-G test set than on zero-shot modalities for several such models. The paper does not attribute that gap to joint fine-tuning of an MLLM and external pixel decoder or to catastrophic forgetting. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • MedPLIB provides coarse localization that roughly captures the lesion region but suffers from false-positive predictions.

    “MedPLIB [13] provides coarse localization that roughly captures the lesion region but suffers from false-positive predictions.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [13]

    The Conclusion shows MedPLIB being prompted to segment lesions, and Additional information (page not provided) acknowledges hallucinations and substantial pixel-grounding errors for small targets. The paper does not characterize its lesion localization as coarse or specifically report false-positive predictions. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • The released split of MeCoVQA-G is 100K training pairs and 2,344 test pairs.

    “The released split is 100 K training pairs and 2 , 344 test pairs.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [13]

  • This process disrupts the MLLM's original text output space and increases the risk of catastrophic forgetting, thereby compromising semantic generalization by deviating from language-based outputs.

    “This process disrupts the MLLM's original text output space and increases the risk of catastrophic forgetting, thereby compromising semantic generalization by deviating from languagebased outputs.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [12]

    The Introduction describes the challenge of balancing visual question answering and pixel-level analysis, while Multi-stage Training says expert mixing preserves each expert’s prior knowledge. Neither passage says that the process disrupts the MLLM’s text output space or increases catastrophic forgetting. Searches for those terms and for impaired semantic generalization found no passage addressing that claim. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 71

  1. Ben Abacha, A.; Sarrouti, M.; Demner-Fushman, D.; Hasan, S. A.; and Müller, H. 2021. Overview of the vqa-med task at imageclef 2021: Visual question answering and gen- eration in the medical domain. In Proceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum- working notes. 21-24 September 2021.
  2. Cao, H.; Wang, Y.; Chen, J.; Jiang, D.; Zhang, X.; Tian, Q.; and Wang, M. 2022. Swin-unet: Unet-like pure transformer for medical image segmentation. In European conference on computer vision, 205-218. Springer.
  3. Chen, J.; Guo, L.; Sun, J.; Shao, S.; Yuan, Z.; Lin, L.; and Zhang, D. 2024. Eve: Efficient vision-language pre-training with masked prediction and modality-aware moe. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 38, 1110-1119.
  4. Chen, J.; Lu, Y.; Yu, Q.; Luo, X.; Adeli, E.; Wang, Y.; Lu, L.; Yuille, A. L.; and Zhou, Y. 2021. Transunet: Transform- ers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306. · arXiv
  5. Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Kr- ishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023. Minigpt-v2: large language model as a unified inter- face for vision-language multi-task learning. arXiv preprint arXiv:2310.09478. · arXiv
  6. Cheng, J.; Ye, J.; Deng, Z.; Chen, J.; Li, T.; Wang, H.; Su, Y.; Huang, Z.; Chen, J.; Jiang, L.; et al. 2023. Sam-med2d. arXiv preprint arXiv:2308.16184. · arXiv
  7. Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P. N.; and Hoi, S. 2024. Instructblip: Towards general-purpose vision-language models with in- struction tuning. Advances in Neural Information Process- ing Systems, 36.
  8. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint arXiv:1810.04805. · arXiv
  9. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. · arXiv
  10. Fedus, W.; Zoph, B.; and Shazeer, N. 2022. Switch trans- formers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Re- search, 23(120): 1-39.
  11. He, S.; Nie, Y.; Chen, Z.; Cai, Z.; Wang, H.; Yang, S.; and Chen, H. 2024. MedDr: Diagnosis-Guided Bootstrapping for Large-Scale Medical Vision-Language Learning. arXiv preprint arXiv:2404.15127. · arXiv
  12. He, X. 2021. Towards Visual Question Answering on Pathology Images. In Proceedings of the 59th annual meet- ing of the association for computational linguistics and the 11th international joint conference on natural language pro- cessing, volume 2.
  13. Hendrycks, D.; and Gimpel, K. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. · arXiv
  14. Hochreiter, S.; and Schmidhuber, J. 1997. Long short-term memory. Neural computation, 9(8): 1735-1780.
  15. Hu, Y.; Li, T.; Lu, Q.; Shao, W.; He, J.; Qiao, Y.; and Luo, P. 2024. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22170-22183.
  16. Hu, Y.; Wang, Q.; Shao, W.; Xie, E.; Li, Z.; Han, J.; and Luo, P. 2023. Beyond one-to-one: Rethinking the referring image segmentation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, 4067-4077.
  17. Huang, X.; Li, H.; Cao, M.; Chen, L.; You, C.; and An, D. 2024. Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation. arXiv preprint arXiv:2404.02845. · arXiv
  18. Jacobs, R. A.; Jordan, M. I.; Nowlan, S. J.; and Hinton, G. E. 1991. Adaptive mixtures of local experts. Neural computa- tion, 3(1): 79-87.
  19. Kim, J.-H.; Jun, J.; and Zhang, B.-T. 2018. Bilinear atten- tion networks. Advances in neural information processing systems, 31.
  20. Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2024. Lisa: Reasoning segmentation via large lan- guage model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9579-9589.
  21. Lau, J. J.; Gayen, S.; Ben Abacha, A.; and Demner- Fushman, D. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1): 1-10.
  22. Lee, J.; Yoon, W.; Kim, S.; Kim, D.; Kim, S.; So, C. H.; and Kang, J. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinfor- matics, 36(4): 1234-1240.
  23. Li, C.; Wong, C.; Zhang, S.; Usuyama, N.; Liu, H.; Yang, J.; Naumann, T.; Poon, H.; and Gao, J. 2024. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Sys- tems, 36.
  24. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen im- age encoders and large language models. In International conference on machine learning, 19730-19742.
  25. PMLR.
  26. Li, Z.; Li, Y.; Li, Q.; Wang, P.; Guo, D.; Lu, L.; Jin, D.; Zhang, Y.; and Hong, Q. 2023b. Lvit: language meets vision transformer in medical image segmentation. IEEE transac- tions on medical imaging.
  27. Lin, B.; Tang, Z.; Ye, Y.; Cui, J.; Zhu, B.; Jin, P.; Zhang, J.; Ning, M.; and Yuan, L. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947. · arXiv
  28. Liu, B.; Zhan, L.-M.; Xu, L.; Ma, L.; Yang, Y.; and Wu, X.-M. 2021. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 1650-1654. IEEE.
  29. Liu, F.; Zhu, T.; Wu, X.; Yang, B.; You, C.; Wang, C.; Lu, L.; Liu, Z.; Zheng, Y.; Sun, X.; et al. 2023a. A medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6(1): 226.
  30. Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024. Visual in- struction tuning. Advances in neural information processing systems, 36.
  31. Long, Z.; Killick, G.; McCreadie, R.; and Camarasa, G. A. 2023. Multiway-adapater: Adapting large-scale multi-modal models for scalable image-text retrieval. arXiv preprint arXiv:2309.01516. · arXiv
  32. Luo, Y.; Zhang, J.; Fan, S.; Yang, K.; Wu, Y.; Qiao, M.; and Nie, Z. 2023. Biomedgpt: Open multimodal genera- tive pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442. · arXiv
  33. PMLR.
  34. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, 8748-8763.
  35. PMLR.
  36. Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 234-241. Springer.
  37. Simonyan, K.; and Zisserman, A. 2014. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. · arXiv
  38. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  39. Wu, C.; Zhang, X.; Zhang, Y.; Wang, Y.; and Xie, W. 2023a. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463. · arXiv
  40. Wu, J.; Fu, R.; Fang, H.; Liu, Y.; Wang, Z.; Xu, Y.; Jin, Y.; and Arbel, T. 2023b. Medical sam adapter: Adapting seg- ment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620. · arXiv
  41. Yang, Z.; Wang, J.; Tang, Y.; Chen, K.; Zhao, H.; and Torr, P. H. 2022. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18155-18165.
  42. Ye, J.; Cheng, J.; Chen, J.; Deng, Z.; Li, T.; Wang, H.; Su, Y.; Huang, Z.; Chen, J.; Jiang, L.; et al. 2023a. Sa-med2d- 20m dataset: Segment anything in 2d medical imaging with 20 million masks. arXiv preprint arXiv:2311.11969. · arXiv
  43. Ye, Q.; Xu, H.; Xu, G.; Ye, J.; Yan, M.; Zhou, Y.; Wang, J.; Hu, A.; Shi, P.; Shi, Y.; et al. 2023b. mplug-owl: Modular- ization empowers large language models with multimodal- ity. arXiv preprint arXiv:2304.14178. · arXiv
  44. You, H.; Zhang, H.; Gan, Z.; Du, X.; Zhang, B.; Wang, Z.; Cao, L.; Chang, S.-F.; and Yang, Y. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704. · arXiv
  45. Zhang, A.; Fei, H.; Yao, Y.; Ji, W.; Li, L.; Liu, Z.; and Chua, T.-S. 2024. VPGTrans: Transfer visual prompt generator across LLMs. Advances in Neural Information Processing Systems, 36.
  46. Zhang, K.; and Liu, D. 2023. Customized segment any- thing model for medical image segmentation. arXiv preprint arXiv:2304.13785. · arXiv
  47. Zhang, X.; Wu, C.; Zhao, Z.; Lin, W.; Zhang, Y.; Wang, Y.; and Xie, W. 2023. Pmc-vqa: Visual instruction tun- ing for medical visual question answering. arXiv preprint arXiv:2305.10415. · arXiv
  48. Zhu, B.; Jin, P.; Ning, M.; Lin, B.; Huang, J.; Song, Q.; Pan, M.; and Yuan, L. 2024. LLMBind: A Unified Modality-Task Integration Framework. arXiv preprint arXiv:2402.14891. · arXiv
  49. Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understand- ing with advanced large language models. arXiv preprint arXiv:2304.10592. · arXiv
  50. Zou, X.; Yang, J.; Zhang, H.; Li, F.; Li, L.; Wang, J.; Wang, L.; Gao, J.; and Lee, Y. J. 2024. Segment everything every- where all at once. Advances in Neural Information Process- ing Systems, 36. References
  51. Ben Abacha, A.; Sarrouti, M.; Demner-Fushman, D.; Hasan, S. A.; and Müller, H. 2021. Overview of the vqa-med task at imageclef 2021: Visual question answering and gen- eration in the medical domain. In Proceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum- working notes. 21-24 September 2021.
  52. Cheng, J.; Ye, J.; Deng, Z.; Chen, J.; Li, T.; Wang, H.; Su, Y.; Huang, Z.; Chen, J.; Jiang, L.; et al. 2023. Sam-med2d. arXiv preprint arXiv:2308.16184. · arXiv
  53. Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P. N.; and Hoi, S. 2024. Instructblip: Towards general-purpose vision-language models with in- struction tuning. Advances in Neural Information Process- ing Systems, 36.
  54. He, X. 2021. Towards Visual Question Answering on Pathology Images. In Proceedings of the 59th annual meet- ing of the association for computational linguistics and the 11th international joint conference on natural language pro- cessing, volume 2.
  55. Hendrycks, D.; and Gimpel, K. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. · arXiv
  56. Hu, Y.; Li, T.; Lu, Q.; Shao, W.; He, J.; Qiao, Y.; and Luo, P. 2024. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22170-22183.
  57. Hu, Y.; Wang, Q.; Shao, W.; Xie, E.; Li, Z.; Han, J.; and Luo, P. 2023. Beyond one-to-one: Rethinking the referring image segmentation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, 4067-4077.
  58. Huang, X.; Li, H.; Cao, M.; Chen, L.; You, C.; and An, D. 2024. Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation. arXiv preprint arXiv:2404.02845. · arXiv
  59. Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2024. Lisa: Reasoning segmentation via large lan- guage model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9579-9589.
  60. Lau, J. J.; Gayen, S.; Ben Abacha, A.; and Demner- Fushman, D. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1): 1-10.
  61. Li, C.; Wong, C.; Zhang, S.; Usuyama, N.; Liu, H.; Yang, J.; Naumann, T.; Poon, H.; and Gao, J. 2024. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Sys- tems, 36.
  62. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen im- age encoders and large language models. In International conference on machine learning, 19730-19742.
  63. PMLR.
  64. Li, Z.; Li, Y.; Li, Q.; Wang, P.; Guo, D.; Lu, L.; Jin, D.; Zhang, Y.; and Hong, Q. 2023b. Lvit: language meets vision transformer in medical image segmentation. IEEE transac- tions on medical imaging.
  65. Liu, B.; Zhan, L.-M.; Xu, L.; Ma, L.; Yang, Y.; and Wu, X.-M. 2021. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 1650-1654. IEEE.
  66. Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024. Visual in- struction tuning. Advances in neural information processing systems, 36.
  67. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, 8748-8763.
  68. PMLR.
  69. Wu, C.; Zhang, X.; Zhang, Y.; Wang, Y.; and Xie, W. 2023. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463. · arXiv
  70. Yang, Z.; Wang, J.; Tang, Y.; Chen, K.; Zhao, H.; and Torr, P. H. 2022. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18155-18165.
  71. Zhang, X.; Wu, C.; Zhao, Z.; Lin, W.; Zhang, Y.; Wang, Y.; and Xie, W. 2023. Pmc-vqa: Visual instruction tun- ing for medical visual question answering. arXiv preprint arXiv:2305.10415. · arXiv

Cited by

Sorted by citations

Authors
Huang, X., Shen, L., Liu, J., Shang, F., Li, H., Huang, H., & Yang, Y.
Affiliations
  • China Agricultural University (CN)
  • Chinese Academy of Sciences (CN)
  • Baidu (China) (CN)
  • Peking University (CN)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2025-04-11
Volume
39
Issue
4
Pages
3779-3787
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v39i4.32394
OpenAlex
W4409368518
Cited by
14
References
0
Citations by year
2026: 11 · 2025: 3
FWCI
10.58
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
multimodal large language models, visual question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review