CoChat
Sign Up
PreprintCited by 2Multimodal Machine Learning ApplicationsTopic ModelingHandwritten Text Recognition Techniques

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

CMRAG: Co-modality-based visual document retrieval and question answering

Open access
Chen, Wang, Yu, Wenhan, Qi, Guanqiang, Li, Weikang, Li, Yang, Sha, Lei, Xia, Deguo, Huang, Jizhou·2025·arXiv
DOI 10.48550/arxiv.2509.02123arXiv 2509.02123
Open this paper in CoChat — free accountOpen PDF

Abstract

Retrieval-Augmented Generation (RAG) has become a core paradigm in document question answering tasks. However, existing methods have limitations when dealing with multimodal documents: one category of methods relies on layout analysis and text extraction, which can only utilize explicit text information and struggle to capture images or unstructured content; the other category treats document segmentation as visual input and directly passes it to visual language models (VLMs) for processing, yet it ignores the semantic advantages of text, leading to suboptimal retrieval and generation results. To address these research gaps, we propose the Co-Modality-based RAG (CMRAG) framework, which can simultaneously leverage texts and images for more accurate retrieval and generation. Our framework includes two key components: (1) a Unified Encoding Model (UEM) that projects queries, parsed text, and images into a shared embedding space via triplet-based training, and (2) a Unified Co-Modality-informed Retrieval (UCMR) method that statistically normalizes similarity scores to effectively fuse cross-modal signals. To support research in this direction, we further construct and release a large-scale triplet dataset of (query, text, image) examples. Experiments demonstrate that our proposed framework consistently outperforms single-modality--based RAG in multiple visual document question-answering (VDQA) benchmarks. The findings of this paper show that integrating co-modality information into the RAG framework in a unified manner is an effective approach to improving the performance of complex VDQA systems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2509.02123

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 71

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. Omar Adjali, Olivier Ferret, Sahar Ghannay, and Hervé Le Borgne. Multi-level information retrieval augmented generation for knowledge-based visual question answering. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16499-16513. Association for Computational Linguistics, 2024.
  3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
  4. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pp. 5803-5812, 2017.
  5. Md Adnan Arefeen, Biplob Debnath, Md Yusuf Sarwar Uddin, and Srimat Chakradhar. irag: Advancing rag for videos with an incremental approach. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 4341-4348, 2024.
  6. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. · arXiv
  7. Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activ- itynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
  8. Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1818-1826, 2024.
  9. Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16495-16504, 2022.
  10. Wang Chen, Guanqiang Qi, Weikang Li, Yang Li, Deguo Xia, and Jizhou Huang. Pairs: Parametric-verified adaptive information retrieval and selection for efficient rag. arXiv preprint arXiv:2508.04057, 2025. · arXiv
  11. Wang Chen, Guanqiang Qi, Weikang Li, Yang Li, Deguo Xia, and Jizhou Huang. Decide then retrieve: A training-free framework with uncertainty-guided triggering and dual-path retrieval. arXiv preprint arXiv:2601.03908, 2026. · arXiv
  12. Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5558-5570, 2022.
  13. Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? arXiv preprint arXiv:2302.11713, 2023. · arXiv
  14. Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi- modal retrieval is what you need for multi-page multi-document understanding. arXiv preprint arXiv:2411.04952, 2024. · arXiv
  15. Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long doc- ument benchmark integrating understanding, reasoning, and locating. arXiv preprint arXiv:2412.18424, 2024. · arXiv
  16. Yuyang Dong, Nobuhiro Ueda, KrisztiĂAn Boros, Daiki Ito, Takuya Sera, and Masafumi Oyamada. Scan: Semantic document layout analysis for textual and visual retrieval- augmented generation. arXiv preprint arXiv:2505.14381, 2025. · arXiv
  17. Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chen- lin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025. · arXiv
  18. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDE- LOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025.
  19. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1), 2023. · arXiv
  20. Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021. · arXiv
  21. Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214, 2024. · arXiv
  22. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. · arXiv
  23. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
  24. Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng. Mrag-bench: Vision-centric evaluation for retrieval-augmented multimodal models. In The Thirteenth International Conference on Learning Representations, 2025.
  25. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM international conference on multimedia, pp. 4083-4091, 2022.
  26. Yulong Hui, Yao Lu, and Huanchen Zhang. Uda: A benchmark suite for retrieval augmented generation in real-world document analysis. Advances in Neural Information Processing Systems, 37:67200-67217, 2024.
  27. Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. Videorag: Retrieval- augmented generation over video corpus. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 21278-21298, 2025.
  28. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
  29. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781, 2020.
  30. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision, pp. 498-517. Springer, 2022.
  31. Reno Kriz, Kate Sanders, David Etter, Kenton Murray, Cameron Carpenter, Hannah Rec- knor, Jimena Guallar-Blasco, Alexander Martin, Eugene Yang, and Benjamin Van Durme. Multivent 2.0: A massive multilingual benchmark for event-centric video retrieval. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pp. 24149-24158, 2025.
  32. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020.
  33. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision- language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14369-14387, 2024.
  34. Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Fei Huang, Jingren Zhou, et al. Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent. In The Thirteenth International Conference on Learning Representations, 2025.
  35. Weizhe Lin and Bill Byrne. Retrieval augmented visual question answering with outside knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 11238-11254, 2022.
  36. Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 36:22820-22840, 2023.
  37. Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093, 2024. · arXiv
  38. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6492-6505, 2024a.
  39. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963-96010, 2024b.
  40. Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. Drvideo: Document retrieval based long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18936-18946, 2025.
  41. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
  42. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, 2022.
  43. Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3113-3124, 2023.
  44. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reason- ing over scientific plots. In Proceedings of the ieee/cvf winter conference on applications of computer vision, pp. 1527-1536, 2020.
  45. Arnau Perez and Xavier Vizcaino. Advanced ingestion process powered by llm parsing for rag system. arXiv preprint arXiv:2412.15262, 2024. · arXiv
  46. Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, and Lifu Huang. Rora-vlm: Robust retrieval-augmented vision language models. arXiv preprint arXiv:2410.08876, 2024a. · arXiv
  47. Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. Long2rag: Evaluating long-context & long-form retrieval-augmented generation with key point recall. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4852- 4872, 2024b.
  48. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
  49. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316-1331, 2023.
  50. Arun Reddy, Alexander Martin, Eugene Yang, Andrew Yates, Kate Sanders, Kenton Mur- ray, Reno Kriz, Celso M de Melo, Benjamin Van Durme, and Rama Chellappa. Video- colbert: Contextualized late interaction for text-to-video retrieval. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19691-19701, 2025.
  51. Monica Riedler and Stefan Langer. Beyond text: Optimizing rag with multimodal inputs for industrial applications. arXiv preprint arXiv:2410.21943, 2024. · arXiv
  52. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
  53. Ray Smith. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pp. 629-633. IEEE, 2007.
  54. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: complex question answering over text, tables and images. In International Conference on Learning Repre- sentations, 2021.
  55. Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13636- 13645, 2023.
  56. Yang Tian, Fan Liu, Jingyuan Zhang, V. W., Yupeng Hu, and Liqiang Nie. CoRe-MMRAG: Cross-source knowledge reconciliation for multimodal RAG. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32967-32982, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.1583. URL https://aclanthology.org/ 2025.acl-long.1583/. · DOI
  57. David Wan, Han Wang, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Clamr: Contextualized late-interaction for multimodal content retrieval. arXiv preprint arXiv:2506.06144, 2025. · arXiv
  58. Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents. arXiv preprint arXiv:2502.18017, 2025a. · arXiv
  59. Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, and Feng Zhao. Vrag-rl: Empower vision-perception-based rag for visually rich information understanding via iterative reasoning with reinforcement learning. arXiv preprint arXiv:2505.22019, 2025b. · arXiv
  60. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4581- 4591, 2019.
  61. Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barze- lay, and Leonid Karlinsky. Real-mm-rag: A real-world multi-modal retrieval benchmark. arXiv preprint arXiv:2502.12342, 2025. · arXiv
  62. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
  63. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1192-1200, 2020.
  64. Yibin Yan and Weidi Xie. Echosight: Advancing visual-language models with wiki knowl- edge. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1538-1551, 2024.
  65. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. · arXiv
  66. Jeff Yang, Duy-Khanh Vu, Minh-Tien Nguyen, Xuan-Quang Nguyen, Linh Nguyen, and Hung Le. Superrag: Beyond rag with layout-aware graph modeling. arXiv preprint arXiv:2503.04790, 2025b. · arXiv
  67. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. In The Thirteenth International Conference on Learning Representations, 2025a.
  68. Wenhan Yu, Wang Chen, Guanqiang Qi, Weikang Li, Yang Li, Lei Sha, Deguo Xia, and Jizhou Huang. Bbox docvqa: A large scale bounding box grounded dataset for enhancing reasoning in document visual question answer. arXiv preprint arXiv:2511.15090, 2025b. · arXiv
  69. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
  70. Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, and Kyusong Lee. Omagent: A multi- modal agent framework for complex video understanding with task divide-and-conquer. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10031-10045, 2024a.
  71. Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Yuxuan Zhao, Zehua Xie, et al. mr 2 ag: Multimodal retrieval-reflection- augmented generation for knowledge-based vqa. arXiv preprint arXiv:2411.15041, 2024b. Model Metric MMLongBench Finreport Finslides Techreport Techslides LongDocURL BGE Recall@1 8.45% 38.45% 21.96% 32.92% 55.02% 2.53% Recall@5 22.84% 64.95% 46.01% 55.41% 77.62% 8.97% Recall@10 34.50% 73.97% 62.45% 65.69% 85.38% 15.25% nDCG@1 13.31% 38.45% 21.96% 32.92% 55.02% 6.23% nDCG@5 18.51% 52.56% 34.23% 44.63% 67.37% 7.35% nDCG@10 22.96% 55.46% 39.55% 47.95% 69.92% 9.92% MRR@1 13.31% 38.45% 21.96% 32.92% 55.02% 6.23% MRR@5 21.19% 48.43% 30.36% 41.07% 63.93% 10.15% MRR@10 23.29% 49.62% 32.55% 42.44% 65.01% 11.69% CLIP-B/32 Recall@1 12.31% 3.17% 11.88% 1.47% 31.09% 7.81% Recall@5 33.08% 10.20% 31.65% 8.50% 61.08% 23.17% Recall@10 47.89% 18.99% 49.71% 16.69% 75.85% 34.18% nDCG@1 17.93% 3.17% 11.88% 1.47% 31.09% 11.82% nDCG@5 25.74% 6.50% 21.56% 4.74% 46.57% 17.92% nDCG@10 30.98% 9.31% 27.39% 7.35% 51.32% 22.04% MRR@1 17.93% 3.17% 11.88% 1.47% 31.09% 11.82% MRR@5 26.60% 5.31% 18.28% 3.54% 41.79% 19.55% MRR@10 28.61% 6.45% 20.68% 4.59% 43.73% 21.24% CLIP-L/14-336 Recall@1 23.87% 26.26% 43.16% 26.12% 50.89% 25.81% Recall@5 46.01% 52.17% 77.19% 50.46% 79.39% 53.74% Recall@10 57.42% 65.77% 89.26% 62.98% 88.85% 66.47% nDCG@1 31.15% 26.26% 43.16% 26.12% 50.89% 37.53% nDCG@5 39.07% 39.85% 61.64% 38.98% 66.35% 45.26% nDCG@10 43.26% 44.27% 65.58% 42.99% 69.42% 50.08% MRR@1 31.15% 26.26% 43.16% 26.12% 50.89% 37.53% MRR@5 39.83% 35.77% 56.44% 35.17% 61.97% 48.66% MRR@10 41.17% 37.61% 58.08% 36.80% 63.25% 50.27% SigLIP Recall@1 28.31% 27.43% 49.05% 33.62% 61.60% 31.13% Recall@5 53.33% 55.10% 80.13% 61.67% 86.26% 62.34% Recall@10 64.12% 67.76% 90.68% 73.26% 92.84% 73.78% nDCG@1 37.43% 27.43% 49.05% 33.62% 61.60% 44.55% nDCG@5 45.78% 41.93% 65.82% 48.26% 74.82% 53.25% nDCG@10 49.67% 46.06% 69.23% 52.01% 76.95% 57.62% MRR@1 37.43% 27.43% 49.05% 33.62% 61.60% 44.55% MRR@5 46.17% 37.58% 61.04% 43.82% 71.00% 56.52% MRR@10 47.32% 39.29% 62.45% 45.37% 71.88% 57.74% SigLIP2 Recall@1 23.46% 0.23% 1.33% 1.16% 1.55% 1.37% Recall@5 47.32% 2.11% 8.46% 4.87% 11.60% 4.99% Recall@10 59.54% 6.45% 22.15% 10.05% 28.43% 11.22% nDCG@1 31.98% 0.23% 1.33% 1.16% 1.55% 2.60% nDCG@5 40.01% 1.22% 4.45% 2.97% 6.30% 3.84% nDCG@10 44.39% 2.57% 8.80% 4.63% 11.68% 6.07% MRR@1 31.98% 0.23% 1.33% 1.16% 1.55% 2.60% MRR@5 40.67% 0.93% 3.19% 2.35% 4.60% 4.31% MRR@10 42.02% 1.45% 4.93% 3.03% 6.78% 5.43% SigLIP + BGE Recall@1 28.31% 46.54% 54.56% 42.97% 69.94% 36.50% Recall@5 53.83% 72.45% 86.60% 71.72% 93.13% 65.80% Recall@10 64.12% 83.00% 92.78% 81.61% 97.05% 77.20% nDCG@1 37.43% 46.54% 54.56% 42.97% 69.94% 53.60% nDCG@5 45.99% 60.21% 72.04% 57.99% 82.48% 58.40% nDCG@10 49.67% 63.56% 74.00% 61.15% 83.75% 62.70% MRR@1 37.43% 46.54% 54.56% 42.97% 69.94% 53.60% MRR@5 46.45% 56.25% 67.15% 53.65% 78.92% 63.70% MRR@10 47.48% 57.49% 67.97% 54.94% 79.44% 64.90% CMRAG-R(ours) Recall@1 28.95% 30.48% 54.56% 34.62% 67.65% 31.30% Recall@5 53.50% 58.03% 86.60% 64.30% 91.95% 62.60% Recall@10 64.12% 70.81% 93.25% 76.28% 97.49% 74.50% nDCG@1 38.17% 30.48% 54.56% 34.62% 67.70% 45.20% nDCG@5 49.75% 44.29% 72.04% 50.11% 81.00% 61.10% nDCG@10 52.10% 48.52% 74.00% 54.14% 82.80% 63.80% MRR@1 38.17% 30.48% 54.56% 34.62% 67.70% 45.20% MRR@5 46.50% 40.08% 67.15% 45.56% 77.30% 56.90% MRR@10 47.64% 41.85% 67.97% 47.22% 78.10% 58. · arXiv

Cited by

Sorted by citations

See all 2

Related

See all

Metadata

Authors
Chen, W., Yu, W., Qi, G., Li, W., Li, Y., Sha, L., Xia, D., & Huang, J.
Affiliations
  • University of Hong Kong (HK)
  • Beihang University (CN)
  • Baidu (China) (CN)
  • Peking University (CN)
ORCID
Xia, Deguo
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2509.02123
arXiv
2509.02123
OpenAlex
W7079618395
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
Keywords
retrieval-augmented generation, vision-language models, multimodal texts

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: CMRAG: Co-modality-based visual document retrieval and question answering. https://app.cochat.ai/papers/10.48550/arxiv.2509.02123
The paper
Chen, W., Yu, W., Qi, G., Li, W., Li, Y., Sha, L., Xia, D., & Huang, J. (2025). CMRAG: Co-modality-based visual document retrieval and question answering. arXiv. https://doi.org/10.48550/arxiv.2509.02123