CoChat
Sign Up
PreprintCited by 20Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationImage Retrieval and Classification Techniques

The DOI matches its DataCite record.

Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding

Open access
Zou, Ke, Bai, Yang, Liu, Bo, Chen, Yidi, Chen, Zhihao, Zhou, Yang, Yuan, Xuedong, Wang, Meng, Shen, Xiaojing, Cao, Xiaochun, Tham, Yih Chung, Fu, Huazhu·2024·arXiv
DOI 10.48550/arxiv.2404.06798arXiv 2404.06798
Open this paper in CoChat — free accountOpen PDF

Abstract

Medical phrase grounding is crucial for identifying relevant regions in medical images based on phrase queries, facilitating accurate image analysis and diagnosis. However, current methods rely on manual extraction of key phrases from medical reports, reducing efficiency and increasing the workload for clinicians. Additionally, the lack of model confidence estimation limits clinical trust and usability. In this paper, we introduce a novel task called Medical Report Grounding (MRG), which aims to directly identify diagnostic phrases and their corresponding grounding boxes from medical reports in an end-to-end manner. To address this challenge, we propose uMedGround, a robust and reliable framework that leverages a multimodal large language model to predict diagnostic phrases by embedding a unique token, , into the vocabulary to enhance detection capabilities. A vision encoder-decoder processes the embedded token and input image to generate grounding boxes. Critically, uMedGround incorporates an uncertainty-aware prediction model, significantly improving the robustness and reliability of grounding predictions. Experimental results demonstrate that uMedGround outperforms state-of-the-art medical phrase grounding methods and fine-tuned large visual-language models, validating its effectiveness and reliability. This study represents a pioneering exploration of the MRG task, marking the first-ever endeavor in this domain. Additionally, we demonstrate the applicability of uMedGround in medical visual question answering and class-based localization tasks, where it highlights visual evidence aligned with key diagnostic phrases, supporting clinicians in interpreting various types of textual inputs, including free-text reports, visual question answering queries, and class labels.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.06798

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 55

  1. Z. Chen, Y. Zhou, A. Tran, J. Zhao, and et al., "Medical phrase grounding with region-phrase context contrastive alignment," in Medical Image Computing and Computer Assisted Intervention -MICCAI 2023. Cham: Springer Nature Switzerland, 2023, pp. 371-381.
  2. J. Deng, Z. Yang, T. Chen, W. Zhou, and H. Li, "Transvg: End-to-end visual grounding with transformers," in ICCV, 2021, pp. 1769-1779.
  3. Y. Du, Z. Fu, Q. Liu, and Y. Wang, "Visual grounding with transformers," in ICME. IEEE, 2022, pp. 1-6.
  4. C. Zhu, Y. Zhou, Y. Shen, G. Luo, X. Pan, M. Lin, C. Chen, L. Cao, X. Sun, and R. Ji, "Seqtr: A simple yet universal network for visual grounding," in ECCV. Springer, 2022, pp. 598-615.
  5. L. Zhou, Z. Zhou, K. Mao, and Z. He, "Joint visual grounding and tracking with natural language specifica- tion," in CVPR, 2023, pp. 23 151-23 160.
  6. R. He, P. Cascante-Bonilla, Z. Yang, A. C. Berg, and V. Ordonez, "Improved visual grounding through self- consistent explanations," in CVPR, 2024, pp. 13 095- 13 105.
  7. A. Ichinose, T. Hatsutani, K. Nakamura, Y. Kitamura, S. Iizuka, E. Simo-Serra, S. Kido, and N. Tomiyama, "Visual grounding of whole radiology reports for 3d ct images," in MICCAI. Springer, 2023, pp. 611-621.
  8. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  9. H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning," NeurIPS, vol. 36, 2024.
  10. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu et al., "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks," in CVPR, 2024, pp. 24 185-24 198.
  11. Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma et al., "How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites," arXiv preprint arXiv:2404.16821, 2024. · arXiv
  12. T. Nair, D. Precup, D. L. Arnold, and T. Arbel, "Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation," Medical image analysis, vol. 59, p. 101557, 2020.
  13. L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng, "Uncertainty-aware self-ensembling model for semi- supervised 3d left atrium segmentation," in MICCAI. Springer, 2019, pp. 605-613.
  14. G. Wang, W. Li, S. Ourselin, and T. Vercauteren, "Auto- matic brain tumor segmentation using convolutional neural networks with test-time augmentation," in International MICCAI Brainlesion Workshop. Springer, 2018, pp. 61-72.
  15. L. Huang, S. Ruan, P. Decazes, and T. Denoeux, "Lym- phoma segmentation from 3d pet-ct images using a deep evidential network," International Journal of Approximate Reasoning, vol. 149, pp. 39-60, 2022.
  16. K. Zou, X. Yuan, X. Shen, M. Wang, and H. Fu, "Tbrats: Trusted brain tumor segmentation," in MICCAI. Springer, 2022, pp. 503-513.
  17. M. Wang, T. Lin, L. Wang, A. Lin, K. Zou, X. Xu, Y. Zhou, Y. Peng, Q. Meng, Y. Qian et al., "Uncertainty- inspired open set learning for retinal anomaly identifica- tion," Nature Communications, vol. 14, no. 1, p. 6757, 2023.
  18. M. Sensoy, L. Kaplan, and M. Kandemir, "Evidential deep learning to quantify classification uncertainty," in NeurIPS, 2018, pp. 3183-3193.
  19. J. Mukhoti, J. van Amersfoort, P. H. Torr, and Y. Gal, "Deep deterministic uncertainty for semantic segmenta- tion," in International Conference on Machine Learning Workshop on Uncertainty and Robustness in Deep Learn- ing, 2021.
  20. J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal, "Uncertainty estimation using a single deep deterministic neural network," in ICML. PMLR, 2020, pp. 9690-9700.
  21. B. A. Plummer, P. Kordas, M. H. Kiapour, S. Zheng, R. Piramuthu, and S. Lazebnik, "Conditional image-text embedding networks," in ECCV, 2018, pp. 249-264.
  22. J. Wang and L. Specia, "Phrase localization without paired training examples," in ICCV, 2019, pp. 4663-4672.
  23. J. H. Moon, H. Lee, W. Shin, Y.-H. Kim, and E. Choi, "Multi-modal understanding and generation for medical images and text via vision-language pre-training," IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 12, pp. 6070-6080, 2022.
  24. F. Wang, Y. Zhou, S. Wang, V. Vardhanabhuti, and L. Yu, "Multi-granularity cross-modal alignment for generalized medical visual representation learning," NeurIPS, vol. 35, pp. 33 536-33 549, 2022.
  25. C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, "Medklip: Medical knowledge enhanced language-image pre-training," ICCV, 2023.
  26. B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Naumann, A. Nori, J. Alvarez-Valle, H. Poon, and O. Oktay, "Making the most of text semantics to improve biomedical vision- language processing," in ECCV, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds. Cham: Springer Nature Switzerland, 2022, pp. 1-21.
  27. K. Vilouras, P. Sanchez, A. Q. O'Neil, and S. A. Tsaftaris, "Zero-shot medical phrase grounding with off-the-shelf diffusion models," IEEE Journal of Biomedical and Health Informatics, 2024.
  28. S. Wang, Z. Zhao, X. Ouyang, Q. Wang, and D. Shen, "Chatcad: Interactive computer-aided diagnosis on medical image using large language models," arXiv preprint arXiv:2302.07257, 2023. · arXiv
  29. M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and P. Rajpurkar, "Med- flamingo: a multimodal medical few-shot learner," in Machine Learning for Health (ML4H). PMLR, 2023, pp. 353-367.
  30. C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, "Towards generalist foundation model for radiology," arXiv preprint arXiv:2308.02463, 2023. · arXiv
  31. W. Zhao, Z. Deng, S. Yadav, and P. S. Yu, "Heterogeneous knowledge grounding for medical question answering with retrieval augmented large language model," in Companion Proceedings of the ACM Web Conference 2024, 2024, pp. 1590-1594.
  32. Y. Gao, R. Li, E. Croxford, S. Tesch, D. To, J. Caskey, B. W. Patterson, M. M. Churpek, T. Miller, D. Dligach et al., "Large language models and medical knowledge grounding for diagnosis prediction," medRxiv, pp. 2023- 11, 2023.
  33. S. Bannur, K. Bouzid, D. C. Castro, A. Schwaighofer, S. Bond-Taylor, M. Ilse, F. Pérez-García, V. Salvatelli, H. Sharma, F. Meissen et al., "Maira-2: Grounded radiol- ogy report generation," arXiv preprint arXiv:2406.04449, 2024. · arXiv
  34. L. Huang, S. Ruan, Y. Xing, and M. Feng, "A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods," Medical Image Analysis, p. 103223, 2024.
  35. S. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. Maier-Hein, S. Eslami, D. Jimenez Rezende, and O. Ronneberger, "A probabilistic u-net for segmenta- tion of ambiguous images," NeurIPS, vol. 31, 2018.
  36. A. Mehrtash, W. M. Wells, C. M. Tempany, P. Abol- maesumi, and T. Kapur, "Confidence calibration and predictive uncertainty estimation for deep medical image segmentation," IEEE Transactions on Medical Imaging, vol. 39, no. 12, pp. 3868-3878, 2020.
  37. Y. Luo, Q. Yang, Y. Fan, H. Qi, and M. Xia, "Measure- ment guidance in diffusion models: Insight from medical image synthesis," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
  38. L. Huang, S. Ruan, P. Decazes, and T. Denœux, "Deep evidential fusion with uncertainty quantification and reliability learning for multimodal medical image segmen- tation," Information Fusion, vol. 113, p. 102648, 2025.
  39. X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia, "Lisa: Reasoning segmentation via large language model," arXiv preprint arXiv:2308.00692, 2023. · arXiv
  40. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., "Lora: Low-rank adaptation of large language models." ICLR, vol. 1, no. 2, p. 3, 2022.
  41. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., "Segment anything," arXiv preprint arXiv:2304.02643, 2023. · arXiv
  42. Q. Du, V. Faber, and M. Gunzburger, "Centroidal voronoi tessellations: Applications and algorithms," SIAM review, vol. 41, no. 4, pp. 637-676, 1999.
  43. R. Girshick, "Fast r-cnn," in ICCV, 2015, pp. 1440-1448.
  44. H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, "Generalized intersection over union: A metric and a loss for bounding box regression," in CVPR, 2019, pp. 658-666.
  45. C. Rupprecht, I. Laina, R. DiPietro, M. Baust, F. Tombari, N. Navab, and G. D. Hager, "Learning in an uncertain world: Representing ambiguity through multiple hypothe- ses," in ICCV, 2017, pp. 3591-3600.
  46. A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng, "Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports," Scientific Data, vol. 6, no. 1, p. 317, 2019.
  47. X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, "Chestx-ray8: Hospital-scale chest x- ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases," in CVPR, 2017, pp. 2097-2106.
  48. X. Hu, L. Gu, Q. An, M. Zhang, L. Liu, K. Kobayashi, T. Harada, R. M. Summers, and Y. Zhu, "Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering," in ACM KDD, 2023, pp. 4156-4165.
  49. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," in Proceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171-4186.
  50. S. Jain, A. Agrawal, A. Saporta, S. Q. Truong, D. N. Duong, T. Bui, P. Chambon, Y. Zhang, M. P. Lungren, A. Y. Ng et al., "Radgraph: Extracting clinical entities and relations from radiology reports," arXiv preprint arXiv:2106.14463, 2021. · arXiv
  51. A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y. Ng, and M. P. Lungren, "Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert," arXiv preprint arXiv:2004.09167, 2020. · arXiv
  52. L. Muchen and S. Leonid, "Referring transformer: A one- step approach to multi-task visual grounding," in NeurIPS, 2021.
  53. H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, "Capabilities of gpt-4 on medical challenge problems," arXiv preprint arXiv:2303.13375, 2023. · arXiv
  54. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., "Llama: Open and efficient foundation lan- guage models," arXiv preprint arXiv:2302.13971, 2023. · arXiv
  55. J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, "Segment anything in medical images," Nature Communi- cations, vol. 15, no. 1, p. 654, 2024.

Cited by

Sorted by citations

Metadata

Authors
Zou, K., Bai, Y., Liu, B., Chen, Y., Chen, Z., Zhou, Y., Yuan, X., Wang, M., Shen, X., Cao, X., Tham, Y. C., & Fu, H.
Affiliations
  • Sichuan University (CN)
  • Agency for Science, Technology and Research (SG)
  • Hong Kong Polytechnic University (HK)
  • Tianjin University (CN)
  • Sun Yat-sen University (CN)
  • National University of Singapore (SG)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
47
Issue
12
Pages
11315-11329
ISSN
0162-8828 (ISSN-L)
DOI
10.48550/arxiv.2404.06798
arXiv
2404.06798
OpenAlex
W4413050402
Cited by
20
References
0
Citations by year
2026: 18 · 2025: 2
FWCI
7.02
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Research Foundation
  • Agency for Science, Technology and Research
  • National Natural Science Foundation of China (62025604, 62411540034)
  • China Scholarship Council (202206240082)
  • Department of Science and Technology of Sichuan Province (2023YFG0273, 2022YFS0071)
  • Science and Technology Department of Guangxi Zhuang Autonomous (2025GXNSFAA069531)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
uncertainty estimation, multimodal large language models, medical visual question answering
MeSH terms
Algorithms, Humans, Image Interpretation, Computer-Assisted, Natural Language Processing, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review