Cited by 354Multimodal Machine Learning ApplicationsAI in cancer detectionImage Retrieval and Classification Techniques

MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

Open access
Zhang, Zizhao, Xie, Yuanpu, Xing, Fuyong, McGough, Mason, Yang, Lin·2017·2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Abstract

The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.

Integrity signals

Registry

Crossref · 10.1109/cvpr.2017.378

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

2 of the 42 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L.
Affiliations
University of Florida (US)
Venue
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2017-07
Pages
3549-3557
ISSN
1063-6919 (ISSN-L)
OpenAlex
W2963409068
Cited by
354
References
42
Citations by year
2026: 10 · 2025: 30 · 2024: 43 · 2023: 40 · 2022: 49 · 2021: 49 · 2020: 49 · 2019: 48 · 2018: 34 · 2017: 2
FWCI
10.63
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
medical image diagnosis, computer-aided diagnosis, multimodal mapping, medical report generation, attention visualization, interpretability, image-text retrieval, language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review