Mapping visual features to semantic profiles for retrieval in medical imaging
Abstract
Content based image retrieval is highly relevant in medical imaging, since it makes vast amounts of imaging data accessible for comparison during diagnosis. Finding image similarity measures that reflect diagnostically relevant relationships is challenging, since the overall appearance variability is high compared to often subtle signatures of diseases. To learn models that capture the relationship between semantic clinical information and image elements at scale, we have to rely on data generated during clinical routine (images and radiology reports), since expert annotation is prohibitively costly. Here we show that re-mapping visual features extracted from medical imaging data based on weak labels that can be found in corresponding radiology reports creates descriptions of local image content capturing clinically relevant information. We show that these semantic profiles enable higher recall and precision during retrieval compared to visual features, and that we can even map semantic terms describing clinical findings from radiology reports to localized image volume areas.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Content-based image retrieval at the end of the early years
Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895972 · Cited by 6,050 · see citing works
Statistical and structural approaches to texture
Haralick, R. M. (1979) · Proceedings of the IEEE · doi:10.1109/proc.1979.11328 · Cited by 5,769 · see citing works
Semantic texton forests for image categorization and segmentation
Shotton, J., Johnson, M., & Cipolla, R. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587503 · Cited by 1,027 · see citing works
Fast Keypoint Recognition Using Random Ferns
Özuysal, M., Calonder, M., Lepetit, V., & Fua, P. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.23 · Cited by 594 · see citing works
Tree Induction for Probability-Based Ranking
Provost, F., & Domingos, P. (2003) · Machine Learning · doi:10.1023/a:1024099825458 · Cited by 541 · see citing works
RadLex: A New Method for Indexing Online Educational Materials
Langlotz, C. P. (2006) · Radiographics · doi:10.1148/rg.266065168 · Cited by 440 · see citing works
Automatic Attribute Discovery and Characterization from Noisy Web Data
Berg, T. L., Berg, A. C., & Shih, J. J. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15549-9_48 · Cited by 418 · see citing works
Fast Keypoint Recognition in Ten Lines of Code
Özuysal, M., Fua, P., & Lepetit, V. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383123 · Cited by 415 · see citing works
Unsupervised feature selection applied to content-based retrieval of lung images
Dy, J., Brodley, C. E., Kak, A. C., Broderick, L. S., & Aisen, A. M. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1182100 · Cited by 255 · see citing works
MDCT-based 3-D texture classification of emphysema and early smoking related lung pathologies
Xu, Y., Sonka, M., McLennan, G., et al. (2006) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2006.870889 · Cited by 235 · see citing works
Joint multi-label multi-instance learning for image classification
Zha, Z.-J., Hua, X., Mei, T., Wang, J., Qi, G.-J., & Wang, Z. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587384 · Cited by 222 · see citing works
Multiple Instance Metric Learning from Automatically Labeled Bags of Faces
Guillaumin, M., Verbeek, J. J., & Schmid, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15549-9_46 · Cited by 182 · see citing works
MIForests: Multiple-Instance Learning with Randomized Trees
Leistner, C., Saffari, A., & Bischof, H. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15567-3_3 · Cited by 141 · see citing works
Multiple instance learning with generalized support vector machines
Andrews, S. J., Hofmann, T. F., & Tsochantaridis, I. (2002) · National Conference on Artificial Intelligence · doi:10.5555/777092.777234 · Cited by 88 · see citing works
Near-Affine-Invariant Texture Learning for Lung Tissue Analysis Using Isotropic Wavelet Frames
Depeursinge, A., Ville, D. V. D., Platon, A., Geissbühler, A., Poletti, P., & Müller, H. (2012) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2012.2198829 · Cited by 83 · see citing works
Learning a distance metric from multi-instance multi-label data
Jin, R., Wang, S., & Zhou, Z. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206684 · Cited by 79 · see citing works
Concurrent Multiple Instance Learning for Image Categorization
Qi, G.-J., Hua, X., Rui, Y., Mei, T., Tang, J., & Zhang, H.-J. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383152 · Cited by 49 · see citing works
Content-based image retrieval for pulmonary computed tomography nodule images
Lam, M., Disney, T., Pham, M., et al. (2007) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.710297 · Cited by 44 · see citing works
Bartholmai, B. J., Karwoski, R. A., Zavaletta, V. A., Robb, R. A., & Holmes, D. (2006) · The Insight Journal · doi:10.54294/hzdcno · Cited by 27 · see citing works
Texture Bags: Anomaly Retrieval in Medical Images Based on Local 3D-Texture Similarity
Burner, A., Donner, R., Mayerhoefer, M. E., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-28460-1_11 · Cited by 24 · see citing works
Randomized Locality Sensitive Vocabularies for Bag-of-Features Model
Mu, Y., Sun, J., Han, T. X., Cheong, L., & Yan, S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15558-1_54 · Cited by 22 · see citing works
Over-segmentation of 3D medical image volumes based on monogenic cues
Holzer, M. (2014) · reposiTUm (TU Wien) · doi:10.34726/hss.2014.25724 · Cited by 13 · see citing works
Content-based image retrieval for interstitial lung diseases
Dash, J. K., Khandelwal, N. K., Gupta, R., et al. (2012) · doi:10.1109/ispcc.2012.6224347 · Cited by 8 · see citing works
Depeursinge, A. (2010) · Archive ouverte UNIGE (University of Geneva) · doi:10.13097/archive-ouverte/unige:6550 · Cited by 5 · see citing works
Building Implicit Dictionaries Based on Extreme Random Clustering for Modality Recognition
Pauly, O., Mateus, D., & Navab, N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-28460-1_5 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759 · see citing works
“…to existing hand-crafted image features, for Chest X-rays [10] and CT lung nodule identification [9], [12]; and 3) performing unsupervised pre-training on natural or medical images and fine-tuning on medical target images using CNN or other types of deep learning models [18], [19], [20], [21].”
MethodBecker, A. S., Mueller, M., Stoffel, E., Marcon, M., Ghafoor, S., & Boss, A. (2018) · The British Journal of Radiology · doi:10.1259/bjr.20170576 · Cited by 157 · see citing works
Huo, T., Li, L., Chen, X., Wang, Z., Zhang, X., Liu, S., Huang, J., Zhang, J., Yang, Q., Wu, W., et al. (2023) · Scientific Reports · doi:10.1038/s41598-022-26771-1 · Cited by 38 · see citing works
- Authors
- Hofmanninger, J., & Langs, G.
- Affiliations
- Medical University of Vienna (AT)
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 457-465
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1901943601
- Cited by
- 38
- References
- 25
- Citations by year
- 2024: 2 · 2023: 2 · 2022: 6 · 2021: 5 · 2020: 5 · 2019: 4 · 2018: 1 · 2017: 8 · 2016: 5
- FWCI
- 1.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- content-based image retrieval, diagnostic imaging, semantic profiles, radiology reports, weak labels, image similarity measures, clinical features, radiology report annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
