The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
TransMed: Transformers Advance Multi-Modal Medical Image Classification
Open accessAbstract
Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it cannot deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model on two datasets, parotid gland tumors classification and knee injury classification. Combining our contributions, we achieve an improvement of 10.1% and 1.9% in average accuracy, respectively, outperforming other state-of-the-art CNN-based models. The results of the proposed method are promising and have tremendous potential to be applied to a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to multi-modal medical image classification.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
A Threshold Selection Method from Gray-Level Histograms
Otsu, N. (1979) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1979.4310076 · Cited by 43,398
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008
UNETR: Transformers for 3D Medical Image Segmentation
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00181 · Cited by 3,586 · cited 1×
“UNETR [35] utilizes a pure transformer as the encoder to effectively capture the multi-scale information.”
MethodUNETR: Transformers for 3D Medical Image Segmentation
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H., & Xu, D. (2021) · arXiv · doi:10.48550/arxiv.2103.10504 · Cited by 3,580
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498
MultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmentation
Ibtehaz, N., & Rahman, M. S. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.08.025 · Cited by 2,311
Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
Qiu, Z., Yao, T., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.590 · Cited by 1,874
Bottleneck Transformers for Visual Recognition
Srinivas, A., Lin, T.-Y., Parmar, N., Shlens, J., Abbeel, P., & Vaswani, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01625 · Cited by 1,385
3D MRI Brain Tumor Segmentation Using Autoencoder Regularization
Myronenko, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11726-9_28 · Cited by 1,283
Setio, A. A. A., Ciompi, F., Litjens, G., Gerke, P., Jacobs, C., van Riel, S. J., Wille, M. M. W., Naqibullah, M., Sanchez, C. I., & van Ginneken, B. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2536809 · Cited by 1,258
(2020) · International Conference on Machine Learning · Cited by 1,012
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
A review: Deep learning for medical image segmentation using multi-modality fusion
Zhou, T., Ruan, S., & Canu, S. (2019) · Array · doi:10.1016/j.array.2019.100004 · Cited by 656
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
Xie, Y., Zhang, J., Shen, C., & Xia, Y. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87199-4_16 · Cited by 653
Pérez-García, F., Sparks, R., & Ourselin, S. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106236 · Cited by 602
HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation
Dolz, J., Gopinath, K., Yuan, J., Lombaert, H., Desrosiers, C., & Ben Ayed, I. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2878669 · Cited by 580
Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge
Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., & Maier-Hein, K. H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-75238-9_25 · Cited by 527
Weakly Supervised Deep Learning for COVID-19 Infection Detection and Classification From CT Images
Hu, S., Gao, Y., Niu, Z., Jiang, Y., Li, L., Xiao, X., Wang, M., Fang, E. F., Menpes-Smith, W., Xia, J., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3005510 · Cited by 387
Self-citationZhu, Z., Zheng, M., Qi, G., Wang, D., & Xiang, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2898111 · Cited by 328
Fully convolutional networks for multi-modality isointense infant brain image segmentation
Nie, D., Wang, L., Gao, Y., & Shen, D. (2016) · 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2016.7493515 · Cited by 304 · cited 2×
“[16] fused final features from modality-specific paths to make final decisions.”
BackgroundMethodSelf-citationClinical Prognostic Factors in Malignant Parotid Gland Tumors
Lima, R. A., Tavares, M. R., Dias, F. L., Kligerman, J., Nascimento, M. F., Barbosa, M. M., Cernea, C. R., Soares, J. R., Santos, I. C., & Salviano, S. (2005) · Otolaryngology–Head and Neck Surgery · doi:10.1016/j.otohns.2005.08.001 · Cited by 224
Han, C., Rundo, L., Murao, K., Noguchi, T., Shimahara, Y., Milacski, Z. Á., Koshino, S., Sala, E., Nakayama, H., & Satoh, S. (2021) · BMC Bioinformatics · doi:10.1186/s12859-020-03936-1 · Cited by 207
Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation
Tseng, K.-L., Lin, Y.-L., Hsu, W., & Huang, C.-Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.398 · Cited by 190
Multi-Modality Medical Image Fusion Using Convolutional Neural Network and Contrast Pyramid
Wang, K., Zheng, M., Wei, H., Qi, G., & Li, Y. (2020) · Sensors · doi:10.3390/s20082169 · Cited by 142
Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries
Casamitjana, A., Catà, M., Muriana, I. S., Combalia, M., & Vilaplana, V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9 · Cited by 137
Wang, Y., Yue, W., Li, X., Liu, S., Guo, L., Xu, H., Zhang, H., & Yang, G. (2020) · IEEE Access · doi:10.1109/access.2020.2980290 · Cited by 99
Yang, G., Chen, J., Gao, Z., Li, S., Ni, H., Angelini, E. D., Wong, T., Mohiaddin, R. H., Nyktari, E. G., Wage, R., et al. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.02.005 · Cited by 98
van Dijk, L. V., Thor, M., Steenbakkers, R. J. H. M., Apte, A., Zhai, T., Borra, R., Noordzij, W., Estilo, C. L., Lee, N. Y., Langendijk, J. A., et al. (2018) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2018.06.012 · Cited by 94
Liu, Y., Sung, K., Yang, G., Mirak, S. A., Hosseiny, M., Azadikhah, A., Zhong, X., Reiter, R. E., Lee, Y., & Raman, S. S. (2019) · IEEE Access · doi:10.1109/access.2019.2952534 · Cited by 93
Guo, Z., Li, X., Huang, H., Guo, N., & Li, Q. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363717 · Cited by 92
MMFNet: A multi-modality MRI fusion network for segmentation of nasopharyngeal carcinoma
Chen, H., Qi, Y., Yin, Y., Li, T., Liu, X., Li, X., Gong, G., & Wang, L. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.002 · Cited by 92
Liu, Y., Yang, G., Hosseiny, M., Azadikhah, A., Mirak, S. A., Miao, Q., Raman, S. S., & Sung, K. (2020) · IEEE Access · doi:10.1109/access.2020.3017168 · Cited by 85
Li, M., Wang, C., Zhang, H., & Yang, G. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103728 · Cited by 63
Rundo, L., Militello, C., Tangherloni, A., Russo, G. I., Vitabile, S., Gilardi, M. C., & Mauri, G. (2017) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.22253 · Cited by 60
Wu, Y., Hatipoğlu, S., Alonso‐Álvarez, D., Gatehouse, P. D., Li, B., Gao, Y., Firmin, D. N., Keegan, J., & Yang, G. (2021) · Diagnostics · doi:10.3390/diagnostics11020346 · Cited by 29
Self-citationA mixture of views network with applications to multi-view medical imaging
Shachor, Y., Greenspan, H. K., & Goldberger, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.09.027 · Cited by 27
Deep RetinaNet for Dynamic Left Ventricle Detection in Multiview Echocardiography Classification
Yang, M., Xiao, X., Liu, Z., Sun, L., Guo, W., Cui, L., Sun, D., Zhang, P., & Yang, G. (2020) · Scientific Programming · doi:10.1155/2020/7025403 · Cited by 22
Improving MRI-based Knee Disorder Diagnosis with Pyramidal Feature Details
Dunnhofer, M., Martinel, N., & Micheloni, C. (2021) · Institutional Research Information System (University of Udine) · Cited by 12
Cited by
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,296
Alzubaidi, L., Bai, J., Al-Sabaawi, A., Santamaría, J., Albahri, A. S., Al-dabbagh, B. S. N., Fadhel, M. A., Manoufali, M., Zhang, J., Al-Timemy, A. H., et al. (2023) · Journal of Big Data · doi:10.1186/s40537-023-00727-2 · Cited by 907
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis
Dalmaz, O., Yurt, M., & Cukur, T. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3167808 · Cited by 597
Advances in medical image analysis with vision Transformers: A comprehensive review
Azad, R., Kazerouni, A., Heidari, M., Aghdam, E. K., Molaei, A., Jia, Y., Jose, A., Roy, R., & Merhof, D. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103000 · Cited by 485
Li, J., Chen, J., Tang, Y., Wang, C., Landman, B. A., & Zhou, S. K. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102762 · Cited by 389
Metadata
- Authors
- Dai, Y., Gao, Y., & Liu, F.
- Venue
- Diagnostics
- Publisher
- MDPI AG
- Published
- 2021-07-31
- Volume
- 11
- Issue
- 8
- Pages
- 1384
- ISSN
- 2075-4418 (ISSN-L)
- OpenAlex
- W3188404242
- Cited by
- 467
- References
- 51
- FWCI
- 22.66
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.