The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Multimodal MR Synthesis via Modality-Invariant Latent Representation
Open accessAbstract
We propose a multi-input multi-output fully convolutional neural network model for MRI synthesis. The model is robust to missing data, as it benefits from, but does not require, additional input modalities. The model is trained end-to-end, and learns to embed all input modalities into a shared modality-invariant latent space. These latent representations are then combined into a single fused representation, which is transformed into the target output modality with a learnt decoder. We avoid the need for curriculum learning by exploiting the fact that the various input modalities are highly correlated. We also show that by incorporating information from segmentation masks the model can both decrease its error and generate data with synthetic lesions. We evaluate our model on the ISLES and BRATS data sets and demonstrate statistically significant improvements over state-of-the-art methods for single input tasks. This improvement increases further when multiple input modalities are used, demonstrating the benefits of learning a common latent space, again resulting in a statistically significant improvement over the current best method. Finally, we demonstrate our approach on non skull-stripped brain images, producing a statistically significant improvement over the previous best method. Code is made publicly available at https://github.com/agis85/multimodal_brain_synthesis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2010) · International Conference on Machine Learning · Cited by 13,995
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 2×
“We also show in our experiments that our architecture and cost function can be used in conjunction with various fusion methods, including the one proposed in HeMIS [26], and the model can be trained end-to-end without the need for the added complexity of curriculum learning [14].”
BackgroundMethodEfficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · cited 3×
“For our DeepMedic [25] test, we instead normalised the data by subtracting the mean and dividing by the standard deviation, as this is a requirement for the model.”
MethodInfluential(2011) · Cited by 2,752
Cited by
Diffusion models in medical imaging: A comprehensive survey
Kazerouni, A., Aghdam, E. K., Heidari, M., Azad, R., Fayyaz, M., Hacihaliloglu, I., & Merhof, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102846 · Cited by 765
SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining
Billot, B., Greve, D. N., Puonti, O., Thielscher, A., Van Leemput, K., Fischl, B., Dalca, A. V., & Iglesias, J. E. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102789 · Cited by 641
Shin, H.-C., Tenenholtz, N. A., Rogers, J. K., Schwarz, C. G., Senjem, M. L., Gunter, J. L., Andriole, K. P., & Michalski, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00536-8_1 · Cited by 638
Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks
Dar, S. U., Yurt, M., Karacan, L., Erdem, A., Erdem, E., & Cukur, T. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2901750 · Cited by 608
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis
Dalmaz, O., Yurt, M., & Cukur, T. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3167808 · Cited by 597
Mazurowski, M. A., Buda, M., Saha, A., & Bashir, M. R. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26534 · Cited by 554
Unsupervised Medical Image Translation With Adversarial Diffusion Models
Özbey, M., Dalmaz, O., Dar, S. U. H., Bedel, H. A., Özturk, Ş., Güngör, A., & Çukur, T. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3290149 · Cited by 517
Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis
Zhou, T., Fu, H., Chen, G., Shen, J., & Shao, L. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2975344 · Cited by 399
Ea-GANs: Edge-Aware Generative Adversarial Networks for Cross-Modality MR Image Synthesis
Yu, B., Zhou, L., Wang, L., Shi, Y., Fripp, J., & Bourgeat, P. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2895894 · Cited by 298
A review on medical imaging synthesis using deep learning and its clinical applications
Wang, T., Lei, Y., Fu, Y., Wynne, J. F., Curran, W. J., Liu, T., & Yang, X. (2021) · Journal of Applied Clinical Medical Physics · doi:10.1002/acm2.13121 · Cited by 279
Deep learning based synthesis of MRI, CT and PET: Review and analysis
Dayarathna, S., Islam, K. T., Uribe, S., Yang, G., Hayat, M., & Chen, Z. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103046 · Cited by 265
Causality-Inspired Single-Source Domain Generalization for Medical Image Segmentation
Ouyang, C., Chen, C., Li, S., Li, Z., Qin, C., Bai, W., & Rueckert, D. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3224067 · Cited by 260
Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network
Sharma, A., & Hamarneh, G. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2945521 · Cited by 254
Multi-modal learning from unpaired images: application to multi-organ segmentation in CT and MRI
Valindria, V. V., Pawlowski, N., Rajchl, M., Lavdas, I., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00066 · Cited by 146
“Multi-modal image synthesis of MRI is explored in [12, 6] by composing the networks of encoders (for each modality), fusing latent representations, and a decoder to produce an output image.”
BackgroundMulti-modal mri image synthesis via gan with multi-scale gate mergence
Zhan, B., Li, D., Wu, X., Zhou, J., & Wang, Y. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3088866 · Cited by 90
Artificial intelligence in glioma imaging: Challenges and advances
Jin, W., Fatehi, M., Abhishek, K., Mallya, M., Toyota, B., & Hamarneh, G. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab8131 · Cited by 55
“[44] proposed a multi-input multi-output deep model trained end-toend, with encoders and decoders for each MRI sequence.”
BackgroundPIMMS: Permutation Invariant Multi-modal Segmentation
Varsavsky, T., Eaton-Rosen, Z., Sudre, C. H., Nachev, P., & Cardoso, M. J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00889-5_23 · Cited by 22
Alam, M. M., & Latifi, S. (2025) · doi:10.20944/preprints202506.1854.v1 · Cited by 17
Metadata
- Authors
- Chartsias, A., Joyce, T., Giuffrida, M. V., & Tsaftaris, S. A.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2018-03
- Volume
- 37
- Issue
- 3
- Pages
- 803-814
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2767044624
- Cited by
- 341
- References
- 38
- FWCI
- 13.82
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.