The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Multimodal MR Synthesis via Modality-Invariant Latent Representation
Open accessAbstract
We propose a multi-input multi-output fully convolutional neural network model for MRI synthesis. The model is robust to missing data, as it benefits from, but does not require, additional input modalities. The model is trained end-to-end, and learns to embed all input modalities into a shared modality-invariant latent space. These latent representations are then combined into a single fused representation, which is transformed into the target output modality with a learnt decoder. We avoid the need for curriculum learning by exploiting the fact that the various input modalities are highly correlated. We also show that by incorporating information from segmentation masks the model can both decrease its error and generate data with synthetic lesions. We evaluate our model on the ISLES and BRATS data sets and demonstrate statistically significant improvements over state-of-the-art methods for single input tasks. This improvement increases further when multiple input modalities are used, demonstrating the benefits of learning a common latent space, again resulting in a statistically significant improvement over the current best method. Finally, we demonstrate our approach on non skull-stripped brain images, producing a statistically significant improvement over the previous best method. Code is made publicly available at https://github.com/agis85/multimodal_brain_synthesis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2010) · International Conference on Machine Learning · Cited by 13,995
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 2×
“We also show in our experiments that our architecture and cost function can be used in conjunction with various fusion methods, including the one proposed in HeMIS [26], and the model can be trained end-to-end without the need for the added complexity of curriculum learning [14].”
BackgroundMethodEfficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · cited 3×
“For our DeepMedic [25] test, we instead normalised the data by subtracting the mean and dividing by the standard deviation, as this is a requirement for the model.”
MethodInfluential(2011) · Cited by 2,752
Static Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
CNN-RNN: A Unified Framework for Multi-label Image Classification
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., & Xu, W. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.251 · Cited by 1,198
The Importance of Skip Connections in Biomedical Image Segmentation
Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., & Pal, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46976-8_19 · Cited by 984
Hierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968 · cited 1×
“Multimodal synthesis: Multimodal image analysis is on the rise, as evidenced by recent multimodal analysis methods for example to solve segmentation (e.g. [25], [26]) or classification (e.g. [ 27]) tasks.”
MethodDeep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis
Li, R., Zhang, W., Suk, H.-I., Wang, L., Li, J., Shen, D., & Ji, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10443-0_39 · Cited by 469
Attenuation Correction Synthesis for Hybrid PET-MR Scanners: Application to Brain Studies
Burgos, N., Cardoso, M. J., Thielemans, K., Modat, M., Pedemonte, S., Dickson, J., Barnes, A., Ahmed, R., Mahoney, C. J., Schott, J. M., et al. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2340135 · Cited by 364
HeMIS: Hetero-Modal Image Segmentation
Havaei, M., Guizard, N., Chapados, N., & Bengio, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_54 · Cited by 332 · cited 5×
“This method was shown in [26] to work very well for image segmentation, producing state of the art results.”
BackgroundMethodInfluentialEstimating CT Image From MRI Data Using Structured Random Forest and Auto-Context Model
Huynh, T., Gao, Y., Kang, J., Wang, L., Zhang, P., Lian, J., & Shen, D. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2461533 · Cited by 282
Random forest regression for magnetic resonance image synthesis
Jog, A., Carass, A., Roy, S., Pham, D. L., & Prince, J. L. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.08.009 · Cited by 212 · cited 6×
“Regression Ensembles with Patch Learning for Image Contrast Agreement (REPLICA) : Our final baseline method is REPLICA [9], a supervised random forest image synthesis approach which uses multi-scale features to achieve accurate synthesis results.”
BackgroundMethodInfluentialLearning Aligned Cross-Modal Representations from Weakly Aligned Data
Castrejon, L., Aytar, Y., Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.321 · Cited by 188 · cited 2×
“Thus, our approach is broadly similar to the statistical regularisation approach in [32], in which cross-modal scene representations are learnt.”
BackgroundResultMagnetic Resonance Image Example-Based Contrast Synthesis
Roy, S., Carass, A., & Prince, J. L. (2013) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2013.2282126 · Cited by 162
Is Synthesizing MRI Contrast Useful for Inter-modality Analysis?
Iglesias, J. E., Konukoğlu, E., Zikic, D., Glocker, B., Leemput, K. V., & Fischl, B. R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40811-3_79 · Cited by 147
Cross-Domain Synthesis of Medical Images Using Efficient Location-Sensitive Deep Network
Nguyen, H. V., Zhou, K., & Vemulapalli, R. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_83 · Cited by 140
Chandar, S., Khapra, M. M., Larochelle, H., & Ravindran, B. (2015) · Neural Computation · doi:10.1162/neco_a_00801 · Cited by 133
Why Does Synthesized Data Improve Multi-sequence Classification?
van Tulder, G., & de Bruijne, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_65 · Cited by 111
Modality Propagation: Coherent Synthesis of Subject-Specific Scans with Data-Driven Regularization
Ye, D. H., Zikic, D., Glocker, B., Criminisi, A., & Ender Konukoglu, A. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40811-3_76 · Cited by 108
Magnetic resonance image synthesis through patch regression
Jog, A., Roy, S., Carass, A., & Prince, J. L. (2013) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2013.6556484 · Cited by 84
PET Attenuation Correction Using Synthetic CT from Ultrashort Echo-Time MR Imaging
Roy, S., Wang, W. T., Carass, A., Prince, J. L., Butman, J. A., & Pham, D. L. (2014) · Journal of Nuclear Medicine · doi:10.2967/jnumed.114.143958 · Cited by 79
MR image synthesis by contrast learning on neighborhood ensembles
Jog, A., Carass, A., Roy, S., Pham, D. L., & Prince, J. L. (2015) · Medical Image Analysis · doi:10.1016/j.media.2015.05.002 · Cited by 76
Robust Multi-modal MR Image Synthesis
Joyce, T., Chartsias, A., & Tsaftaris, S. A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_40 · Cited by 72
Self-citationUnsupervised Cross-Modal Synthesis of Subject-Specific Scans
Vemulapalli, R., Nguyen, H. V., & Zhou, S. K. (2015) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.79 · Cited by 69
Whole Image Synthesis Using a Deep Encoder-Decoder Network
Sevetlidis, V., Giuffrida, M. V., & Tsaftaris, S. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_13 · Cited by 68
Self-citationExtended Modality Propagation: Image Synthesis of Pathological Cases
Cordier, N., Delingette, H., Lê, M., & Ayache, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2589760 · Cited by 62
Pseudo-healthy Image Synthesis for White Matter Lesion Segmentation
Bowles, C. T., Qin, C., Ledig, C., Guerrero, R., Gunn, R. N., Hammers, A., Sakka, E., Dickie, D. A., Hernández, M. V., Royle, N. A., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_9 · Cited by 48
Registration of Pathological Images
Yang, X., Han, X., Park, E., Aylward, S., Kwitt, R., & Niethammer, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_10 · Cited by 31
Huang, Y., Beltrachini, L., Shao, L., & Frangi, A. F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_12 · Cited by 24
Patch Based Synthesis of Whole Head MR Images: Application To EPI Distortion Correction
Roy, S., Chou, Y., Jog, A., Butman, J. A., & Pham, D. L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_15 · Cited by 20
Generalized K-fan Multimodal Deep Model with Shared RepresentationsPreprint
Chen, G., & Srihari, S. N. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.07906 · Cited by 5
Cited by
Diffusion models in medical imaging: A comprehensive survey
Kazerouni, A., Aghdam, E. K., Heidari, M., Azad, R., Fayyaz, M., Hacihaliloglu, I., & Merhof, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102846 · Cited by 765
SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining
Billot, B., Greve, D. N., Puonti, O., Thielscher, A., Van Leemput, K., Fischl, B., Dalca, A. V., & Iglesias, J. E. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102789 · Cited by 641
Shin, H.-C., Tenenholtz, N. A., Rogers, J. K., Schwarz, C. G., Senjem, M. L., Gunter, J. L., Andriole, K. P., & Michalski, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00536-8_1 · Cited by 638
Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks
Dar, S. U., Yurt, M., Karacan, L., Erdem, A., Erdem, E., & Cukur, T. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2901750 · Cited by 608
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis
Dalmaz, O., Yurt, M., & Cukur, T. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3167808 · Cited by 597
Metadata
- Authors
- Chartsias, A., Joyce, T., Giuffrida, M. V., & Tsaftaris, S. A.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2018-03
- Volume
- 37
- Issue
- 3
- Pages
- 803-814
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2767044624
- Cited by
- 341
- References
- 38
- FWCI
- 13.82
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.