The DOI matches its Crossref record; 49 of the 51 references it cites were found in a registry or an index.
Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network
Open accessAbstract
Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However, many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians and automated systems since complementary information provided by the missing sequences is lost. In this paper, we propose a variant of generative adversarial network (GAN) capable of leveraging redundant information contained within multiple available sequences in order to generate one or more missing sequences for a patient scan. The proposed network is designed as a multi-input, multi-output network which combines information from all the available pulse sequences and synthesizes the missing ones in a single forward pass. We demonstrate and validate our method on two brain MRI datasets each with four sequences, and show the applicability of the proposed method in simultaneously synthesizing all missing sequences in any possible scenario where either one, two, or three of the four sequences may be missing. We compare our approach with competing unimodal and multi-modal methods, and show that we outperform both quantitatively and qualitatively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Performance of these methods depend on the generation model that imputes the missing modalities.”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [25]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“4) Curriculum learning: In order to train our proposed method we use a curriculum learning (CL) [45]”
Instance Normalization: The Missing Ingredient for Fast StylizationPreprint
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.08022 · Cited by 4,325
Elastix: a toolbox for intensity-based medical image registration
Klein, S., Staring, M., Murphy, K., Viergever, M. A., & Pluim, J. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2035616 · Cited by 4,311
Brain tumor segmentation with Deep Neural Networks
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., & Larochelle, H. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.05.004 · Cited by 3,278
Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images
Pereira, S., Pinto, A., Alves, V., & Silva, C. A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2538465 · Cited by 2,735
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Deep multi-scale video prediction beyond mean square errorPreprint
Mathieu, M., Couprie, C., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05440 · Cited by 1,592
VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images
Chen, H., Dou, Q., Yu, L., Qin, J., & Heng, P. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.04.041 · Cited by 781
SegAN: Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation
Xue, Y., Xu, T., Zhang, H., Long, L. R., & Huang, X. (2018) · Neuroinformatics · doi:10.1007/s12021-018-9377-x · Cited by 637
Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks
Dar, S. U. H., Yurt, M., Karacan, L., Erdem, A., Erdem, E., & Çukur, T. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2901750 · Cited by 635
GANs for medical image analysis
Kazeminia, S., Baur, C., Kuijper, A., van Ginneken, B., Navab, N., Albarqouni, S., & Mukhopadhyay, A. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101938 · Cited by 608
HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation
Dolz, J., Gopinath, K. K., Yuan, J. Y., Lombaert, H., Desrosiers, C., & Ayed, I. B. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2878669 · Cited by 605
Automatic Brain Tumor Segmentation Using Cascaded Anisotropic Convolutional Neural Networks
Wang, G., Li, W., Sebastien Ourselin, A., & Vercauteren, T. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9_16 · Cited by 579
Maier, O., Menze, B. H., von der Gablentz, J., Häni, L., Heinrich, M. P., Liebrand, M., Winzeck, S., Basit, A., Bentley, P., Chen, L., et al. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.07.009 · Cited by 553
Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge
Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., & Maier-Hein, K. H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9_25 · Cited by 528
MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music GenerationPreprint
Yang, L.-C., Chou, S.-Y., & Yang, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10847 · Cited by 438
Multimodal MR Synthesis via Modality-Invariant Latent Representation
Chartsias, A., Joyce, T., Giuffrida, M. V., & Tsaftaris, S. A. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2764326 · Cited by 360
BET2 : MR-Based Estimation of Brain, Skull and Scalp Surfaces
Jenkinson, M. (2005) · Medical Entomology and Zoology · Cited by 348
HeMIS: Hetero-Modal Image Segmentation
Havaei, M., Guizard, N., Chapados, N., & Bengio, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_54 · Cited by 332 · cited 2×
“Most notably, Havaei et al. [3] and Varsavsky et al. [4] provide methods for tumor segmentation using brain MRI that are robust to missing sequences [3], or to missing sequence labels [4].”
Ea-GANs: Edge-Aware Generative Adversarial Networks for Cross-Modality MR Image Synthesis
Yu, B., Zhou, L., Wang, L., Shi, Y., Fripp, J., & Bourgeat, P. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2895894 · Cited by 306
Adversarial Stain Transfer for Histopathology Image Analysis
BenTaieb, A., & Hamarneh, G. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2781228 · Cited by 235
Self-citationRandom forest regression for magnetic resonance image synthesis
Jog, A., Carass, A., Roy, S., Pham, D. L., & Prince, J. L. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.08.009 · Cited by 212
Learning to Generate Images of Outdoor Scenes from Attributes and Semantic LayoutsPreprint
Karacan, L., Akata, Z., Erdem, A., & Erdem, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00215 · Cited by 143
Is Synthesizing MRI Contrast Useful for Inter-modality Analysis?
Iglesias, J. E., Konukoğlu, E., Zikic, D., Glocker, B., Leemput, K. V., & Fischl, B. R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40811-3_79 · Cited by 142
Adversarial Training and Dilated Convolutions for Brain MRI Segmentation
Moeskops, P., Veta, M., Lafarge, M. W., Eppenhof, K. A. J., & Pluim, J. P. W. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_7 · Cited by 126
Cross-Domain Synthesis of Medical Images Using Efficient Location-Sensitive Deep Network
Nguyen, H. V., Zhou, K., & Vemulapalli, R. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_83 · Cited by 122
Modality Propagation: Coherent Synthesis of Subject-Specific Scans with Data-Driven Regularization
Ye, D. H., Zikic, D., Glocker, B., Criminisi, A., & Ender Konukoglu, A. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40811-3_76 · Cited by 108
Why Does Synthesized Data Improve Multi-sequence Classification?
van Tulder, G., & de Bruijne, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_65 · Cited by 89
Magnetic resonance image synthesis through patch regression
Jog, A., Roy, S., Carass, A., & Prince, J. L. (2013) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2013.6556484 · Cited by 84
Multiple Sclerosis Lesion Synthesis in MRI Using an Encoder-Decoder U-NET
Salem, M., Valverde, S., Cabezas, M., Pareto, D., Oliver, A., Salví, J., Rovira, À., & Lladó, X. (2019) · IEEE Access · doi:10.1109/access.2019.2900198 · Cited by 79
Generative Adversarial Training for MRA Image Synthesis Using Multi-contrast MRI
Olut, Ş., Şahin, Y. H., Demir, U., & Ünal, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00320-3_18 · Cited by 75
Whole Image Synthesis Using a Deep Encoder-Decoder Network
Sevetlidis, V., Giuffrida, M. V., & Tsaftaris, S. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_13 · Cited by 68
Generative adversarial networks to segment skin lesions
Izadi, S., Mirikharaji, Z., Kawahara, J., & Hamarneh, G. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363712 · Cited by 58
Self-citationA review of MRI pulse sequences and techniques in neuroimaging
Jackson, E. F., Ginsberg, L. E., Schomer, D. F., & Leeds, N. E. (1997) · Surgical Neurology · doi:10.1016/s0090-3019(96)00375-8 · Cited by 55
Pseudo-healthy Image Synthesis for White Matter Lesion Segmentation
Bowles, C. T., Qin, C., Ledig, C., Guerrero, R., Gunn, R. N., Hammers, A., Sakka, E., Dickie, D. A., Hernández, M. D. C. V., Royle, N. A., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_9 · Cited by 49
Generative Adversarial Networks for MR-CT Deformable Image RegistrationPreprint
Tanner, C., Özdemir, F., Profanter, R., Vishnevskiy, V., Ender Konukoglu, A., & Göksel, O. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07349 · Cited by 45
Huang, Y., Beltrachini, L., Shao, L., & Frangi, A. F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_12 · Cited by 24
Mehta, R., & Arbel, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00536-8_13 · Cited by 24
PIMMS: Permutation Invariant Multi-modal Segmentation
Varsavsky, T., Eaton-Rosen, Z., Sudre, C. H., Nachev, P., & Cardoso, M. J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00889-5_23 · Cited by 22
Patch Based Synthesis of Whole Head MR Images: Application To EPI Distortion Correction
Roy, S., Chou, Y., Jog, A., Butman, J. A., & Pham, D. L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46630-9_15 · Cited by 20
Random forest FLAIR reconstruction from T1, T2, and PD-weighted MRI
Jog, A., Carass, A., Pham, D. L., & Prince, J. L. (2014) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2014.6868061 · Cited by 18
Tree-Encoded Conditional Random Fields for Image Synthesis
Jog, A., Carass, A., Pham, D. L., & Prince, J. L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19992-4_58 · Cited by 8
Research works printed but not linked: 3
- gauthier, 2014, Conditional generative adversarial nets for convolutional face generation, Class Project Stanford CS231N Convolutional Neural Netw Vis Recognit Winter Semester, 2014, 2
- goodfellow, 2014, Generative adversarial nets, Proc Adv Neural Inf Process Syst, 2672
- 2017, 2017 International MICCAI BraTS Challenge
Cited by
Sorted by citations
Diffusion models in medical imaging: A comprehensive survey
Kazerouni, A., Aghdam, E. K., Heidari, M., Azad, R., Fayyaz, M., Hacihaliloglu, I., & Merhof, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102846 · Cited by 1,006
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis
Dalmaz, O., Yurt, M., & Çukur, T. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3167808 · Cited by 642
Unsupervised Medical Image Translation With Adversarial Diffusion Models
Özbey, M., Dalmaz, O., Dar, S. U. H., Bedel, H. A., Öztürk, Ş., Güngör, A., & Çukur, T. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3290149 · Cited by 590
Data augmentation for medical imaging: A systematic literature review
Garcea, F., Serra, A., Lamberti, F., & Morra, L. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106391 · Cited by 451
Deep learning based synthesis of MRI, CT and PET: Review and analysis
Dayarathna, S., Islam, K. T., Uribe, S. A., Yang, G., Hayat, M., & Chen, Z. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103046 · Cited by 316
Metadata
- Authors
- Sharma, A., & Hamarneh, G.
- Affiliations
- Simon Fraser University (CA)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2020-04
- Volume
- 39
- Issue
- 4
- Pages
- 1170-1183
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2941736865
- Cited by
- 254
- References
- 51
- Citations by year
- 2026: 37 · 2025: 66 · 2024: 50 · 2023: 32 · 2022: 31 · 2021: 20 · 2020: 17
- FWCI
- 11.39
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Summit (Simon Fraser University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Compute Canada
- Western Canada Research Grid
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- brain MRI, scan time reduction, multi-contrast MRI
- MeSH terms
- Deep Learning, Brain, Humans, Image Processing, Computer-Assisted, Magnetic Resonance Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
