Deep multi-modal fusion network with gated unit for breast cancer survival prediction
Abstract
Accurate survival prediction is a critical goal in the prognosis of breast cancer patients because it can help physicians make more patient-friendly decisions and further guide appropriate treatment. Breast cancer is often caused by genetic abnormalities, which prompts researchers to consider information such as gene expression and copy number variation in addition to clinical data in their studies. The integration of these multi-modal data can improve the predictive power of models. However, with the highly unbalanced information of breast cancer patient data, it becomes a new challenge for breast cancer patient survival prediction to fully extract the characteristic information of these multi-modal data and to consider the complementarity of this information. To this end, we propose a deep multi-modal fusion network (DMMFN) to predict the five-year survival of breast cancer patients by integrating clinical data, copy number variation data, and gene expression data. The imbalanced dataset is first processed using the oversampling method SMOTE-NC. Then the abstract modal features of the multi-modal data are extracted by the two-layer one-dimensional convolutional neural network and the bi-directional long short-term memory network. Next, the weight coefficients of each modal data are dynamically adjusted using gated multimodal units to obtain fusion features. Finally, the fusion features are fed into the MaxoutMLP classifier to obtain the final prediction results. We conducted experiments on the METABRIC dataset to verify the validity of the multi-modal data and compared it with other methods. The comprehensive performance evaluation shows that DMMFN has better prediction performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Gated fusion uses mechanisms such as neural network gates or attention mechanisms to dynamically emphasize or de-emphasize different aspects or modalities of the data based on the context.
“Gated fusion [145,146] uses mechanisms such as neural network gates or attention mechanisms to dynamically emphasize or de-emphasize different aspects or modalities of the data based on the context.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [146]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Regression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793 · see citing works
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280 · cited 1× · see citing works
“The most typical multilayer perceptron MLP consists of three layers: input, hidden, and output layers, which are fully connected between the different layers (Rosenblatt, 1958).”
BackgroundThe genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups
METABRIC Group, Curtis, C., Shah, S. P., Chin, S.-F., Turashvili, G., Rueda, O. M., Dunning, M. J., Speed, D., Lynch, A. G., Samarajiwa, S., et al. (2012) · Nature · doi:10.1038/nature10983 · Cited by 6,097 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
(1989) · neural information processing systems · Cited by 3,736 · see citing works
Katzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., & Kluger, Y. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0482-1 · Cited by 1,995 · see citing works
Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data
Ching, T. H., Zhu, X., & Garmire, L. X. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006076 · Cited by 403 · see citing works
The Application of Deep Learning in Cancer Prognosis Prediction
Zhu, W., Xie, L., Han, J., & Guo, X. (2020) · Cancers · doi:10.3390/cancers12030603 · Cited by 384 · see citing works
Sun, D., Wang, M., & Li, A. (2019) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2018.2806438 · Cited by 338 · see citing works
SALMON: survival analysis learning with multi-omics neural networks on breast cancer
Huang, Z., Zhan, X., Xiang, S., Johnson, T. S., Helm, B., Yu, C. Y., Zhang, J., Salama, P., Rizkalla, M., Han, Z., et al. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.00166 · Cited by 262 · cited 1× · see citing works
“Therefore, such methods are often challenging to predict the survival of breast cancer patients effectively and accurately (Huang et al., 2019; Joshi et al., 2021; Tan et al., 2022).”
BackgroundMachine learning and clinical epigenetics: a review of challenges for diagnosis and classification
Rauschert, S., Raubenheimer, K., Melton, P. E., & Huang, R. (2020) · Clinical Epigenetics · doi:10.1186/s13148-020-00842-4 · Cited by 242 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Fast-mRMR: Fast Minimum Redundancy Maximum Relevance Algorithm for High-Dimensional Big Data
Ramírez‐Gallego, S., Lastra, I., Martínez‐Rego, D., Bolón‐Canedo, V., Benítez, J. M., Herrera, F., & Alonso‐Betanzos, A. (2016) · International Journal of Intelligent Systems · doi:10.1002/int.21833 · Cited by 203 · see citing works
Kourou, K. D., Exarchos, K., Papaloukas, C. L., Sakaloglou, P., Exarchos, T., & Fotiadis, D. I. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.006 · Cited by 166 · see citing works
Wang, K., Makond, B., Chen, K.-H., & Wang, K.-M. (2013) · Applied Soft Computing · doi:10.1016/j.asoc.2013.09.014 · Cited by 140 · see citing works
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04559-1 · Cited by 139 · see citing works
Multi-modal advanced deep learning architectures for breast cancer survival prediction
Arya, N., & Saha, S. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106965 · Cited by 127 · see citing works
Kılıçarslan, S., Çelik, M., & Şahin, Ş. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.102231 · Cited by 126 · see citing works
Arya, N., & Saha, S. (2020) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2020.3018467 · Cited by 117 · see citing works
Kalafi, E. Y., Nor, N. A. M., Taib, N. A. M., et al. (2019) · Folia Biologica · doi:10.14712/fb2019065050212 · Cited by 109 · see citing works
A Survey of Machine Learning Approaches Applied to Gene Expression Analysis for Cancer Prediction
Khalsan, M., Machado, L. R., Al-Shamery, E. S., Ajit, S., Anthony, K., Mu, M., & Agyeman, M. O. (2022) · IEEE Access · doi:10.1109/access.2022.3146312 · Cited by 103 · see citing works
Multimodal data fusion for systems improvement: A review
Gaw, N., Yousefi, S., & Gahrooei, M. R. (2022) · IISE Transactions · doi:10.1080/24725854.2021.1987593 · Cited by 101 · see citing works
A deep survival analysis method based on ranking
Jing, B., Zhang, T., Wang, Z., Jin, Y., Liu, K., Qiu, W., Ke, L., Sun, Y. H., He, C., Hou, D., et al. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.06.001 · Cited by 91 · see citing works
Tan, K., Huang, W., Liu, X., Hu, J., & Dong, S. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102260 · Cited by 75 · see citing works
A Comprehensive Review on Medical Diagnosis Using Machine Learning
Bhavsar, K. A., Abugabah, A., Singla, J., AlZubi, A. A., Bashir, A. K., & Nikita, N. (2021) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2021.014943 · Cited by 58 · see citing works
A systematic survey of deep learning in breast cancer
Yu, X., Zhou, Q., Wang, S., & Zhang, Y. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22622 · Cited by 54 · see citing works
Kim, J., Lee, Y., Yu, J., Park, Y., Lee, S. K., Lee, M., Lee, J. E., Kim, S. W., Nam, S. J., Park, Y. H., et al. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.596364 · Cited by 44 · see citing works
Rahman, S. A. E. (2020) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-020-02590-y · Cited by 41 · see citing works
A Comparative Analysis of Deep Learning Approaches for Predicting Breast Cancer Survivability
Gupta, S., & Gupta, M. K. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09679-3 · Cited by 41 · see citing works
Integrative survival analysis of breast cancer with gene expression and DNA methylation data
Bichindaritz, I., Liu, G., & Bartlett, C. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab140 · Cited by 40 · see citing works
Wang, S., Zhang, H., Liu, Z., & Liu, Y. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.800853 · Cited by 39 · see citing works
Kaur, P., Singh, A., & Chana, I. (2022) · Journal of Computational Science · doi:10.1016/j.jocs.2022.101570 · Cited by 38 · see citing works
El-Nabawy, A., El-Bendary, N., & Belal, N. A. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106238 · Cited by 35 · see citing works
Generative Incomplete Multi-View Prognosis Predictor for Breast Cancer: GIMPP
Arya, N., & Saha, S. (2021) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3090458 · Cited by 27 · see citing works
A Multimodal Affinity Fusion Network for Predicting the Survival of Breast Cancer Patients
Guo, W., Liang, W., Deng, Q., & Zou, X. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.709027 · Cited by 27 · see citing works
An Ovarian Cancer Susceptible Gene Prediction Method Based on Deep Learning Methods
Lu, Y., Zhang, Y., Xinying, Y., et al. (2021) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2021.730475 · Cited by 12 · see citing works
Salehi, M., Razmara, J., Lotfi, S., et al. (2021) · The Computer Journal · doi:10.1093/comjnl/bxab096 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Gated fusion [145,146] uses mechanisms such as neural network gates or attention mechanisms to dynamically emphasize or de-emphasize different aspects or modalities of the data based on the context.”
- Authors
- Yuan, H., & Xu, H.
- Affiliations
- East China University of Technology (CN)
- Corresponding author
- Xu, Hongzhen
- ORCID
- Xu, Hongzhen
- Venue
- Computer Methods in Biomechanics and Biomedical Engineering
- Publisher
- Informa UK Limited
- Published
- 2023-05-11
- Volume
- 27
- Issue
- 7
- Pages
- 883-896
- ISSN
- 1025-5842 (ISSN-L)
- 1476-8259
- OpenAlex
- W4376130909
- Cited by
- 10
- References
- 39
- Citations by year
- 2026: 5 · 2025: 2 · 2024: 3
- FWCI
- 0.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Computer Methods in Biomechanics & Biomedical Engineering (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Scientific Research Plan Projects of Education Department of Jiangxi Province of China (GJJ160554)
- Talent Plan Project of Fuzhou City of Jiangxi Province of China (2021ED008)
- Opening Project of Jiangxi Key Laboratory of Cybersecurity Intelligent Perception (JKLCIP202202)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Breast Cancer Treatment Studies
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- breast cancer, survival prediction, multimodal fusion, copy number variation, gene expression, clinical data, class imbalance, SMOTE-NC, 1D convolutional neural network, bidirectional LSTM, METABRIC
- MeSH terms
- Deep Learning, Breast Neoplasms, Female, Humans, Prognosis, Survival Analysis, Neural Networks, Computer, DNA Copy Number Variations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
