Pixel-level multimodal fusion deep networks for predicting subcellular organelle localization from label-free live-cell imaging
Open accessAbstract
Complex intracellular organizations are commonly represented by dividing the metabolic process of cells into different organelles. Therefore, identifying sub-cellular organelle architecture is significant for understanding intracellular structural properties, specific functions, and biological processes in cells. However, the discrimination of these structures in the natural organizational environment and their functional consequences are not clear. In this article, we propose a new pixel-level multimodal fusion (PLMF) deep network which can be used to predict the location of cellular organelle using label-free cell optical microscopy images followed by deep-learning-based automated image denoising. It provides valuable insights that can be of tremendous help in improving the specificity of label-free cell optical microscopy by using the Transformer–Unet network to predict the ground truth imaging which corresponds to different sub-cellular organelle architectures. The new prediction method proposed in this article combines the advantages of a transformer’s global prediction and CNN’s local detail analytic ability of background features for label-free cell optical microscopy images, so as to improve the prediction accuracy. Our experimental results showed that the PLMF network can achieve over 0.91 Pearson’s correlation coefficient (PCC) correlation between estimated and true fractions on lung cancer cell-imaging datasets. In addition, we applied the PLMF network method on the cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new way for the time-resolved study of subcellular components in different cells, especially for cancer cells.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The unsupervised autoencoder models do not utilize any ground truth information during training, in contrast with previous supervised machine learning approaches for Raman cell imaging.
“Finally, we remark that our unsupervised autoencoder models do not utilize any ground truth information during training, in contrast with previous supervised machine learning approaches for Raman cell imaging and characterization where specific target concentrations are used as labels to train the model (e.g., using separately acquired fluorescence images) (90–93).”
From Hyperspectral unmixing for Raman spectroscopy via physics-constrained autoencoders (2024) · cites this paper as [92]
Methods §2.1 describes training a Raman-imaging model with a cell-image dataset, and Results §3.4 identifies fluorescence images as ground truth for predicting organelle locations. This supports the contrast with supervised Raman imaging, but the paper does not say that specific target concentrations were used as training labels. The citing authors’ statement about their own unsupervised autoencoders is not attributed to this source. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00075 · Cited by 6,612 · see citing works
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · see citing works
FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising
Zhang, K., Zuo, W., & Lei Zhang, A. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2839891 · Cited by 2,921 · see citing works
U-Net: deep learning for cell counting, detection, and morphometry
Falk, T., Mai, D., Bensch, R., Çiçek, Ö., Abdulkadir, A., Marrakchi, Y., Böhm, A., Deubner, J., Jäckel, Z., Seiwald, K., et al. (2018) · Nature Methods · doi:10.1038/s41592-018-0261-2 · Cited by 1,912 · see citing works
Chou, K., & Shen, H. (2008) · Nature Protocols · doi:10.1038/nprot.2007.494 · Cited by 1,226 · see citing works
DeepLoc: prediction of protein subcellular localization using deep learning
Armenteros, J. J. A., Sønderby, C. K., Sønderby, S. K., Nielsen, H., & Winther, O. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx431 · Cited by 1,144 · see citing works
FFDNet: Toward a fast and flexible solution for CNN-Based image denoising
Zhang, K., Zuo, W., & Zhang, L. (2018) · TUScholarShare (Temple University) · Cited by 916 · see citing works
Recent progress in protein subcellular location prediction
Chou, K., & Shen, H. (2007) · Analytical Biochemistry · doi:10.1016/j.ab.2007.07.006 · Cited by 769 · see citing works
Esakkirajan, S., Veerakumar, T., Subramanyam, A. N., & PremChand, C. H. (2011) · IEEE Signal Processing Letters · doi:10.1109/lsp.2011.2122333 · Cited by 570 · see citing works
Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy
Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F., & Johnson, G. R. (2018) · Nature Methods · doi:10.1038/s41592-018-0111-2 · Cited by 495 · see citing works
Deep Learning in Label-free Cell Classification
Chen, C. L., Mahjoubfar, A., Tai, L., Blaby, I. K., Huang, A., Niazi, K., & Jalali, B. (2016) · Scientific Reports · doi:10.1038/srep21471 · Cited by 449 · see citing works
Protein localization in disease and therapy
Hung, M., & Link, W. (2011) · Journal of Cell Science · doi:10.1242/jcs.089110 · Cited by 430 · see citing works
Chou, K., & Shen, H. (2010) · Natural Science · doi:10.4236/ns.2010.210136 · Cited by 348 · see citing works
Mitocellular communication: Shaping health and disease
Mottis, A., Herzig, S., & Auwerx, J. H. (2019) · Science · doi:10.1126/science.aax3768 · Cited by 238 · see citing works
A deep learning-based algorithm for 2-D cell segmentation in microscopy images
Al‐Kofahi, Y., Zaltsman, A., Graves, R., Marshall, W. W., & Rusu, M. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2375-z · Cited by 221 · see citing works
Cell segmentation methods for label-free contrast microscopy: review and comprehensive comparison
Vičar, T., Balvan, J., Jaroš, J., Jug, F., Kolář, R., Masařík, M., & Gumulec, J. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2880-8 · Cited by 221 · see citing works
Prediction of human protein subcellular localization using deep learning
Wei, L., Ding, Y., Su, R., Tang, J., & Zou, Q. (2018) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2017.08.009 · Cited by 216 · see citing works
Prospective identification of hematopoietic lineage choice by deep learning
Buggenthin, F., Buettner, F., Hoppe, P. S., Endele, M., Kroiss, M., Strasser, M. K., Schwarzfischer, M., Loeffler, D., Kokkaliaris, K. D., Hilsenbeck, O., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4182 · Cited by 197 · see citing works
SZABO, V., Ventalon, C., de Sars, V., Bradley, J., & Emiliani, V. (2014) · Neuron · doi:10.1016/j.neuron.2014.11.005 · Cited by 185 · see citing works
Prediction of protein subcellular locations by GO–FunD–PseAA predictor
Chou, K., & Cai, Y. (2004) · Biochemical and Biophysical Research Communications · doi:10.1016/j.bbrc.2004.06.073 · Cited by 179 · see citing works
Label‐Free Live‐Cell Imaging of Nucleic Acids Using Stimulated Raman Scattering Microscopy
Zhang, X., Roeffaers, M. B. J., Basu, S., Daniele, J. R., Fu, D., Freudiger, C. W., Holtom, G. R., & Xie, X. S. (2012) · ChemPhysChem · doi:10.1002/cphc.201100890 · Cited by 171 · see citing works
Zhang, L., Wu, Y., Zheng, B., Su, L., Chen, Y., Ma, S., Hu, Q., Zou, X., Yao, L., Yang, Y., et al. (2019) · Theranostics · doi:10.7150/thno.32655 · Cited by 159 · see citing works
Manifold, B., Men, S., Hu, R., & Fu, D. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00309-y · Cited by 152 · see citing works
Rahman, A., Alqahtani, A., Aldhafferi, N., Nasir, M. U., Khan, M. F., Khan, M. A., & Mosavi, A. (2022) · Sensors · doi:10.3390/s22103833 · Cited by 149 · see citing works
Regulation of liver subcellular architecture controls metabolic homeostasis
Parlakgül, G., Arruda, A. P., Pang, S., Cagampan, E., Min, N., Güney, E., Lee, G. Y., Inouye, K., Hess, H. F., Xu, C. S., et al. (2022) · Nature · doi:10.1038/s41586-022-04488-5 · Cited by 136 · see citing works
Denoising of stimulated Raman scattering microscopy images via deep learning
Manifold, B., Thomas, E. C., Francis, A. T., Hill, A. H., & Fu, D. (2019) · Biomedical Optics Express · doi:10.1364/boe.10.003860 · Cited by 125 · see citing works
An Overview of Organs-on-Chips Based on Deep Learning
Li, J., Chen, J., Bai, H., Wang, H., Hao, S., Ding, Y., Peng, B., Zhang, J., Li, L., & Huang, W. (2022) · Research · doi:10.34133/2022/9869518 · Cited by 113 · see citing works
Kobayashi, H., Lei, C., Wu, Y. J., Mao, A., Jiang, Y., Guo, B., Ozeki, Y., & Goda, K. (2017) · Scientific Reports · doi:10.1038/s41598-017-12378-4 · Cited by 107 · see citing works
DeepLoc: prediction of protein subcellular localization using deep learning
Armenteros, J. J. A., Sønderby, C. K., Sønderby, S. K., Nielsen, H., & Winther, O. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx548 · Cited by 101 · see citing works
Fast and efficient median filter for removing 1–99% levels of salt-and-pepper noise in images
Hsieh, M.-H., Cheng, F., Shie, M.-C., & Ruan, S.-J. (2012) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2012.10.012 · Cited by 100 · see citing works
Deep learning-based image processing in optical microscopy
Melanthota, S. K., Gopal, D., Chakrabarti, S., Kashyap, A. A., Radhakrishnan, R. A., & Mazumder, N. (2022) · Biophysical Reviews · doi:10.1007/s12551-022-00949-3 · Cited by 93 · see citing works
Jiang, Y., Lei, C., Yasumoto, A., Kobayashi, H., Aisaka, Y., Ito, T., Guo, B., Nitta, N., Kutsuna, N., Ozeki, Y., et al. (2017) · Lab on a Chip · doi:10.1039/c7lc00396j · Cited by 89 · see citing works
Lee, M., Lee, Y.-H., Song, J., Kim, G., Jo, Y., Min, H., Kim, C. H., & Park, Y. (2020) · eLife · doi:10.7554/elife.49023 · Cited by 77 · see citing works
Kim, G., Ahn, D., Kang, M., Park, J., Ryu, D., Jo, Y., Song, J., Ryu, J. S., Choi, G., Chung, H. J., et al. (2022) · Light Science & Applications · doi:10.1038/s41377-022-00881-x · Cited by 72 · see citing works
Koenig, F., Knittel, J., & Stepp, H. G. (2001) · Science · doi:10.1126/science.292.5520.1401 · Cited by 71 · see citing works
Islam, M. R., Nahiduzzaman, M., Goni, M. O. F., Sayeed, A., Anower, M. S., Ahsan, M., & Haider, J. (2022) · Sensors · doi:10.3390/s22124358 · Cited by 70 · see citing works
Xu, Y.-Y., Yang, F., Zhang, Y., & Shen, H. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt320 · Cited by 70 · see citing works
External Attention Based TransUNet and Label Expansion Strategy for Crack Detection
Fang, J., Yang, C., Shi, Y., Wang, N., & Zhao, Y. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3154407 · Cited by 69 · see citing works
A novel deep learning-based 3D cell segmentation framework for future image-based disease detection
Wang, A., Zhang, Q., Han, Y., Megason, S. G., Hormoz, S., Mosaliganti, K. R., Lam, J. C. K., & Li, V. O. K. (2022) · Scientific Reports · doi:10.1038/s41598-021-04048-3 · Cited by 68 · see citing works
Human Protein Subcellular Localization with Integrated Source and Multi-label Ensemble Classifier
Guo, X., Liu, F., Ju, Y., Wang, Z., & Wang, C. (2016) · Scientific Reports · doi:10.1038/srep28087 · Cited by 60 · see citing works
Yao, K., Rochman, N. D., & Sun, S. X. (2019) · Scientific Reports · doi:10.1038/s41598-019-50010-9 · Cited by 58 · see citing works
Pyramid Real Image Denoising Network
Zhao, Y., Jiang, Z., Men, A., & Ju, G. (2019) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip47243.2019.8965754 · Cited by 55 · see citing works
High-Speed Chemical Imaging by Dense-Net Learning of Femtosecond Stimulated Raman Scattering
Zhang, J., Zhao, J., Lin, H., Tan, Y., & Cheng, J.-X. (2020) · The Journal of Physical Chemistry Letters · doi:10.1021/acs.jpclett.0c01598 · Cited by 54 · see citing works
Siu, D. M. D., Lee, K. C. M., Lo, M. C. K., Stassen, S. V., Wang, M., Zhang, I. Z. Q., So, H. K., Chan, G., Cheah, K. S. E., Wong, K. K. Y., et al. (2020) · Lab on a Chip · doi:10.1039/d0lc00542h · Cited by 52 · see citing works
Deep Learning‐Based Single‐Cell Optical Image Studies
Sun, J., Tárnok, A., & Su, X. (2020) · Cytometry Part A · doi:10.1002/cyto.a.23973 · Cited by 51 · see citing works
Detection of Rare Objects by Flow Cytometry: Imaging, Cell Sorting, and Deep Learning Approaches
Voronin, D. V., Kozlova, A. A., Verkhovskii, R. A., Ermakov, A. V., Makarkin, M. A., Inozemtseva, O. A., & Bratashov, D. N. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21072323 · Cited by 48 · see citing works
Hasan, M. A. M., Ahmad, S., & Molla, M. K. I. (2017) · Molecular BioSystems · doi:10.1039/c6mb00860g · Cited by 46 · see citing works
A deep generative model of 3D single-cell organization
Donovan-Maiye, R. M., Brown, J. M., Chan, C. K., Ding, L., Yan, C., Gaudreault, N., Theriot, J. A., Maleckar, M. M., Knijnenburg, T. A., & Johnson, G. R. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009155 · Cited by 44 · see citing works
Ullah, M., Han, K., Hadi, F., Xu, J., Song, J., & Yu, D. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab278 · Cited by 42 · see citing works
Computational modeling of cellular structures using conditional deep generative networks
Yuan, H., Cai, L., Wang, Z., Hu, X., Zhang, S., & Ji, S. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty923 · Cited by 33 · see citing works
SSGraphCPI: A Novel Model for Predicting Compound-Protein Interactions Based on Deep Learning
Wang, X., Liu, J., Zhang, C., & Wang, S. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms23073780 · Cited by 33 · see citing works
Convolutional Blur Attention Network for Cell Nuclei Segmentation
Le, P. T., Pham, T. V., Hsu, Y., & Wang, J. (2022) · Sensors · doi:10.3390/s22041586 · Cited by 20 · see citing works
Gomariz, Á., Portenier, T., Helbling, P. M., Isringhausen, S., Suessbier, U., Nombela‐Arrieta, C., & Göksel, O. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00379-y · Cited by 19 · see citing works
Chen, X., Li, Y., Wyman, N., Zhang, Z., Fan, H., Le, M., Gannon, S., Rose, C., Zhang, Z., Mercuri, J. J., et al. (2021) · Biomedical Optics Express · doi:10.1364/boe.417478 · Cited by 18 · see citing works
Wei, H., Xie, L., Liu, Q., Shao, C., Wang, X., & Su, X. (2018) · Cytometry Part A · doi:10.1002/cyto.a.23671 · Cited by 17 · see citing works
Self-citationUllah, M., Hadi, F., Song, J., & Yu, D. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac432 · Cited by 15 · see citing works
Generative Adversarial Networks for Morphological–Temporal Classification of Stem Cell Images
Witmer, A., & Bhanu, B. (2021) · Sensors · doi:10.3390/s22010206 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Boslaugh S. Watters P. A. (2008). Statistics in a nutshell: A desktop quick reference. Sebastopol, CA: O'Reilly Media. Google Scholar
Cited by
Sorted by citations
Hyperspectral unmixing for Raman spectroscopy via physics-constrained autoencoders
Georgiev, D., Fernández-Galiana, Á., Vilms Pedersen, S., Papadopoulos, G., Xie, R., Stevens, M. M., & Barahona, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2407439121 · Cited by 50 · see citing works
“Therefore, no ground truth information is provided during training as opposed to previously reported machine learning approaches for Raman hyperspectral analysis and characterization based on supervised learning [58, 59, 60].”
BackgroundCovariance Matrix Reconstruction to Improve DoA Estimation Using Subspace Method in Low SNR Regime
Khichar, S., Santipach, W., & Wuttisittikulkij, L. (2025) · IEEE Access · doi:10.1109/access.2025.3528328 · Cited by 10 · see citing works
Wang, Z., Han, K., Liu, W., Wang, Z., Shi, C., Liu, X., Huang, M., Sun, G., Liu, S., & Guo, Q. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01001-4 · Cited by 10 · see citing works
Self-citationDeep Semantic Segmentation of Cell Painted Nuclei Images using UNet++
Sreekumar, S. P., Palanisamy, R., & Swaminathan, R. (2023) · Current Directions in Biomedical Engineering · doi:10.1515/cdbme-2023-1149 · Cited by 2 · see citing works
Semantic Segmentation of Cell Painted Organelles using DeepLabv3plus Model
Sreekumar, S. P., Palanisamy, R., & Swaminathan, R. (2023) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc40787.2023.10340728 · Cited by 1 · see citing works
Deep Learning‐Assisted Coherent Raman Scattering Microscopy
Liu, J., Liu, R., Zhang, X., Chen, X. G., & Yue, S. (2025) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500072 · Cited by 1 · see citing works
Sreekumar, S. P., Palanisamy, R., & Swaminathan, R. (2023) · Journal of Mechanics in Medicine and Biology · doi:10.1142/s0219519423400365 · Cited by 1 · see citing works
- Authors
- Wei, Z., Liu, X., Yan, R., Sun, G., Yu, W., Liu, Q., & Guo, Q.
- Affiliations
- Beijing Institute of Petrochemical Technology (CN)
- Corresponding author
- Guo, Qianjin
- Venue
- Frontiers in Genetics
- Publisher
- Frontiers Media SA
- Published
- 2022-10-26
- Volume
- 13
- Pages
- 1002327-1002327
- ISSN
- 1664-8021 (ISSN-L)
- PMC
- PMC9644055
- OpenAlex
- W4309361287
- Cited by
- 7
- References
- 58
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 3
- FWCI
- 0.78
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Genetics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Beijing Institute of Petrochemical Technology (22019821001, BIPTAAI-2021007)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Advanced Fluorescence Microscopy Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- label-free live-cell imaging, image denoising, intracellular organization, lung cancer cells, cancer cell imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
